AI Machine Learning Operations (MLOps)
Consultant Engineering Data Analysis AI/Artificial Intelligence AI/ML Machine Learning
Mức lương
Khu vực
Phúc lợi
Lương tháng 13
Cấp laptop
Phúc lợi khác
Cho phép làm tại nhà
Tổng quan về công việc và trách nhiệm
The Opportunity This role owns end-to-end reliability for AI agents in production. Our agents power real user decisions and must be correct, safe, and trustworthy every day. As a startup, we don’t separate QA, evals, safety, and reliability into different teams—you are the single owner of agent quality in the real world. You’ll define what “good” means, build the evaluation and monitoring systems that enforce it, and close the loop from production failures back into fixes. This is a hands-on, high-impact role for engineers who want to focus deeply on agent correctness, trust, and reliability at scale. Core Responsibilities - Own agent correctness and quality in production, from definition of success to enforcement and continuous improvement - Build and maintain automated evaluation systems (golden datasets, regression tests, LLM-as-judge, rule-based checks) - Design and operate production monitoring and observability for agent quality, failures, drift, and regressions - Define and enforce guardrails and safety boundaries across prompts, tools, permissions, and escalation paths - Run tight feedback loops: turn real failures into tests, fixes, and lasting reliability improvements
Kỹ năng và kinh nghiệm tối thiểu
- 4+ years of experience in AI engineering, data analytics engineering, ML systems, or reliability-focused roles (SRE, MLOps, ML QA) - Strong instincts for quality, failure modes, metrics, and regression testing in probabilistic systems - Hands-on experience with Python for evaluation pipelines, automation, and data analysis - Experience working close to production systems, including logs, metrics, dashboards, and incident response - Ability to translate real-world workflows into clear success criteria, KPIs, and test cases - Strong communication skills—you can explain quality tradeoffs to engineers, product, and leadership
Tại sao ứng viên nên làm ở đây
- Build from scratch: Be part of creating the customer insights function for a major ecommerce platform - Real impact: Your analyses will directly inform strategy and drive business decisions - Learn and grow: Work with experienced entrepreneurs, consultants from Bain, data engineers, and leadership - Modern tools: Access to AI tools and data infrastructure to do your best work - International exposure: Opportunity for US office rotations or visits - Competitive package - Hybrid working model - Macbook Pro provided for work - Bonus: 13th month salary - Weekly learning sessions and offsites - Close-knit small team culture
Ưu tiên có kỹ năng và kinh nghiệm
- Experience building or evaluating LLM-powered or agentic systems (prompting, RAG, tool calling, memory) - Familiarity with LLM evaluation techniques, including offline evals, trajectory analysis, and LLM-as-judge - Experience with agent observability: tracing, sampling, confidence signals, or grounding - Background in data systems or analytics platforms, where correctness and trust are critical - SRE or MLOps experience: production monitoring, incident response, on-call rotation - Experience with A/B testing, experimentation frameworks, or statistical evaluation methods - Comfortable working in ambiguity and iterating toward clarity using data and examples
Báo cáo cho
CDO
Quá trình phỏng vấn
Overview interview -> Technical interview, 90mins (at office) -> Fit interview
