AI Machine Learning Operations (MLOps)
Consultant Engineering Machine Learning AI/ML AI/Artificial Intelligence Data Analysis
Salary
Location
Benefits
13th month salary
Laptop/desktop for works
Other benefits
Work-from-home
Job Overview And Responsibility
The Opportunity This role owns end-to-end reliability for AI agents in production. Our agents power real user decisions and must be correct, safe, and trustworthy every day. As a startup, we don’t separate QA, evals, safety, and reliability into different teams—you are the single owner of agent quality in the real world. You’ll define what “good” means, build the evaluation and monitoring systems that enforce it, and close the loop from production failures back into fixes. This is a hands-on, high-impact role for engineers who want to focus deeply on agent correctness, trust, and reliability at scale. Core Responsibilities - Own agent correctness and quality in production, from definition of success to enforcement and continuous improvement - Build and maintain automated evaluation systems (golden datasets, regression tests, LLM-as-judge, rule-based checks) - Design and operate production monitoring and observability for agent quality, failures, drift, and regressions - Define and enforce guardrails and safety boundaries across prompts, tools, permissions, and escalation paths - Run tight feedback loops: turn real failures into tests, fixes, and lasting reliability improvements
Required Skills and Experience
- 4+ years of experience in AI engineering, data analytics engineering, ML systems, or reliability-focused roles (SRE, MLOps, ML QA) - Strong instincts for quality, failure modes, metrics, and regression testing in probabilistic systems - Hands-on experience with Python for evaluation pipelines, automation, and data analysis - Experience working close to production systems, including logs, metrics, dashboards, and incident response - Ability to translate real-world workflows into clear success criteria, KPIs, and test cases - Strong communication skills—you can explain quality tradeoffs to engineers, product, and leadership
Why Candidate should apply this position
- Build from scratch: Be part of creating the customer insights function for a major ecommerce platform - Real impact: Your analyses will directly inform strategy and drive business decisions - Learn and grow: Work with experienced entrepreneurs, consultants from Bain, data engineers, and leadership - Modern tools: Access to AI tools and data infrastructure to do your best work - International exposure: Opportunity for US office rotations or visits - Competitive package - Hybrid working model - Macbook Pro provided for work - Bonus: 13th month salary - Weekly learning sessions and offsites - Close-knit small team culture
Preferred skills and experiences
- Experience building or evaluating LLM-powered or agentic systems (prompting, RAG, tool calling, memory) - Familiarity with LLM evaluation techniques, including offline evals, trajectory analysis, and LLM-as-judge - Experience with agent observability: tracing, sampling, confidence signals, or grounding - Background in data systems or analytics platforms, where correctness and trust are critical - SRE or MLOps experience: production monitoring, incident response, on-call rotation - Experience with A/B testing, experimentation frameworks, or statistical evaluation methods - Comfortable working in ambiguity and iterating toward clarity using data and examples
Report to
CEO
Interview process
Overview interview -> Technical interview, 90mins (at office) -> Fit interview
Apply for this job
Completed jobs (1)
-
![Check]()
Placement for Technical Support Engineer (Microsoft Windows)
