-
Browse Jobs by Category
-
Services
HR & Recruitment Services
Payroll Services
Recruitment Services
Services by Country
Consultant Engineering AI/ML Data Analysis Machine Learning AI/Artificial Intelligence
The Opportunity This role owns end-to-end reliability for AI agents in production. Our agents power real user decisions and must be correct, safe, and trustworthy every day. As a startup, we don’t separate QA, evals, safety, and reliability into different teams—you are the single owner of agent quality in the real world. You’ll define what “good” means, build the evaluation and monitoring systems that enforce it, and close the loop from production failures back into fixes. This is a hands-on, high-impact role for engineers who want to focus deeply on agent correctness, trust, and reliability at scale. Core Responsibilities - Own agent correctness and quality in production, from definition of success to enforcement and continuous improvement - Build and maintain automated evaluation systems (golden datasets, regression tests, LLM-as-judge, rule-based checks) - Design and operate production monitoring and observability for agent quality, failures, drift, and regressions - Define and enforce guardrails and safety boundaries across prompts, tools, permissions, and escalation paths - Run tight feedback loops: turn real failures into tests, fixes, and lasting reliability improvements
- 4+ years of experience in AI engineering, data analytics engineering, ML systems, or reliability-focused roles (SRE, MLOps, ML QA) - Strong instincts for quality, failure modes, metrics, and regression testing in probabilistic systems - Hands-on experience with Python for evaluation pipelines, automation, and data analysis - Experience working close to production systems, including logs, metrics, dashboards, and incident response - Ability to translate real-world workflows into clear success criteria, KPIs, and test cases - Strong communication skills—you can explain quality tradeoffs to engineers, product, and leadership
- Build from scratch: Be part of creating the customer insights function for a major ecommerce platform - Real impact: Your analyses will directly inform strategy and drive business decisions - Learn and grow: Work with experienced entrepreneurs, consultants from Bain, data engineers, and leadership - Modern tools: Access to AI tools and data infrastructure to do your best work - International exposure: Opportunity for US office rotations or visits - Competitive package - Hybrid working model - Macbook Pro provided for work - Bonus: 13th month salary - Weekly learning sessions and offsites - Close-knit small team culture
Chat/Apply
