[Hiring] Lead Engineer, AI Quality @Circle

Role Description

The AI Quality engineering team at Circle owns the foundation for measuring, diagnosing, and improving the quality of Circle's AI-powered features. This team focuses on building the infrastructure to measure, diagnose, and improve production AI systems, rather than ML research or model training.

We're looking for a Lead Engineer to help us build out the evaluation frameworks, observability tooling, and diagnostic infrastructure that tell us whether our AI Agents are working well, where to improve them, and how to make them faster and more cost-efficient. This is a hands-on player-coach role. You’ll also lead and manage the AI Quality engineering team, with an ambitious and growing roadmap.

If you're excited about making AI systems work reliably, efficiently, and at scale, this is for you.

What you'll be doing

• Build and own our evaluation infrastructure.
• Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation.
• Diagnose exactly where quality breaks down.
• Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for AI core engineering can help.
• Grow the datasets that make evaluation possible.
• Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone.
• Run structured experiments across prompts, models, and the agent harness.
• Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity.
• Lead and grow the AI Quality engineering team.
• Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself.
• Partner closely with AI Core engineering.

Qualifications

• 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product.
• Experience with complex, tool-using systems (multiple tools, planning or orchestration, sub-agents) not just simple, single-turn assistants.
• Comfortable in Ruby on Rails / Python or ready to pick them up quickly.
• Experience building evaluation or observability infrastructure for ML/AI systems.
• Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Learn fast, experiment aggressively, and thrive in a highly dynamic environment.
• Comfortable in a fast-paced environment with ambiguity.
• Strong alignment with our company values.
• Proficient in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.

Requirements

• Experience with eval pipelines, scorers, dashboards, or CI/CD for evals.
• Knowledge of evaluation frameworks like Braintrust, LangSmith, or similar.
• Ability to turn raw examples into a dataset through human annotation, AI-generated data, or simulation.

Benefits

• U.S.-benchmarked compensation globally.
• Equity in the company with ongoing refresh grants.
• 35 days of paid time off each year.
• Remote-only team with company retreats in incredible destinations twice a year.
• Benefits package that supports health, wellbeing, and professional growth.