[Hiring] Senior AI/ML Test and Evaluation Engineer @OpenTeams

Role Description

We're looking for a Senior AI/ML Test and Evaluation Engineer to build and operate the benchmarking and evaluation capability at the core of an AI platform. This is a role for someone who is more interested in what a model gets wrong than in what it gets right.

• Build the evaluation harnesses — automated metrics paired with structured human expert judgment, applied to candidate models and to the agentic workflows built on top of them.
• Develop repeatable methodologies for comparing performance against current operational baselines.
• Document the limitations and surface the failure modes that matter.
• Prepare reports for senior stakeholders to inform decisions about which capabilities are ready to field.
• This is hands-on engineering on an open-source toolchain.

Key Responsibilities

• Design and implement a platform for test & evaluation of AI models and Agentic systems.
• Develop evaluation methodologies combining human and AI expert judging and multiple input and output modalities.
• Design and enhance user interfaces for intuitive workflow expression.
• Collaborate closely with subject matter experts, Model/Agent developers, and evaluation designers.
• Architect and implement robust experiment provenance and result tracking systems.
• Ensure performance across integrated tools and scaling systems.
• Provide technical mentorship, code review, design guidance, and clear documentation.

Qualifications

• Proven experience building and deploying software platforms with complex integration surface, preferably handling Machine Learning or agentic workloads.
• Deep experience using Python, including API frameworks and agentic harnesses.
• High level knowledge of Computer vision, language modeling, and/or task specific agent evaluation.
• Hands-on experience with common frameworks and tooling, e.g., PyTorch and Hugging Face ecosystem.
• Strong communication skills and experience collaborating effectively with cross-functional teams.
• Demonstrated experience or potential for leadership, especially in making architectural decisions.

Nice to Have

• Currently hold or eligible to hold U.S. Security Clearance (Secret or higher).
• Prior experience designing test and evaluation systems for AI.
• Experience deploying complex software systems in IL 4 or higher environments.
• Experience participating in security assessment and authorization, hardening systems, and remediating vulnerabilities.
• Previous experience contributing to open source projects and participating in open source communities.
• Exposure to the intelligence community or department of defense subject matter experts.

Benefits

• 100% employer paid medical premiums for employees.
• Self-managed PTO with a minimum time off requirement.
• Opportunities to collaborate with global experts and contribute to open source projects.
• Commitment to diversity, equity, inclusion, and belonging.