[Hiring] Model Test and Measurement Engineer @DEFCON AI

Role Description

You'll join the analytics and AI engineering team behind an AI-assisted platform that:

• Brings together records from dozens of disparate data sources
• Resolves them to the correct individual
• Highlights what analysts should review first
• Provides transparent, explainable recommendations that users can trust

As a Model Test & Measurement Engineer, you'll own the evaluation framework that helps ensure those systems perform as intended. Your responsibilities include:

• Building and maintaining labeled ground truth datasets
• Designing statistically sound audit and sampling methodologies
• Measuring model performance across releases
• Creating evidence packages that support deployment decisions
• Independently validating both scoring and generative AI capabilities
• Working closely with data scientists, AI engineers, and technical leadership while maintaining independence

This is a fully remote role with occasional travel to DEFCON AI headquarters, customer sites, and partner facilities as needed.

Qualifications

• 5+ years with model validation as a named responsibility, not a side task
• Direct experience with ground-truth construction and sampling design
• Strong Python and SQL; comfort working independently from the teams whose models you evaluate
• US Citizenship Required
• Active US Secret clearance
• Elevated personnel security requirements apply to portions of this work and are discussed during screening

Requirements

• Regulated-industry or government model-risk background
• Experience with fairness / disparate-effect testing and human-oversight documentation
• NIST AI RMF or comparable practice
• Active Top Secret clearance

Benefits

• A fully remote, results-based environment
• Competitive salary, bonus, and equity package
• 100% employer paid, comprehensive health insurance including medical, dental, and vision for you and your family
• Unlimited PTO, with your manager's approval
• Flexible work environment where you manage your work day
• 14 weeks of fully-paid parental leave
• Salary Range: $150,000–$190,000 based on experience, skills, and other factors