[Hiring] Intelligence Analyst @YO AI Labs

Role Description

We are seeking experienced Intelligence Analysts to contribute to a project focused on evaluating and improving next-generation AI systems. In this role, you will apply your intelligence analysis expertise to assess AI-generated outputs, evaluate analytical reasoning, and help strengthen the accuracy, reliability, and consistency of AI models. Prior experience in AI is not required; relevant analytical expertise is the primary requirement.

• Develop and map analytical cases to established taxonomies across multiple evaluation areas, ensuring comprehensive coverage of relevant intelligence scenarios.
• Create complex test cases to evaluate model performance in source evaluation, analytical rigor, and the expression of uncertainty.
• Assess and grade AI-generated responses based on reasoning quality, source use, recognition of limitations, and practical utility.
• Identify instances where legitimate analytical exploration is unnecessarily refused by the model, providing clear documentation and actionable feedback.
• Document edge cases, analytical disagreements, and ambiguous scenarios to support continuous improvement of evaluation methodologies.
• Collaborate with project leads to refine procedures for source evaluation, confidence-level assessment, and analytical review.
• Provide insights to help calibrate analytical standards and improve the overall quality and consistency of AI evaluation benchmarks.

Qualifications

• Demonstrated all-source intelligence analysis experience or deep expertise in a specialized analytical discipline, ideally within a civilian Intelligence Community (IC) environment.
• Strong proficiency in source evaluation, confidence-level assessment, and established intelligence analysis standards.
• Experience teaching, mentoring, or leading analytical reviews, with a proven ability to promote analytical rigor.
• Excellent written and verbal communication skills, with a strong emphasis on clarity, precision, and evidence-based reasoning.
• Ability to identify and explain nuanced model limitations, including cases of unnecessary refusal.
• Familiarity with structured analytical techniques and intelligence tradecraft standards.
• Previous experience in AI evaluation, model assessment, or related quality assurance initiatives is an advantage.

Ideal Candidate Profile

The ideal candidate combines strong intelligence analysis expertise with sound judgment, critical thinking, and attention to detail. They can evaluate complex analytical outputs, assess source credibility, communicate uncertainty effectively, and identify gaps in reasoning. Their feedback will help improve the reliability, consistency, and analytical quality of AI-generated intelligence assessments.