[Hiring] Staff Applied Scientist - AI Evaluation & Trust @Sayari

Role Description

Sayari builds AI systems for high-consequence analytical work where being "wrong" carries real-world weight. We are looking for a Staff or Principal Applied Scientist to join our AI Innovation Group as the trusted expert on AI Evaluation and Trust. You will own the "Judgment Layer" of our system:

• Building the specialized judge models, statistical benchmarks, and multi-turn frameworks that ensure our agents act with the high bar of trustworthiness required by our national security and enterprise customers.

Qualifications

• 10+ years of Machine Learning experience with a focus on Deep Neural Network activities, evaluating model performance & trust.
• 1-2+ years’ experience focused on post-training activities.
• 1+ year experience creating benchmarks to evaluate LLMs.

Requirements

• Deep expertise in LLM-as-judge architectures, multi-turn evaluation, and Reinforcement Learning (RL/RLHF/RLAIF).
• Mastery of statistics and experimental design, including significance testing, distribution analysis, and inter-rater reliability.
• Experience with Mixture-of-Experts (MoE) systems, routing behavior, and expert specialization.
• Proven ability to own the path from data collection to production deployment; we are a small team and every role is "hands-on."
• Understanding of Graph RAG and the unique challenges of evaluating non-deterministic, agentic workflows.

Preferred

• Experience building, fine-tuning (LoRA, etc.), or pre-training models specifically for judgment, preference modeling, or classification tasks.
• Background in cognitive science, intelligence community tradecraft, or research literature on expert judgment under uncertainty.
• Experience building or managing large-scale annotation infrastructure and quality assurance protocols.
• A record of published research or recognized work in preference modeling or AI alignment.

Benefits

• 100% fully paid medical, vision, and dental for employees and their dependents.
• Generous time off; we observe all US federal holidays, close our office for a winter break (12/24-12/31), in addition to granting 18 PTO days and 10 sick days.
• Outstanding compensation package; competitive commissions for revenue roles and bonuses for non-revenue positions.
• A strong commitment to diversity, equity, and inclusion.
• Eligibility to participate in additional benefits such as 401k match up to 5%, 100% paid life insurance (up to $100,000 coverage), and parental leave.
• A collaborative and positive culture - your team will be as smart and driven as you.
• Limitless growth and learning opportunities.

Pay Range

$195,000 — $205,000 USD