[Hiring] AI Research Engineer @Dscout

Role Description

At Dscout, we’re building the most flexible and powerful UX research platform on the market—trusted by the world’s top brands in finance, healthcare, consumer goods, and tech. Our tools help teams deeply understand the humans behind their products, so they can build better ones. We are expanding our smart and driven team and would love for you to join us.

Dscout AI Training & Evals

Dscout AI Training & Evals builds human data and evals that show whether AI works for real people. For ten years Dscout has captured how people behave in real context, with consent and scientific rigor. AI Training & Evals points that engine at AI: consented human behavior and judgment, turned into the training data, evals, and benchmarks AI teams trust.

• Capability has outrun trust.
• Models ace benchmarks and still surprise people in the wild.
• A benchmark says how capable a model is; it does not say whether the model works for the person using it.
• We serve AI researchers, product builders, and research teams.
• We are a new unit inside an established company with live customers and a mature platform.
• We work AI-native.

Why this seat

• Work on a problem the field has not solved: measuring whether AI works for real people, over real time.
• Study something most evals do not have: real participants, doing real tasks in their own context.
• Own the benchmark; you are founding it.
• Publish methods and benchmarks to the research community.
• Join a small senior team with direct access to the Head of AI Research and the GM.

Job Description

This is a research-first role focused on questions where the right rubric, benchmark, or method does not yet exist. You will find the gaps, propose bold directions, and take projects from first hypothesis through experiments and benchmark construction to publication.

• Design rubrics and test validity, reliability, and sensitivity.
• Measure reliability of human raters and identify noise versus real signal.
• Calibrate AI judges against real human judgment.
• Build quality signals that put evidence behind every data point.
• Design Dscout’s first benchmark: real users, in real context, over long horizons.
• Define protocol, task design, scoring, and release.
• Keep a living benchmark honest over time.
• Write the paper and own the public release.
• Study when real-world human feedback improves a model.
• Explore how preference and judgment data differ from expert data.
• Propose new programs and co-author studies with outside labs.

A good first year

• A QA and rubric framework running on live customer programs.
• Benchmark v0 released, with a paper submitted to a top venue.
• At least one external team using or citing our work.
• Delivery colleagues using your methods independently.

Qualifications

• Published research or strong preprints in evals, preference learning, or human-AI interaction.
• A PhD, a PhD in progress, or an MS with strong research internships.
• Strong applied statistics skills.
• Hands-on experience with LLM-as-judge and rubric-based evaluation.
• Strong Python skills for building research prototypes.
• Experience running experiments on messy human data.
• Clear technical writing skills.
• Based in the Bay Area.

Requirements

• A strong publication record is preferred.
• Ability to find important questions and design rigorous answers.

Benefits

• Compensation: $250-400K OTE + Equity.
• Built-in bonus and equity program.
• Publication freedom and conference budget.
• Flexible PTO and 15 holidays.
• 12 weeks of paid parental leave.
• 401k match and education stipend.

Company Description

Dscout is a team of passionate, empathetic, and curious professionals. We are at the cutting edge of experience research technology, prioritizing learning, sharing, and building. We value diversity and inclusivity and seek team members who align with that belief.