[Hiring] Staff Applied Scientist @Robots and Pencils

Role Description

We're looking for a Staff Applied Scientist to lead the design and build of an evaluation framework for an enterprise planner-based agent. You will work directly with the client's engineering team to benchmark their agent across planner models, executor LLMs, and inference optimization levers. Every proposed change becomes a controlled, statistically sound experiment with a measurable dollar outcome.

This is a research-heavy, customer-facing role with a significant build component. You will own both the experimental method and the tooling that makes it repeatable:

• A scenario-driven user simulator
• Isolated experiment execution
• Statistical comparison
• Automated insights in a locally hosted dashboard the client can adopt as their own

The work runs in three overlapping streams:

• Evaluation pipeline
• Grading and statistics
• Dashboard and experiment campaign

The role ends in a recommendation the client can act on.

What You'll Do

• Evaluation Pipeline & Simulation
  • Design the end-to-end evaluation workflow for a planner-executor agent: scenarios, user simulation, configuration, execution, grading, comparison, and insights
  • Define the scenario schema, grading spec, and validators with the client's team
  • Build an LLM-based user simulator with a hidden goal and a stop rule
  • Build isolated, parallel experiment execution
  • Package optimization levers as plug-ins and implement a shortlist of 3 to 4 for v1
  • Tag every model call at the gateway and send traces to a trace store
• Grading, Statistics & Insights
  • Grade each run against the expected end state
  • Compute the core metrics: verified success, cost per success, p95 latency, and failures traced to pipeline stage
  • Compare experiments on the same scenarios with paired methods
  • Convert differences into projected dollar savings at production volume
  • Build an automated insight engine
• Dashboard, Experiments & Readout
  • Build the locally hosted dashboard
  • Run the staged campaign
  • Analyze generalization by scenario category
  • Deliver a final recommendation with confidence intervals and dollar impact
• Collaboration & Technical Leadership
  • Partner with the client's engineers and leadership
  • Write clear technical reports and leadership summaries
  • Set the experimental standards for the pod and mentor other engineers

What You'll Bring

• Experience with LLM and agent evaluation methods
• Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production
• A strong background in experimental design and statistical inference
• Experience with multi-objective optimization or cost-aware ML
• Familiarity with LLM routing, model cascades, or model selection
• Experience with observability and tracing for LLM systems
• The ability to build production-quality research tooling
• Prior customer-facing or consulting experience
• The ability to explain technical findings clearly to non-technical audiences

Helpful Extras and Unique Skills

• Publications or open-source contributions in LLM evaluation, agents, or efficient inference
• Experience building internal dashboards or web UIs
• Hands-on use of an LLM gateway or tracing stack
• Experience in a regulated or high-stakes enterprise domain
• A point of view on cost-quality tradeoffs gained from running them in production

Salary

Our salary range is $200,000 - $300,000