Principal Applied Scientist @Robots and Pencils
Artificial Intelligence
Salary $200,000 - $300..
Remote Location
πŸ‡ΊπŸ‡Έ USA Only
Employment Type full-time
Posted 2d ago

[Hiring] Principal Applied Scientist @Robots and Pencils

2d ago - Robots and Pencils is hiring a remote Principal Applied Scientist. πŸ’Έ Salary: $200,000 - $300,000 per year πŸ“Location: USA

Role Description

We're looking for a Principal Applied Scientist to lead the design and build of an evaluation framework for an enterprise planner-based agent. You will work directly with the client's engineering team to benchmark their agent across planner models, executor LLMs and inference optimization levers. Every proposed change, whether a model upgrade, a harness change or a routing strategy, becomes a controlled, statistically sound experiment with a measurable dollar outcome.

This is a research-heavy, customer-facing role with a significant build component. You will own both the experimental method and the tooling that makes it repeatable:

  • A scenario-driven user simulator
  • Isolated experiment execution
  • Statistical comparison
  • Automated insights in a locally hosted dashboard the client can adopt as their own

The work runs in three overlapping streams:

  • Evaluation pipeline
  • Grading and statistics
  • Dashboard and experiment campaign

It ends in a recommendation the client can act on.

What You'll Do

Evaluation Pipeline & Simulation

  • Design the end-to-end evaluation workflow for a planner-executor agent: scenarios, user simulation, configuration, execution, grading, comparison and insights
  • Define the scenario schema, grading spec and validators with the client's team, who own scenario creation
  • Build an LLM-based user simulator with a hidden goal and a stop rule that holds realistic multi-turn conversations
  • Build isolated, parallel experiment execution: config files, one run per config and scenario, and a harness adapter
  • Package optimization levers as plug-ins and implement a shortlist of 3 to 4 for v1
  • Tag every model call at the gateway and send traces to a trace store

Grading, Statistics & Insights

  • Grade each run against the expected end state as the primary signal
  • Compute the core metrics: verified success, cost per success, p95 latency, and failures traced to pipeline stage
  • Compare experiments on the same scenarios with paired methods
  • Convert differences into projected dollar savings at production volume
  • Build an automated insight engine where a statistics layer computes every number

Dashboard, Experiments & Readout

  • Build the locally hosted dashboard (Launch, Live, Results, Compare, Insights and Traces views)
  • Run the staged campaign: production baseline, model screen, lever screen, then confirm the best combination
  • Analyze generalization by scenario category
  • Deliver a final recommendation with confidence intervals and dollar impact

Collaboration & Technical Leadership

  • Partner with the client's engineers and leadership to agree on scenarios, margins and success criteria
  • Write clear technical reports and leadership summaries
  • Set the experimental standards for the pod and mentor other engineers

Qualifications

  • Experience with LLM and agent evaluation methods
  • Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production
  • A strong background in experimental design and statistical inference
  • Experience with multi-objective optimization or cost-aware ML
  • Familiarity with LLM routing, model cascades or model selection
  • Experience with observability and tracing for LLM systems
  • The ability to build production-quality research tooling
  • Prior customer-facing or consulting experience
  • The ability to explain technical findings clearly to non-technical audiences

Helpful Extras and Unique Skills

  • Publications or open-source contributions in LLM evaluation, agents or efficient inference
  • Experience building internal dashboards or web UIs
  • Hands-on use of an LLM gateway or tracing stack
  • Experience in a regulated or high-stakes enterprise domain
  • A point of view on cost-quality tradeoffs gained from running them in production

Salary

Our salary range is $200,000 - $300,000.

Before You Apply
️
πŸ‡ΊπŸ‡Έ Be aware of the location restriction for this remote position: USA Only
β€Ό Beware of scams! When applying for jobs, you should NEVER have to pay anything. Learn more.
Principal Applied Scientist @Robots and Pencils
Artificial Intelligence
Salary $200,000 - $300..
Remote Location
πŸ‡ΊπŸ‡Έ USA Only
Employment Type full-time
Posted 2d ago
Apply for this position
Did not apply βœ“
Applied βœ“
Sent Follow-Up βœ“
Interview Scheduled βœ“
Interview Completed βœ“
Offer Accepted βœ“
Offer Declined βœ“
Application Denied βœ“
Unlock 130,000+ Remote Jobs
️
πŸ‡ΊπŸ‡Έ Be aware of the location restriction for this remote position: USA Only
β€Ό Beware of scams! When applying for jobs, you should NEVER have to pay anything. Learn more.
Apply for this position
Did not apply βœ“
Applied βœ“
Sent Follow-Up βœ“
Interview Scheduled βœ“
Interview Completed βœ“
Offer Accepted βœ“
Offer Declined βœ“
Application Denied βœ“
Unlock 130,000+ Remote Jobs
Γ—
Apply to the best remote jobs
before everyone else

Access 130,000+ vetted remote jobs and get daily alerts.

4.9 β˜…β˜…β˜…β˜…β˜… from 500+ reviews

⚑ 133,663+ remote jobs, refreshed hourly

πŸ”” Real-time alerts: Apply first, direct to employer

πŸ›‘οΈ Vetted companies, no scams, true remote only

Unlock All Jobs Now

Maybe later