[Hiring] Research Lead, Evaluations and Benchmarks @Alice

Role Description

You ship a benchmark every two to three weeks. Each one measures a frontier risk that nobody has measured yet. Some go public, while others go only to the labs. Some benchmarks and papers are done in collaboration with leading AI Labs and universities.

You will not write every eval yourself. Each benchmark pairs you with an in-house researcher who owns that harm area, and you get a budget for freelancers you direct. You own the taxonomy, the harness, the quality bar, and the release.

The seat sits in the CTO office alongside the research lead who sets our public research agenda. Around 150 researchers here work on harms directly, and you can pull any of them onto a subject.

Key Responsibilities

• Ship the benchmark cadence: A benchmark every two to three weeks.
• Own the quality bar: Ensure the test is simple and verifiers hold.
• Run the process: Hold the plan and the calendar, keeping other researchers on timeline.
• Set the roadmap with the forum: Sit with the CTO and the pod and research leads monthly.
• Stay ahead of the curve: Spend around 20% of your time on the ecosystem.

Qualifications

• PhD or Masters in computer science, machine learning, or a related field, or equivalent depth from industry research.
• 3+ years building and running safety or security evaluations for language models in production.
• 5+ relevant research publications in the field of AI safety and security, including lead author on at least 2 of them.
• Strong engineering skills: Evaluation harnesses, distributed inference, vLLM, reading and fixing a codebase.
• Ability to build a taxonomy and direct researchers and freelancers without formal management.
• Strong English, written and spoken, for effective internal communication across time zones.
• Curiosity about the harms themselves, learning a new subject every three weeks.

Requirements

• Post-training experience: SFT, DPO, GRPO.
• Agentic evaluation experience: tool use, orchestration, permissions, prompt injection.
• Publications at top conferences.
• Willingness to present your own work on client calls.
• Strong communication skills, both verbal and written, with the ability to present to large and/or senior audiences.
• Travel to conferences at least 3 times a year.

Benefits

• Opportunity to work with leading AI Labs and universities.
• Collaborative environment with a focus on cutting-edge research.
• Engagement in meaningful work that impacts the AI landscape.

Company Description

Alice is a trust, safety, and security company built for the AI era. We safeguard the communicative technologies people use to create, collaborate, and interact—whether with each other or with machines.

• Alice clients are the top 8 AI Labs in the world.
• We provide end-to-end coverage across the entire AI lifecycle.
• Alice is widely considered a global leader in online safety and AI security.
• We safeguard over 3 billion users across the largest AI and tech platforms.

If you're creative and driven to secure the future of AI, we want to hear from you!