[Hiring] Queue Manager / AI Data Quality Lead @Prolific Academic Ltd

Role Description

We are seeking a Queue Manager / AI Data Quality Lead to join a high-priority initiative in partnership with a leading AI company. This is an operational, hands-on role — you will own day-to-day execution for one or more annotation queues on a live AI training/eval program:

• Managing throughput
• Running quality calibrations
• Tracking contributor performance
• Building out dashboards/reporting to keep stakeholders informed in real time

We're looking for people who have actually run queues and quality on AI data programs before — not people who've managed generic ops queues and are pivoting into AI. You should be able to speak to specific projects, specific calibration processes you've built, and specific contributor performance systems you've run.

For this opening, we specifically want candidates with a trust & safety / integrity background — someone who can tell when responses are shallow, low-effort, or phoned-in, and who keeps queues healthy on quality, not just catches bad actors.

What "good" looks like for us:

• A quality analyst who has scaled up to running a full AI data program end-to-end — owning calibrations and contributor performance management at program level, not just task level.
• A queue manager who also operates as an AI training consultant — someone comfortable advising on process/methodology, not just executing it.

Qualifications

• Proven, direct experience running queues and quality on AI data programs — ideally at a competitor (Scale AI, Surge, Appen, Sama, Invisible, Mercor, Alignerr, Handshake AI, iMerit, CloudFactory, Toloka, or similar)
• 3+ years in high-throughput ops (data labeling, trust & safety, BPO, content ops, CX ops), with recent, specific AI data ops experience
• Trust & safety / integrity background required
• Able to identify shallow, low-effort, or phoned-in responses
• Proven ability to design and run calibration processes from scratch, not just follow existing ones
• Track record building or maintaining dashboards/reporting (spreadsheets required; SQL/BI/Looker/Tableau a strong plus)
• Experience managing contributor performance at scale — including retention/removal decisions
• Comfortable with ambiguity and fast-changing priorities on live programs

Requirements

• Manage work queue: intake, triage, prioritization, task assignment, and SLA delivery
• Monitor task queue and backlog control
• Identify and troubleshoot bottlenecks (unclear policy, tooling issues, supply gaps)
• Perform QA: make corrections, provide annotator feedback, flag systematic confusion to Client Delivery Lead
• Monitor quality at project and individual annotator/contributor level
• Design and run calibration sessions, drive guideline adherence, reduce rework
• Manage annotator/contributor performance: unassign underperformers, identify and retain high-performers
• Build and maintain dashboards to track throughput, disagreement rates, defect/rework rates, and contributor performance
• Turn raw queue data into clear, actionable reporting for stakeholders
• Provide feedback for process improvements based on data trends
• Train new annotators/contributors on project guidelines
• Conduct live or recorded webinars as participants join a project
• Host "office hours" for participants
• Oversee dedicated Slack channels: answer questions, provide support, escalate complex issues to Client Delivery Lead

Benefits

• Competitive pay rates
• Flexible hours
• Ability to work from home