[Hiring] Annotation Program Manager @Gather AI

Role Description

We are looking for an experienced Annotation Program Manager to own and scale our production annotation operations supporting computer vision and machine learning initiatives. This role is responsible for managing external annotation partners, improving annotation quality, maintaining documentation, and ensuring high-quality labeled datasets are delivered on schedule to support model development.

You will work closely with Machine Learning Engineers, Data Engineers, and Operations teams while serving as the primary owner of our annotation programs and vendor relationship. Success in this role requires a combination of technical problem-solving, operational excellence, and strong project management.

What You'll Do

• Own day-to-day management of our annotation vendor relationship, including workload planning, throughput monitoring, and SLA management.
• Coordinate annotation batches, scheduling, and delivery timelines across multiple production programs.
• Monitor annotation quality and continuously improve labeling accuracy through structured QA processes.
• Develop, maintain, and improve annotation SOPs, taxonomies, and documentation.
• Build and maintain Python-based workflows for annotation data preparation, API integrations, reporting, and validation.
• Partner with ML teams to prioritize annotation work that maximizes model performance.
• Track work across Jira and communicate project status, risks, and capacity needs to stakeholders.
• Drive operational improvements that reduce annotation turnaround time while maintaining high quality.

First 90 Days

• Own Annotation Operations: Take over day-to-day coordination with CloudFactory from existing program owners.
• Own weekly turnaround time (TAT) reporting, batch scheduling, and vendor escalation management.
• Become the primary operational contact for annotation delivery.
• Become Fluent in Existing Annotation Programs: Develop working expertise in both active production annotation programs:
• MHEV forklift event labeling (PICK_UP / DROP_OFF) using the internal MHEV annotation application.
• Drone Human-in-the-Loop (HIL) annotation pipeline supporting case counting and empty detection through the CloudFactory API.
• Improve Annotation Quality: Audit completed annotation batches across both programs.
• Run inter-annotator agreement (IAA) analysis against gold-set datasets.
• Identify common annotation errors and present recommendations to ML leads.
• Lead calibration sessions with the annotation vendor as needed.
• Own Documentation: Take ownership of:
• MHEV Event Annotation SOP.
• Barcode + Location Root Cause Analysis (RCA) SOP.
• Responsibilities include:
• Version control.
• Closing documentation gaps.
• Adding edge-case guidance.
• Ensuring all annotators are using current documentation.

Required Technical Skills

• Annotation vendor management, including onboarding, batch management, SLA tracking, and calibration sessions.
• Python scripting for data preparation, automation, reporting, validation, and API integrations.
• Writing annotation SOPs, taxonomies, and detailed labeling guidelines.
• Annotation quality assurance using inter-annotator agreement (IAA), gold-set validation, and structured quality programs.
• Cloud storage and data handling using Azure Blob Storage or AWS S3, along with JSON and CSV data formats.
• Strong Jira project management and task organization across multiple concurrent annotation programs.

Nice-to-Have Skills

• Computer vision annotation experience including video event labeling, bounding boxes, polygon segmentation, or similar workflows.
• Experience with annotation platforms such as Labelbox, Scale AI, CVAT, Hasty, or V7 Darwin.
• Familiarity with active learning or ML-assisted annotation prioritization.

Qualifications

• Required:
• 3–6 years of experience in data operations, annotation program management, ML data engineering, or a closely related role.
• Experience managing an external annotation vendor operating under production SLAs.
• Strong Python programming skills for data preparation and API integrations.
• Experience authoring annotation guidelines or standard operating procedures.
• Understanding of data privacy and confidentiality requirements when working with customer operational data.
• Preferred:
• Experience with CloudFactory.
• Experience in warehouse automation, robotics, or computer vision applications.
• Experience with additional annotation platforms and APIs.

Required Technologies

• Python
• REST APIs (CloudFactory API or equivalent annotation platform APIs)
• Azure Blob Storage or AWS S3
• Jira

Success Traits

• Highly organized and able to manage multiple concurrent workstreams.
• Collaborative, respectful, and an excellent cross-functional partner.
• Detail-oriented with a strong focus on annotation quality.
• Resourceful and comfortable building processes where they don't yet exist.
• Persistent in driving vendor accountability and meeting SLAs.
• Comfortable managing external teams and influencing without direct authority.
• Analytical, with the ability to identify root causes of annotation quality issues and implement lasting improvements.