[Hiring] ML Annotation QA Engineer @Gather AI

Role Description

We are looking for an ML Annotation QA Engineer to own the quality of annotated data across our computer vision and machine learning programs. This role is responsible for the judgment-heavy analysis that cannot be reliably outsourced, for the decision rules behind it, and for turning annotation output into an ongoing read on how our systems are actually performing in the field.

We work with an external annotation partner at production volume, and that continues. What we need in house is someone who can:

• Analyse the annotated data.
• Make and defend the calls the vendor cannot make consistently.
• Build the aggregate view that shows which facilities and equipment are degrading and why.

Success in this role requires a combination of analytical rigor, sound judgment under ambiguity, and clear written communication.

What You’ll Do

• Own the judgment-heavy quality analysis on annotated data that cannot be reliably outsourced.
• Own, version, and refine the verdict taxonomy, decision rules, and quality guidelines for the categories you cover.
• Build and maintain performance trackers over annotated data.
• Detect anomalies against that baseline and flag them the day they appear.
• Run root cause analysis on flagged anomalies.
• Report findings to engineering and ML with reproducible evidence.
• Identify systematic failure patterns and maintain a documented pattern library.
• Query and analyse annotation data directly with Python and SQL.
• Feed annotation-quality findings back as concrete SOP and instruction changes.
• Specify annotation tool improvements.
• Stand up quality analysis and reporting for new annotation programs.
• Track work across Jira and contribute to pre-release validation.

First 90 Days

• Own the Quality Analysis.
• Become Fluent in the Annotation Pipeline.
• Build the Performance Tracker.
• Own Documentation.

Responsibilities

• Version control.
• Closing documentation gaps.
• Adding edge-case guidance.
• Feeding changes back to the annotation vendor’s SOPs.

Qualifications

• BS in Computer Science/Engineering, Electrical Engineering, or equivalent experience.
• 2–5 years of experience in ML QA, annotation quality, data quality, or analytics at an AI/ML company.
• Hands-on experience with annotated ML datasets, including assessing label quality.
• Demonstrated experience finding, diagnosing, and reporting data anomalies to a technical audience.
• Able to get to a defensible answer from unfamiliar data without someone preparing it first.
• Understanding of data privacy and confidentiality requirements.

Required Technical Skills

• Python.
• SQL, or an equivalent query language.
• Jira.

Success Traits

• Analytical, able to tell a real pattern from noise.
• Detail-oriented.
• Comfortable with ambiguity.
• Persistent in chasing a cause across sites, programs, and builds.
• Resourceful and comfortable building processes and reports.
• A strong written communicator.
• Collaborative, respectful, and an excellent cross-functional partner.

What Success Looks Like

• Established yourself as the primary owner of annotation quality analysis.
• A performance tracker the team relies on.
• Closed the reporting cycle to same day.
• Standardized the verdict taxonomy and decision rules.
• Built a documented pattern library of known failure modes.
• Improved the annotation tool in at least one way.
• Given ML and engineering a clearer picture of error patterns.
• Created a repeatable method for standing up annotation quality on the next program.