[Hiring] AI Data Quality Analyst @TaskUs

Role Description

As AI-driven solutions expand across industries, ensuring the fidelity of training and evaluation datasets is essential for building reliable models. TaskUs needs a diligent, detail-focused Data Quality Analyst who can maintain annotation standards and drive continuous improvements so that essential AI data remains accurate, efficient, and scalable.

Data Analysis

• Quality Audits - Perform quality audits on annotated datasets to ensure that they meet established guidelines and quality benchmarks.
• Statistical Reporting - Leverage statistical based quality metrics such as F1 score and inter-annotator agreement to evaluate data quality.
• Root Cause Analysis - Analyze annotation errors, trends, project processes, and project documentation to identify and understand the root cause of errors and propose remediation strategies.
• Edge-Case Management - Resolve and analyze edge-case annotations to ensure quality and identify areas for improvement.
• Tooling - Become proficient in using annotation and quality control tools to perform reviews and track quality metrics.
• Guidelines - Become an expert in the project specific guidelines and provide feedback for potential clarifications or improvements.

Continuous Improvement

• Automation - Identify opportunities to use automation to help enhance analytics, provide deeper insights, and improve efficiency.
• Documentation - Develop and maintain up-to-date documentation on quality standards, annotation guidelines, and quality control procedures.
• Feedback - Provide regular feedback that identifies areas for improvement across the annotation pipeline.

Collaboration & Communication

• Cross-Functional Teamwork - Work closely with key project stakeholders and clients to understand project requirements and improve annotation pipelines.
• Training - Assist with training annotators, providing guidance, feedback, and support to ensure data quality.
• Reporting - Provide regular updates that highlight data quality metrics, key findings, and actionable insights for continuous process improvements.

Qualifications

• 1+ years of experience as a data analyst with exposure to data quality and/or data annotation - ideally within an AI/ML context.
• Familiarity with the basic concepts of AI/ML pipelines and data.

Requirements

• Strong analytical and problem-solving skills with an exceptional eye for detail.
• Excellent written and verbal communication skills, with the ability to clearly articulate quality issues and collaborate with diverse teams.
• Ability to work independently and manage time effectively to meet deadlines.
• A strong problem-solver who thinks critically and drives innovation and continuous optimization.
• A quick learner with the ability to work independently in a fast-paced environment.
• A strong focus on detail, balanced against strategic priorities.
• A positive can-do attitude and the ability to easily adapt to new environments.
• Not afraid to speak up.

Nice to have

• Familiarity with data annotation tools (e.g. Labelbox, Dataloop, LabelStudio etc.).
• Experience working with multi-modal AI/ML datasets (images, videos, text, audio).
• Prior experience in an agile or fast-paced tech environment with exposure to AI/ML pipelines.
• Knowledge of programming languages (e.g. Python).
• Knowledge of the concepts and principles of data quality for AI/ML models and the impacts it can have on model performance.
• Working understanding of common quality metrics and statistical methods used in AI/ML data quality.
• Knowledge of AI/ML concepts and experience with data for AI/ML models.
• Experience in prompt engineering and leveraging LLMs in your day-to-day work.

Benefits

• Work remote: Work from anywhere you’re most productive!
• Every dataset is unique: Audit, analyze, and improve complex AI/ML datasets, no two projects are ever the same.
• Visible impact: Your quality checks, root cause analyses, and automation ideas will directly shape how AI models perform in real-world applications.
• Room to grow: Build deep expertise in AI data quality and scale your influence as our projects and capabilities expand.
• Mission-driven team: Work alongside analysts, engineers, and stakeholders who share a passion for creating accurate, reliable, and trustworthy AI.

The impact you’ll make

• Guard data integrity through audits: Conduct quality audits on annotated datasets against established guidelines and statistical benchmarks (e.g., F1 score, inter‑annotator agreement) to uphold data reliability.
• Pinpoint and resolve annotation issues: Analyze error patterns, edge-case annotations, and root causes to surface insights and elevate annotation quality.
• Optimize annotation processes: Become proficient in annotation and QA tools, propose automation, and maintain clear documentation of quality standards, guidelines, and procedures.
• Collaborate and coach: Work closely with stakeholders and annotators to clarify requirements, provide constructive feedback, and support skill development within the annotation pipeline.
• Track and communicate quality performance: Regularly report on data quality metrics, key findings, and actionable insights to drive continuous improvements across projects.