[Hiring] Data Annotation Specialist, Generalist @Cohere

Role Description

We are on a mission to build machines that understand the world and make them safely accessible to all. Data quality is foundational to this process. Machines (or Large Language Models, to be exact) learn in similar ways to humans, by way of feedback. By labelling, ranking, auditing, and correcting model output, you will improve Large Language Models' performance for iterations to come, thus having a lasting impact on Cohere's technology. We are hiring Generalist professionals with broad backgrounds that span multiple consumer-facing or personal domains.

This is a judgment-driven role, not passive data entry. You will:

• Evaluate and rank model outputs:
• Complete preference and comparison tasks, assessing which responses best conform to project guidelines for accuracy, helpfulness, tone, and safety, and writing clear justifications for your judgments.
• Stress-test and break models:
• Probe models adversarially to surface failure modes, unsafe behavior, and capability gaps, and document reproducible cases that engineering and research teams can act on.
• Create datasets:
• Author high-quality prompts, responses, and exemplars to build training and evaluation datasets, following detailed specifications and editing machine-written or human-written outputs to standard.
• Build and apply rubrics and taxonomies:
• Contribute to the design of grading criteria and rubrics, then apply them consistently to produce structured, high-quality annotations across task types.
• Annotate and correct multimodal data:
• Label, audit, and rectify inaccuracies across text, image, and structured data, maintaining a high standard of data integrity and accuracy.
• Calibrate and maintain consistency:
• Participate in calibration exercises and inter-annotator agreement checks to align on standards, and flag ambiguous or uncovered edge cases rather than guessing.
• Adapt to experimental work:
• Take on new and evolving task types as project needs shift, applying sound judgment in areas where guidelines are still being developed.
• Report on model performance:
• Surface and communicate quality and performance trends in model and agent behavior, giving cross-functional partners clear, well-evidenced feedback on where models succeed, fail, and degrade.

Qualifications

• 1+ years of experience in AI data annotation, LLM evaluation, content moderation, research, or a related analytical role, with exposure to quality assurance, and/or preference ranking.
• Experience applying detailed guidelines to complex and often ambiguous content, with strong contextual and sociocultural judgment.
• Comfort with ambiguity and a willingness to flag unclear or uncovered edge cases rather than guess.
• A sharp, curious eye for inconsistencies, subtle errors, and model failure modes.
• Excellent command of written English and strong reading comprehension.
• Strong attention to detail and commitment to accuracy.
• Comfort working with annotation platforms and structured formats such as JSON, CSV/TSV, Markdown, XML, and YAML.
• Strong execution in a remote environment, including good time management and the ability to work independently.

Requirements

• This is a part-time independent contractor position available within Canada only.
• Commitment to 16 hours per week at a CAD $30/hour contract rate.
• This role is BYOD (Bring Your Own Device, laptop).
• This position is remote.

Candidate Journey

• Initial Screening: Once you have submitted your application, our Talent Team will review your resume and writing samples.
• Virtual Annotation Test: This assignment will test your written and technical skills through various language-based tasks.
• Video Screen: If selected to move forward, you will have a short video call with a member of our Operations Team!
• Offer: Independent Contractor Agreement.

Important Information

• As an independent contractor, continuous work is not guaranteed.
• The client-contractor relationship is fundamentally project-based.
• Prospective candidates, please be advised: this role involves working with human-generated and model-generated tasks that may involve exposure to not safe for work (NSFW) text content.