[Hiring] Generalist Annotator @Mercor

Role Description

Evaluate the relevance and clarity of AI responses. Ensure the AI answers the question in understandable language for non-experts.

• Detect sycophancy in AI interactions. Identify when the AI tells users what they want to hear instead of what they need.
• Assess the tone and outcome of conversations. Determine if the user leaves informed and calm or anxious and uninformed.
• Provide written justification for ratings below the top option. Explain what went wrong and suggest improvements.
• Work independently to review conversations ranging from 1 to 7 turns. Spend 2–13 minutes per conversation based on length.

Qualifications

• Strong written English skills.
• Ability to explain judgments in two to four specific sentences.
• Careful reading to notice omissions in responses.
• Comfort with applying a detailed rubric consistently.
• Reliable availability for time-boxed batches.

Requirements

• Experience in annotation, human-feedback, evaluation, or content review.
• Background in explaining technical concepts to non-experts.
• Experience with rubric-based grading or quality assurance.

Benefits

• Compensation: $20–$160/hour
• Location: Remote

Application Process

• Upload resume
• AI interview based on your resume
• Submit form

Resources & Support

• For details about the interview process and platform information, please check: Interview Process
• For any help or support, reach out to: [email protected]