[Hiring] ML Evaluation Designer - Expert - AI Trainer @Mercor

Role Description

• Design ML/LLM evaluation tasks, rubrics, and metrics to enhance model performance.
• Grade model/agent outputs and improve evaluation quality through thorough review.
• Apply training-side judgment in SFT, RLHF, and reward modeling to evaluation design.
• Collaborate with AI research teams to refine evaluation signals and improve model outputs.
• Work independently and asynchronously to meet deadlines and improve AI model performance.

Qualifications

• Must-Have: 5+ years as a Machine Learning Engineer with hands-on training and evaluation experience.
• Proficiency in PyTorch, JAX, and Hugging Face.
• Strong written communication skills.
• Preferred: Fluency in SFT, RLHF, reward modeling, and evaluation metrics.

Requirements

• Interview Process:
• Mercor Technical Screen: Short assessment.
• Live Code Review Session: Interactive evaluation.
• Domain Expert Interview: In-depth discussion.
• You're paid $200 for completing all three, regardless of outcome.

Benefits

• Application Process (Takes 20–30 mins to complete):
• Upload resume
• AI interview based on your resume
• Submit form
• Resources & Support:
• For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
• For any help or support, reach out to: [email protected]
• Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.