[Hiring] Software Engineer - Benchmark Auditor @Mercor

Role Description

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks.

• Assess repository-level tasks, reference patches, test harnesses, and grading integrity.
• Provide clear, rubric-based written feedback to improve AI model training.
• Audit reference patches, test runners, and Docker isolation to detect answer leakage and reward hacking.
• Work independently and asynchronously to meet deadlines and enhance AI model performance.

Qualifications

• 3+ years professional software engineering experience.
• Real open-source contribution or maintainer experience (merged PRs, committer/maintainer roles).
• Strong ability to audit reference patches, test runners, and Docker isolation.
• Fluency across common ecosystems (Python and at least one of Java/Go/TypeScript/C++).

Requirements

• Familiarity with SWE-Bench (Verified) or similar repository benchmarks.
• Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.).
• Prior code-review or task-grading experience.

Benefits

• Compensation: $70–$90/hour.
• Location: Remote.

Application Process

• Upload resume.
• AI interview based on your resume.
• Submit form.

For details about the interview process and platform information, please check: Interview Process .

For any help or support, reach out to: [email protected] .

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.