[Hiring] AI Trajectory Annotator and Reviewer @Athyna

Role Description

We're looking for an AI Trajectory Annotator and Reviewer for a US-based AI research company building custom reinforcement learning environments and coding agent benchmarks. This is a contract role with flexible scheduling.

• Read and annotate AI agent trajectories step by step, evaluating each command against the actual state of the environment.
• Review existing annotations from AI tooling or other annotators, confirming, correcting, or rejecting each step with clear technical reasoning.
• Identify the precise point in a run where the agent first went off course, not just where it visibly failed.
• Distinguish between incorrect agent decisions and merely inefficient ones, as both are scored differently on the platform.
• Flag cases where the agent attempted to game the evaluation rather than solve the task genuinely.
• Apply human judgment throughout the entire annotation process, with no AI tools used for any decision-making step.

Qualifications

• 2+ years of hands-on experience in software engineering, DevOps, or site reliability, with real debugging in real codebases.
• Strong proficiency in Python or Go, plus the ability to read and reason about unfamiliar languages under time pressure.
• Solid Linux knowledge: logs, running processes, build failures, and containerized environments.
• Everyday comfort with Git, diffs, pull requests, and issue tracking.
• Ability to debug effectively with no test suite and no error message pointing directly to the cause.
• Capacity to hold context across long agent runs where a decision at step 70 may depend on what happened at step 10.
• Clear written English: every annotation judgment requires a technical explanation that another engineer can independently verify.

Requirements

• Advanced English proficiency.
• Location: LATAM (Brazil, Argentina, or Colombia) - Reporting to US, EST.
• 100% Remote.
• Contract, hourly position.

Skills that will make you stand out

• Experience with Rust, C, or C++.
• Prior AI evaluation work such as RLHF, supervised fine-tuning, model evaluation, or red teaming.
• Familiarity with Docker and containerized test environments.
• Background in concurrency, asynchronous code, or distributed systems.