[Hiring] Member of Technical Staff, Evals @Handshake

Role Description

We are hiring a Member of Technical Staff, Coding Evals to help define how frontier coding agents are measured, understood, and improved. This is a high-ownership role for someone who is deeply passionate about building software with agents—and about creating the evaluations that reveal where those agents truly succeed and fail.

• Design and publish coding benchmarks that meaningfully challenge state-of-the-art agents.
• Work with AI researchers, software engineers, and domain experts to turn difficult, real-world software tasks into high-signal evaluation environments, datasets, verifiers, and feedback systems.
• Shape how the frontier evaluates coding agents today and influence the future direction of the field.
• Early members of the team will have unusual influence over our technical direction, operating culture, and the open-source benchmarks, software, and research products we build.
• We care more about demonstrated technical depth, judgment, and a builder's mindset than a specific title, degree, or career path.

Qualifications

• Deep enthusiasm for agentic software development, with clear evidence that you actively build, experiment with, or think seriously about coding agents.
• Strong software engineering skills and the ability to write clean, reliable, maintainable code.
• Strong Python skills and comfort working with modern ML tooling, evaluation infrastructure, and data workflows.
• Sound experimental judgment: you can form hypotheses, choose meaningful metrics, diagnose failures, and distinguish genuine capability improvement from evaluation artifacts.
• Experience designing systems—not only implementing specifications—including the ability to make tradeoffs around validity, quality, scale, reliability, and reuse.
• Comfort operating in an ambiguous, fast-moving environment with substantial ownership.
• Collaborative, low-ego communication and the ability to work effectively with researchers, engineers, domain experts, and customers.

Requirements

• Experience working on a widely used coding-AI benchmark or evaluation suite.
• Published research on AI for coding or code generation at a leading venue, such as NeurIPS, ICML, ICLR, or COLM.
• Software engineering experience at a top technology company, or a strong public GitHub profile, paired with deep knowledge of agentic AI and a clear passion for building software with agents.

Especially compelling experience

• Building or maintaining coding benchmarks, coding-agent environments, repository-level evaluation suites, or open-source developer tools.
• Developing automated graders, test harnesses, programmatic verifiers, reward models, or reinforcement-learning environments for software tasks.
• Researching code generation, program synthesis, AI agents, reinforcement learning, post-training, or software-engineering productivity.
• Experience with post-training methods such as reinforcement learning, RLHF, preference optimization, supervised fine-tuning, or reward modeling.
• Strong public contributions through GitHub, papers, benchmarks, technical writing, or developer communities.
• Experience turning research prototypes or repeated customer work into robust, reusable products or platforms.

Benefits

• Ownership: Equity in a fast-growing company
• Financial Wellness: 401(k) match, competitive compensation, financial coaching
• Family Support: Paid parental leave, fertility benefits, parental coaching
• Wellbeing: Medical, dental, and vision, mental health support, $500 wellness stipend
• Growth: $2,000 learning stipend, ongoing development
• Office: Commuting support, free lunch, and gym in our SF office
• Time Off: Flexible PTO, 15 holidays + 2 flex days
• Connection: Team outings & referral bonuses