[Hiring] Software Engineer – AI Evaluation Expert @Patronus AI, Inc.

Role Description

We are looking for experienced software engineers (task-based contract) to design realistic, challenging tasks for frontier AI models and to judge how well those models handle them. You'll apply your day-to-day engineering judgment to show where AI gets code, reasoning, and technical instructions right, and where it falls short.

Depending on the project, you may:

• Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code, and follow precise instructions.
• Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs, and datasets.
• Evaluate AI-generated code and technical content for correctness, efficiency, clarity, and alignment with real-world engineering practice.
• Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
• Help improve how AI understands programming, system design, data workflows, and engineering best practices.
• Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.
• Test and refine content to reflect real-world complexity.
• Review other contributors’ work for accuracy, clarity, and practical relevance.
• Incorporate feedback promptly and flag gaps or unclear guidance.

Qualifications

• 2+ years of professional experience in software engineering, data science, or analytics.
• Experience with system design, data pipelines, ML or LLM-based applications.
• Prior work in code review, technical writing, AI training, or model evaluation.
• Strong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript, or Go.
• Experience building, reviewing, and debugging production-grade code.
• Ability to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.
• Excellent written communication and close attention to detail.
• A bachelor's degree in computer science, engineering, or a related field, or equivalent experience.

Requirements

• All tasks and solutions must be human-authored. LLM-generated content is not permitted.
• Tasks must be original and non-searchable.
• Accurate, carefully checked contributions.
• Quality over volume.

Benefits

• Pay: $30–$100 per hour and some tasks would have a flat rate.
• Location: Remote (India).
• Type: Hourly, project/task-based contract (not a full-time role).
• Commitment: Flexible, typically 15–20 hours per week; can be done alongside a full-time job.