[Hiring] LLM Engineering Expert @Careerflow.ai

Role Description

We are looking for experienced AI Evaluation Engineers (Engineering Simulation & Design) to author and validate "model-breaking," simulation-based engineering design problems to train and evaluate state-of-the-art AI agents. Operating across major engineering disciplines—including Electrical, Mechanical, Control Systems, Aerospace, Systems, and Robotics—you will:

• Create complex, multi-constraint tasks where AI agents must interpret requirements, navigate trade-offs, configure open-source simulation tools, diagnose failures, and iterate toward valid solutions.
• Analyze agent execution logs, expose systemic reasoning gaps, and build automated, objective graders to elevate frontier model performance.

Qualifications

• Master’s degree or PhD in Electrical, Mechanical, Aerospace, with 10+ years of hands-on engineering design experience.
• Bachelor's degree or equivalent practical experience in any field.
• Experience in AI evaluation, data annotation, content review, quality assurance, or a related analytical role is preferred but not required.

Requirements

• Proficiency with at least one domain-relevant open-source simulation package (e.g., ngspice, PySpice, OpenFOAM, FEniCSx, CalculiX, python-control, CadQuery, build123d, OpenModelica, Cantera, Gmsh) combined with strong Python scripting skills.
• Hands-on experience with modern LLMs/coding agents and evaluation concepts (pass@k, failure-mode analysis, nondeterministic behavior), with the ability to audit trajectory logs and isolate core reasoning/tool-use failures.
• Uncompromising attention to physical plausibility, unit consistency, boundary conditions, convergence criteria, and technical documentation.
• Talent must have weekend on-call availability (part-time engagement is acceptable).
• Personal desktop/laptop equipped with a stable, high-speed internet connection in a remote setup.

Job Responsibilities

• Author original, self-contained engineering design tasks with competing constraints, explicit optimization targets, validated reference solutions, and objective autograders.
• Build, run, and validate problem environments using open-source simulation tools and custom Python test benches.
• Evaluate coding agent outputs and execution logs across repeated trials to identify systemic failure modes (e.g., misinterpreting simulator feedback, premature design convergence, physically impossible geometries).
• Iteratively refine problem difficulty based on empirical model performance data without introducing ambiguity or missing information.
• Partner with AI researchers, pod leads, and domain experts to integrate high-rigor benchmarks into the model evaluation pipeline.

Location

• Bangladesh
• India
• Indonesia
• Egypt
• Ghana
• Kenya
• Nigeria
• Turkey
• Brazil
• Colombia

Offer Details

• Commitments Required: 30-40 hours per week with 4 hours of overlap with PST.
• Engagement type: Contractor
• Engagement Length: Up to 24 weeks
• Rate Range: $500 per task (considering 30-40 hours of AHT per task)
• This is a Pay-Per-Task model which means you'll get paid per task for an estimated hourly rate.

Evaluation Process

• Shortlisted candidates will be sent a Job Interest Form.
• Finalized talents will go through delivery review & proceed further accordingly.