[Hiring] AI Evaluation Engineer @Scopic

Role Description

This is a remote position. Are you a QA or eval engineer who would rather build the testing system than run someone else’s? We’re looking for a founding AI Evaluation Engineer to create the evaluation, testing, and observability infrastructure for production-grade AI agents. These agents handle complex reasoning, multi-turn conversations, and direct tool execution (APIs, code execution, database calls). You’ll go beyond UI automation and manual prompt checks to build headless, statistical, and deterministic evaluation pipelines for non-deterministic LLM behavior.

• Design and deploy automated CI/CD testing pipelines that validate agent behavior before deployment
• Build and maintain golden datasets of regression prompts, edge cases, and synthetic scenarios to measure agent accuracy, safety, and latency
• Implement fast, low-cost code-based checks before relying on LLM-as-a-judge. Examples include JSON schema validation, regex matching, API status checks, and embedding similarity.
• Design test suites for multi-turn conversations, validating context retention, state drift, and safeguards against infinite tool-calling loops
• Create adversarial test suites covering prompt injections, malformed inputs, and chaotic tool responses to keep sandboxes and guardrails secure
• Implement tracing with OpenTelemetry, LangSmith, Phoenix, or similar tools, and reduce LLM API costs through caching and mocking

Qualifications

• Proficiency in Python or TypeScript, with a focus on API testing, headless execution, and asynchronous systems
• Hands-on experience building LLM evaluation pipelines or statistical testing frameworks, such as confidence intervals or pass rates over N runs
• Experience integrating tests into GitHub Actions or similar CI systems using SDK-level execution
• Expertise in validating structured outputs: JSON schema, OpenAPI specs, and function-calling payloads
• Familiarity with tracing tools such as OpenTelemetry, LangSmith, Arize Phoenix, W&B Weave, or custom logging
• Ability to independently design and own an evaluation architecture from scratch
• Strong written and spoken English
• At least 3 hours of daily overlap with US Central Time business hours
• Experience with code execution sandboxes or agent frameworks such as LangChain, LlamaIndex, AutoGen, or CrewAI
• Deep understanding of embedding models and vector databases
• Experience load testing non-deterministic endpoints

Benefits

• Your growth is our growth. We invest in your future with paid training and other professional opportunities.
• We’re industry innovators. You’ll work with modern technologies and a team that values strong engineering and continuous improvement.
• Your location. Your schedule. Pick your time-zone, choose your preferred hours, and work from where you feel most productive.
• Flexibility and freedom are in our DNA. As long as you have a stable internet connection and the drive to thrive, you can work from anywhere.
• A workload you can rely on. We ensure a consistent flow of engaging, challenging product work.
• Recognition and reward. We acknowledge diligence and strong performance with annual pay increases.

Company Description

Scopic offers equal opportunity. We value diversity and do not discriminate based on race, religion, color, marital status, national origin, gender, veteran status, sexual orientation, age, or disability status.

Have the skills, the drive, and the passion to join the Scopic family? Apply today to join our growing team of remote professionals from around the world.