[Hiring] AI Evaluation Engineer @Sedgwick

Role Description

The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle.

Fully remote role, open to U.S. based candidates.

Essential Functions and Responsibilities

• Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release.
• Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency.
• Defines evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., faithfulness, groundedness, relevance, task completion, tool-use correctness, safety).
• Builds reusable evaluation frameworks, templates, and libraries that pods can adopt with minimal effort.
• Partners with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds.
• Develops automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes.
• Implements offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts).
• Supports A/B and shadow testing for model, prompt, and architecture changes.
• Designs, validates, and maintains LLM-as-a-judge evaluators, including rubric design, bias mitigation (position, verbosity, self-preference), and calibration against human labels.
• Establishes human annotation workflows, labeling guidelines, and inter-annotator agreement practices with SMEs.
• Tracks judge reliability over time and re-calibrates as models and use cases change.
• Builds and curates golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches.
• Applies approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data.
• Implements red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing.
• Produces evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews.
• Contributes to responsible AI standards and release-readiness criteria.
• Embeds with pods to onboard them to evaluation practices, then coaches teams toward ownership of their own eval suites.
• Builds dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends.
• Evaluates and recommends eval tooling and platforms; documents standards and best practices.

Qualifications

• Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred.
• 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications.

Skills & Knowledge

• Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs.
• A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance.
• Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review).
• Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.
• Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation.
• Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement.
• Familiarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent).
• Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services.
• Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders.
• Excellent oral and written communication skills, including presentation skills.
• Ability to work in a team environment.
• Ability to meet or exceed Performance Competencies.

Preferred Qualifications

• Experience evaluating agentic systems (multi-step reasoning, tool calling, trajectory evaluation).
• Background in ML evaluation, search/relevance, NLP, or QA automation.
• Experience with LLMOps, monitoring and observability, and production feedback loops.
• Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments.
• Experience with data annotation workflows and working with SMEs.
• Knowledge of enterprise controls: SSO, RBAC, private networking, logging, and encryption.

Success Measures

• Standard evaluation framework adopted by pods.
• Automated quality gates in CI/CD for production-bound solutions.
• Calibrated LLM judges with documented agreement against human labels.
• Reduction in regressions reaching production and in time to validate model or prompt changes.

Work Environment

• Mental: Clear and conceptual thinking ability; excellent judgment, troubleshooting, problem solving, analysis, and discretion; ability to handle work-related stress; ability to handle multiple priorities simultaneously; and ability to meet deadlines.
• Physical: Computer keyboarding, travel as required.
• Auditory/Visual: Hearing, vision and talking.