[Hiring] Senior QA & Evaluation Engineer @Encora

Role Description

We at Coforge are hiring a Mid-Senior QA & Evaluation Engineer (N-5) with the following skill set.

• Design and maintain automated test suites covering unit, integration, API, and end-to-end testing.
• Integrate automated testing into CI/CD pipelines and establish automated suites as release quality gates.
• Build evaluation frameworks for AI capabilities, including scored rubrics, LLM-as-a-judge approaches, and normalized pass thresholds.
• Build and maintain held-out evaluation datasets, measure AI behavior against ground truth, and report defect escape and evaluation results as part of delivery metrics.

Qualifications

• 4+ years of experience in Quality Engineering with a strong focus on test automation.
• Hands-on experience with automated testing frameworks, preferably within the .NET ecosystem.
• Strong experience with API testing and automated test development.
• Experience integrating automated test suites into CI/CD pipelines.
• Understanding of testing strategies for non-deterministic and probabilistic AI systems.
• Ability to design evaluation approaches based on scored thresholds rather than exact-match assertions.
• Experience working with test data, ground truth datasets, and automated validation.
• Strong analytical and problem-solving skills.
• Strong working proficiency in English.
• Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field, or equivalent professional experience.

Requirements

• Direct experience with LLM evaluation, including rubric design, LLM-as-a-judge, golden/held-out datasets, or tools such as Azure AI Foundry Evaluators, promptfoo, Ragas, or DeepEval.
• Experience testing information extraction or classification pipelines using accuracy-based measurements.
• Experience with regulated-industry QA environments where test evidence and results must support formal review.
• Familiarity with AI/ML testing methodologies and evaluation frameworks.

Company Description

At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.