[Hiring] Senior SQA Engineer @Techsurge Private Limited

Role Description

Design and own the end-to-end QA strategy for the Conversational Banking Platform, covering:

• Functional evaluation
• Regression testing
• Performance assessment
• Security checks
• AI-specific evaluation

Build and maintain:

• Golden datasets
• Eval suites
• LLM-as-judge frameworks to validate conversational quality across intents, languages, and tenants

Define the tenant onboarding QA gate, including:

• The certification checklist every new business unit must pass before going live

Establish regression strategies for:

• Prompt changes
• Model upgrades
• Retrieval index updates
• Guardrail policy changes

Use Langfuse traces to drive evaluation by:

• Mining production failures
• Converting failures into test cases
• Closing the loop with engineering

Test NeMo Guardrails configurations against:

• Jailbreaks
• Prompt injection
• Off-topic drift
• False-positive over-blocking

Validate governance and compliance behaviors, including:

• Data residency
• PII handling
• Regulated-product disclosures
• Off-limits topics

Build automated test harnesses for Spring AI services, including:

• Tool-calling validation
• RAG groundedness
• Integration with Cosmos DB and MongoDB data layers

Partner with the Platform team on:

• Quality metrics
• SLOs
• The platform eval scorecard

Coach feature engineers and tenant teams on:

• Writing their own evals
• Making platform-grade quality self-service over time

Qualifications

• 6+ years in software QA, with at least 1–2 years testing LLM-based, RAG, or conversational AI systems in production
• Hands-on experience with LLM observability and evaluation tools such as Langfuse, LangSmith, Arize, or Phoenix
• Working knowledge of eval frameworks such as Ragas, DeepEval, Promptfoo, or TruLens — including metrics like faithfulness, groundedness, answer relevance, and context precision
• Practical understanding of how to test non-deterministic systems: golden datasets, semantic similarity, LLM-as-judge, and statistical regression detection
• Experience testing guardrail or policy frameworks (NeMo Guardrails, Guardrails AI, or similar)
• Solid foundation in API testing, automation frameworks (e.g., pytest, JUnit, Karate, RestAssured), and CI/CD integration
• Familiarity with Spring and Spring Boot applications and JVM-based services
• Comfortable writing queries against NoSQL stores (MongoDB, Cosmos DB) for test data setup and trace inspection
• Strong written communication: able to produce clear test plans, defect reports, and tenant readiness assessments

Good to Have

• Experience in banking, financial services, or another regulated industry
• Exposure to multi-tenant platforms: understanding how shared infrastructure changes the testing problem
• Familiarity with red-teaming, adversarial prompt testing, and prompt injection defense
• Working knowledge of vector databases, embedding models, and retrieval evaluation
• Experience with multi-language conversational systems
• Performance and load testing experience for AI workloads (token throughput, latency percentiles, cost per conversation)
• Contributions to open-source eval or AI testing tooling
• Experience working with compliance, risk, or audit teams on AI assurance