[Hiring] Senior QA Engineer @Delta Exchange

Role Description

Delta operates three AI products used by live traders: a customer support chatbot, an API Copilot that generates and executes trading scripts, and an MCP server. This role owns the quality of those products and is responsible for measuring it objectively.

Testing probabilistic systems differs fundamentally from testing deterministic ones:

• The same input produces different output on every run.
• An incorrect answer can be entirely fluent.
• A prompt change in one flow can degrade another without any visible signal.

The core of this role is establishing what correct means for these systems, building the datasets and scoring that measure it, and operating the release gate that prevents a regression from reaching customers.

This is an emerging discipline with no established playbook. We are defining the methodology as we build it, and the role carries a high degree of autonomy and ownership.

Key Responsibilities

• Build and maintain golden datasets for our AI products, trace mined from production or generated, and verified against a documented source of truth.
• Design layered scoring for AI outputs: deterministic rules where behaviour can be asserted, and model-graded evaluation where it cannot.
• Manage evaluation runs end to end: schedule and execute them against release candidates, compare results against prior baselines, and triage failures into product defects, incorrect expectations, and platform issues.
• Own and operate the release gate that determines whether a prompt or model change is approved for production.
• Identify failure modes ahead of customers, including hallucination, incorrect tool selection, loss of context across conversation turns, and PII exposure.
• Investigate production traces to distinguish retrieval failures from generation failures from tool failures.
• Convert findings into actionable engineering evidence and into permanent regression coverage.
• Define and report quality metrics for AI surfaces, and drive improvement against them.

Qualifications

• At least 1 year owning quality for AI products in a lead or primary-owner capacity, including chatbots and conversational assistants, code or content generation products, and agentic systems.
• Demonstrated hands-on experience building or operating an evaluation harness for an AI system, whether in-house or using a framework such as DeepEval, RAGAS or Promptfoo.
• 4-6 years of QA / SDET experience.
• Working proficiency in Python.
• Strong API testing experience.
• Sufficient engineering ability to build your own tooling and automation.
• Clear written communication.

Requirements

• Ability to analyse traces and tool calls, using observability tooling such as Opik, Langfuse or LangSmith or an equivalent, to determine root cause rather than reporting the symptom.

Bonus

• Experience with trading or exchange platforms, including familiarity with futures and options, crypto derivatives, margin, or settlement flows.

Benefits

• Play a pivotal role in shaping the regulatory landscape for digital assets and Web3 in India.
• Work directly with founders and senior leadership on high-impact strategic initiatives.
• Be part of a mission-driven, fast-growing organisation at the forefront of financial innovation.
• Competitive compensation, leadership exposure, and significant growth opportunities.