[Hiring] Senior QA Engineer @Vi

Role Description

Vi Operate puts AI agents into the daily operations of large health systems, providers, and pharma organizations. Augment is the voice capability underneath it: the agents that call patients and providers, hold a real conversation, and take action in the systems where the work actually lives. This is a dedicated quality role for that work.

Testing a voice agent is not like testing software that returns the same answer twice. A similar call can go three different ways. A model update can quietly change behavior that nothing in a unit test would catch. When a call goes badly, the cause could be the speech recognition, the model, the voice synthesis, the telephony layer, or the integration underneath, and the transcript alone won't tell you which. Making that testable, repeatable, and catchable before a client hears it is the job.

You will own quality for agents that call real patients, providers, and caregivers. That means building the test and evaluation systems our engineers develop against, deciding what has to pass before something ships, and being the person who can take a bad call recording and say what actually went wrong. We need someone who has done classical QA properly and has since tested voice and LLM systems in production, because this role needs all three.

What You'll Own

• Voice testing and debugging: Call recordings, transcripts, latency traces, and logs.
• Evaluation and regression testing: Golden test sets, simulated calls, judged transcripts, accuracy on classification and extraction, and structured output validation.
• Testing things that aren't deterministic: Deciding what "passing" means when the output varies run to run.
• Release criteria: What has to be true before an agent goes live for a client, and what has to stay true afterward.
• Classical QA for the rest of the product: Test plans, regression suites, bug triage, reproductions, and CI integration for everything around the agent.
• Quality in production: Sampling live calls after release, spotting drift and degradation.
• Breaking it on purpose: Red-teaming the agent before someone outside does.

Qualifications

• QA on production voice agents: You have tested a system that talks to people on the phone.
• QA on LLM systems: You have tested something built on a language model in production.
• Real classical QA depth: Test strategy, test planning, regression suites, bug triage, and release process.
• Testing non-deterministic systems: You have dealt with output that varies and made it testable anyway.
• Python and test automation: You write your own tooling and wire it into CI.
• Judgment about severity: You can tell the difference between a cosmetic defect and something that will mislead a patient.
• A system mindset: You look at the whole system rather than one piece.

Requirements

• Background in IVR, contact center, or CCaaS quality (Genesys, NICE, Five9, Amazon Connect, Twilio, or similar).
• Audio and network condition testing: codecs, jitter, packet loss, barge-in and interruption handling.
• Familiarity with evaluation tooling such as LLM-as-judge harnesses, prompt regression frameworks, or simulated conversation testing.
• Healthcare or pharma experience, and comfort working under HIPAA.
• Experience standing up a QA function rather than joining one.
• Load and performance testing on real-time systems.
• Spanish proficiency, enough to listen to a call and judge whether the agent handled it.

Company Description