[Hiring] Senior Automation Test Engineer @Our client

Role Description

Our client is a fast-growing healthcare technology company building AI voice agents that answer every call for physician practices — scheduling appointments, routing patients, and resolving routine requests around the clock. They are looking for a Senior Automation Test Engineer who can plug in and build rapidly, owning the test strategy and automation infrastructure for their voice AI platform and building the frameworks that make non-deterministic AI behavior measurable and regressions obvious.

• Design, build, and maintain automated test infrastructure across unit, integration, end-to-end, and load/performance layers for a real-time voice AI platform.
• Build evaluation and regression harnesses that score conversation outcomes and intent fulfillment for non-deterministic, LLM-driven agent behavior.
• Generate synthetic conversation data and de-identified patient scenarios to expand test coverage for rare edge cases.
• Develop test frameworks and tooling that enable the broader engineering team to write and run reliable tests efficiently.
• Define and enforce quality standards and testing best practices across engineering teams.
• Build simulation and regression frameworks for real-time voice and conversational AI scenarios, including heavy rules-based logic layered on top of nondeterministic AI behavior.
• Own test environments and CI/CD pipeline integration for automated test execution.
• Investigate and triage production quality issues, driving root cause analysis and prevention.
• Develop testing approaches that satisfy HIPAA and healthcare compliance requirements.
• Ramp quickly: within the first 30 days, understand the tech stack and business, and begin shaping a test suite strategy for the platform.

Qualifications

• 10+ years in software quality engineering with a strong focus on test automation.
• Proven experience designing and owning test automation frameworks from the ground up.
• Proficiency in Python, TypeScript, or equivalent, with strong scripting and tooling skills.
• Experience with CI/CD integration (GitHub Actions, Jenkins, or equivalent).
• Hands-on AI evaluation experience — testing AI-powered applications, including prompt and model regression testing.
• Demonstrated ability to test non-deterministic conversational flows, designing assertions and evaluation criteria that tolerate variation without masking real regressions.
• Experience generating synthetic data at scale — realistic conversation transcripts, edge-case inputs, and de-identified patient scenarios.
• Ability to move quickly and independently in a fast-scaling environment with minimal ramp-up time.

Requirements

• Nice to have: experience with real-time systems, telephony, or voice platforms.
• LLM evaluation tooling and prompt/model versioning.
• Evaluating speech-to-text or text-to-speech quality.
• Load/performance testing for high-throughput systems.
• HIPAA-regulated environment experience.
• Contract testing or service virtualization.