[Hiring] Quality Engineer, AI & Test Automation @Baylor Scott & White Health

Role Description

The Quality Engineer, AI & Test Automation helps modernize BSWH’s quality approach by moving beyond traditional manual testing toward automation-first, engineering-integrated, and AI-enabled quality practices.

• This role designs, builds, executes, and maintains automated test suites, validation utilities, evaluation assets, and quality reporting that improve release confidence and speed of delivery across digital products, enterprise applications, and AI-enabled use cases.
• The role partners closely with Product, Engineering, Architecture, Data & Analytics, AI Foundation, Ontology, Security, Privacy, Clinical, Operations, and other delivery teams to define quality expectations early and embed quality practices throughout the development lifecycle.
• This role is expected to combine strong testing discipline with technical skills, including automation, API validation, data validation, CI/CD integration, quality metrics, defect triage, and emerging AI-enabled testing practices.
• The Quality Engineer, AI & Test Automation reports to the Director of Quality Engineering and helps establish reusable practices that support safe, reliable, efficient, and scalable delivery.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Systems, Data Science, or related technical field.
• Equivalent combination of education and relevant technical experience may be considered.
• 5–8+ years of experience in quality engineering, software testing, test automation, SDET, software engineering, AI quality, or related technology delivery roles.
• Hands-on experience designing and implementing automated tests for APIs, web or mobile applications, workflows, data validation, integrations, or end-to-end product experiences.
• Experience working with engineering teams in agile, DevSecOps, CI/CD, or modern product-delivery environments.
• Experience with defect triage, root-cause analysis, test evidence documentation, release readiness, and quality metrics.
• Exposure to AI-enabled products, machine learning systems, large language models, conversational experiences, agentic workflows, automation, or decision-support technologies preferred.
• Healthcare, life sciences, financial services, or other regulated-industry experience preferred but not required.

Requirements

• Strong understanding of software testing concepts, including test design, test coverage, functional testing, regression testing, integration testing, data validation, exploratory testing, and release validation.
• Hands-on experience with test automation frameworks, scripting, test data, CI/CD integration, and defect management tools.
• Ability to validate APIs, backend services, user interfaces, data flows, workflows, and enterprise integrations.
• Programming, scripting, or data analysis skills, preferably Python, JavaScript/TypeScript, Java, SQL, or comparable languages used to build or maintain automated test assets.
• Familiarity with AI-enabled testing and evaluation concepts, including prompt/response validation, model-output evaluation, RAG or retrieval testing, agent workflow testing, hallucination risk, guardrails, and human-in-the-loop workflows.
• Ability to use logs, traces, telemetry, monitoring, dashboards, test artifacts, and production feedback to investigate quality issues.

Benefits

• Immediate eligibility for health and welfare benefits
• 401(k) savings plan with dollar-for-dollar match up to 5%
• Tuition Reimbursement
• PTO accrual beginning Day 1

Key Success Factors

• Demonstrated ability to build and maintain meaningful automated tests rather than relying primarily on late-stage manual testing.
• Strong technical judgment across API testing, UI testing, integration testing, data validation, regression testing, defect triage, CI/CD integration, and release readiness.
• Ability to test AI-enabled systems where behavior may be probabilistic, workflow-dependent, or context-sensitive rather than purely deterministic.
• Ability to use AI-enabled productivity tools responsibly to improve test generation, analysis, defect triage, and quality reporting while maintaining human judgment over release quality.
• Strong collaboration with product and engineering teams to define quality expectations early and build quality into delivery practices rather than inspect quality at the end.
• Strong analytical and communication skills, including the ability to explain quality risks, evidence, tradeoffs, and release recommendations to technical and non-technical stakeholders.
• Comfort operating in a regulated or high-trust environment where safety, privacy, security, compliance, auditability, and customer trust are critical.