[Hiring] Senior Software Development Engineer in Test @Qualia

Role Description

We are hiring pioneering and collaborative engineers to build our next generation of intelligent products. Our quality engineers are conscientious hackers who thrive on collaboration and are adept at driving large-scale projects in the fast-changing domain of artificial intelligence. You will help build the automation, tooling, observability, and quality systems that let us ship complex, AI-driven products with confidence, bringing rigor and trust to a data-rich industry.

As a Senior AI/Agentic Quality Infrastructure SDET, you're a software engineer first. You build:

• Test infrastructure
• Agentic validation systems
• AI/ML evaluation harnesses
• Observability platforms
• Engineering tooling

You will help evolve our quality architecture from primarily deterministic automation toward a combination of traditional automation and agentic validation, building systems where AI agents can understand application state, execute workflows, use engineering tools, validate outcomes, investigate failures, and provide evidence for their conclusions.

You can take a hard, loosely defined problem and solve it. You're trusted to make technical decisions on frameworks, architecture, evaluation strategy, observability, and validation across every layer of the testing pyramid. You work closely with Engineering and Quality leadership, understand the systems you're testing at a deep technical level, and mentor engineers on building reliable tests, tooling, infrastructure, and AI-assisted validation systems.

Responsibilities

• Build and maintain test infrastructure, AI/ML evaluation systems, agentic validation harnesses, release orchestration tooling, observability, and performance testing platforms as production-grade software rather than scripts.
• Design agentic test harnesses that let AI agents interact with applications, browsers, APIs, databases, logs, telemetry, and CI/CD systems to execute and validate complex workflows, reasoning about application state rather than relying solely on predefined test scripts.
• Build AI/ML evaluation frameworks for validating non-deterministic systems, including deterministic assertions, semantic and model-based evaluation, regression datasets, and human-in-the-loop review, defining measurable quality signals such as correctness, reliability, latency, cost, and hallucination/error rates.
• Build observability and telemetry into our automation and agentic infrastructure—structured logs, metrics, distributed tracing, and execution timelines—so engineers can understand why an automated or agentic workflow succeeded or failed.
• Develop systems that automatically distinguish product, test, infrastructure, and AI/model reasoning failures, correlating them with logs, traces, deployments, and commits to accelerate root-cause analysis.
• Design and build our test automation frameworks, including the migration from Cypress to Playwright, and own core infrastructure such as test environment provisioning, parallelization, and CI/CD integration at scale.
• Apply AI to engineering-quality problems including test generation, flaky-test triage, failure clustering, and release-risk analysis, while building guardrails so agent conclusions are backed by observable evidence.
• Partner with Software Engineers, Quality Engineers, SRE, and Platform teams to make these capabilities reusable platforms, and mentor engineers on test architecture, observability, and AI-assisted engineering.

Qualifications

• 5+ years of experience as an SDET or software engineer building test infrastructure, developer tooling, platform engineering, observability, or release engineering systems—not primarily writing test cases or automation scripts.
• Strong programming ability in JavaScript/TypeScript, including designing backend services, libraries, frameworks, and production-quality infrastructure from scratch.
• Strong software engineering fundamentals including system design, distributed systems, APIs, asynchronous systems, and production operations.
• Experience building or scaling CI/CD systems, release orchestration, test infrastructure, or developer platforms, and designing observability/telemetry systems using structured logging, metrics, and distributed tracing.
• Hands-on experience with LLMs, AI-assisted engineering, or agentic systems beyond simply using an AI coding assistant, including tool/function calling, agent workflows, and multi-step AI workflows.
• Experience or familiarity with LLM and agent evaluation, including evaluation datasets, deterministic and model-based evaluators, and the unique challenges of testing probabilistic/non-deterministic systems.
• Strong understanding of the testing pyramid and experience determining when validation belongs at the unit, integration, API, UI, performance, or agentic layer.
• Comfortable mentoring engineers and raising the technical quality bar across teams.

Benefits

• Comprehensive health plans
• 401k program
• Commuter benefits
• Professional development
• Parental leave
• Flexible time off policy
• Robust online onboarding program
• Biweekly all hands meetings
• Variety of internal virtual events

Company Description

Qualia is made up of incredibly bright, mission-driven coworkers who are passionate about using technology to solve real-world problems—and we're growing quickly. We believe diverse perspectives and backgrounds are critical to building great technology, and our goal is to cultivate an environment where people feel equally valued and respected. Qualia is proud to be an equal-opportunity workplace, and we welcome applicants from all backgrounds regardless of race, color, ancestry, religion, gender identity or expression, sexual orientation, marital status, age, citizenship, socioeconomic status, disability, or veteran status.