[Hiring] Senior Software Development Engineer in Test @Nexxa.AI

Role Description

We’re looking for a Senior / Staff SDET to own quality for Nexxa’s AI agent systems, combining deep test automation with hands-on manual and exploratory testing. Our agents plan, call tools, and take multi-step actions autonomously in industrial environments. You’ll write production-grade code to build evaluation harnesses, simulation environments, and CI/CD quality gates, and you’ll also get hands-on with the agents yourself: probing behavior, reviewing traces, and validating releases where automation alone can’t judge whether the agent did the right thing.

You’ll work closely with ML engineers, backend engineers, and Forward Deployed Engineers to define what “good” looks like for an agent operating in high-stakes industrial settings, then build and run the systems and processes that measure it continuously.

Key Responsibilities

• Automation and Infrastructure
  • Design, build, and maintain test frameworks, evaluation harnesses, and regression suites for LLM-based agents.
  • Build automated pipelines that run evals and test suites on every model, prompt, or tool-integration change.
  • Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs.
  • Implement API, contract, integration, and end-to-end tests for the services, tools, and orchestration layers.
  • Develop performance, load, and reliability tests for agent workflows.
• Manual and Exploratory Testing
  • Perform hands-on exploratory testing of agent behavior.
  • Manually review agent traces and outputs to judge reasoning quality.
  • Execute manual regression and release verification passes.
  • Conduct manual red-teaming sessions in partnership with security teams.
  • Support user acceptance testing and pilot rollouts with Forward Deployed Engineers.
  • Write clear test plans, test charters, and bug reports.
• Datasets, Metrics, and Quality Strategy
  • Create and manage golden datasets and labeled test sets.
  • Design rubrics and run human-in-the-loop evaluations.
  • Define and instrument quality metrics beyond simple accuracy.
  • Test agent behavior across the full action loop.
  • Investigate and triage failures where the root cause could be the model, the prompt, the tool/API, or the orchestration logic.
  • Establish quality bars, release criteria, and sign-off processes for new agent capabilities.
  • Decide what to automate and what to keep manual.
  • Mentor engineers on testing strategies and test-driven practices.
  • Advocate for testability, observability, and tracing in agent architecture.

Qualifications

• 5+ years as an SDET, QA engineer, or software engineer with a strong testing focus.
• Proven experience in both manual and automated testing.
• Strong programming skills (Python preferred).
• Experience building test frameworks and infrastructure from the ground up.
• Hands-on experience with CI/CD systems and containerized test environments.
• Experience testing APIs and microservices.
• Hands-on experience testing LLM-based products, chatbots, or AI agents.
• Practical experience with eval frameworks or tooling.
• Understanding of how LLM agents work.
• Experience designing test data and labeled datasets.
• Familiarity with LLM-specific failure modes.
• Comfort operating in ambiguity.
• Strong written communication skills.

Preferred

• Experience with human-in-the-loop evaluation workflows.
• Experience with performance and load testing tools.
• Background in ML/data science.
• Experience red-teaming or doing adversarial/security testing on ML systems.
• Familiarity with observability and tracing tools for LLM applications.
• Experience testing systems in industrial, IoT, or operational technology environments.
• Experience supporting UAT, pilots, or customer-facing validation of complex products.
• Experience setting up test and eval infrastructure from scratch.

What We’re Looking For

• An engineer who is equally comfortable writing a test framework and sitting down to break an agent by hand.
• Someone who can build rigorous evaluation systems for problems that don’t have a single right answer.
• A systems thinker who enjoys turning ambiguous agent behavior into measurable, trustworthy signals.
• A strong collaborator who partners well with ML engineers, backend engineers, and Forward Deployed teams.

Why Join Nexxa.AI?

• Innovative Environment: Play a critical role in transforming heavy industries through groundbreaking AI and automation technologies.
• Collaborative Culture: Be part of a team that values innovation, discipline, and continuous improvement.
• Professional Growth: Benefit from significant opportunities for career development and advancement.
• Competitive Compensation: Enjoy a comprehensive salary and equity package reflective of your expertise and contributions.