[Hiring] ML Engineer, Retrieval & Grounded Generation @Red Cell Partners

Role Description

As QA Engineer you will own quality for an AI-enabled decision-support system: the test plan, the automated coverage, and the evidence that what was delivered is what was specified. This role sits on the product side of the organization, not the engineering side.

You will report through the Product Lead and take direction from product, while working with engineering and platform every day. This role verifies what was specified, ensuring that a test suite written from the code passes while still meeting customer requirements. As the first dedicated quality engineering hire, you are establishing the approach rather than inheriting one.

This is a fully remote role with occasional travel (up to 25%) to DEFCON AI HQ, customer sites, and vendor facilities as required.

Key Responsibilities

• Test Strategy and Acceptance Criteria:
  • Own the system test plan covering functional, integration, performance, and load testing, and maintain it as a formal deliverable.
  • Translate the acceptance criteria set by product into executable tests.
  • Review acceptance criteria with product before development begins, and identify requirements that are ambiguous or not verifiable as written.
  • Hold engineering, platform, and partner teams accountable to the acceptance criteria, and define what evidence must exist before a change is considered complete.
  • Work alongside the engineering team day to day, close enough to test their work well and independent enough to report what the testing shows.
  • Plan and coordinate user acceptance testing with the customer, and track findings through to resolution.
• Test Automation and Coverage:
  • Build and maintain automated test suites, and integrate them into the CI/CD pipeline as release gates.
  • Establish and hold an automated coverage standard, and report against it.
  • Build performance and load testing that validates the system under its expected user load.
  • Maintain regression coverage across releases.
• Measurement and Evidence:
  • Build and maintain labeled evaluation sets, working with subject-matter experts whose time is limited.
  • Design and run sampled audits confirming that automated decisions were correct.
  • Measure and report what the system misses alongside what it gets right.
  • Measure how often users override the system, and report the trend.
  • Document how each committed measure is computed, on what data, and what evidence accompanies a release.
  • Produce this evidence independently of the people building the models.
• Release Governance:
  • Maintain the version inventory across builds, models, configurations, and thresholds.
  • Produce test reports, evaluation records, and rollback criteria for each release.
  • Work with platform engineering to confirm that test environments match production closely enough for results to be meaningful.
  • Hold the gate: no release ships without the evidence its acceptance requires.

Qualifications

• 5+ years in software quality engineering, test automation, or software engineering with substantial dedicated test ownership.
• A clear point of view on how testing should be done, and the judgment to know which practices are worth holding to.
• Strong coding ability in a language and framework used for test automation, such as Python with pytest, or JavaScript or TypeScript with Playwright, Cypress, or Selenium.
• Routine use of AI-assisted development, with informed judgment about where it adds value and where its output requires verification.
• Demonstrated experience building automated test suites and integrating them into CI/CD as release gates.
• Experience owning a test plan through review and acceptance.
• Demonstrated habit of testing against the requirement rather than the implementation.
• Experience with performance and load testing, including establishing what a result does and does not demonstrate.
• Working statistical literacy, including sampling design and the confidence a measurement can support.
• Experience measuring what a system misses, not only what it gets right.
• Willingness to report a result the team did not want, and to keep the measurement independent of the people whose work it evaluates.
• Ability to gate work owned by people who do not report to you, and to earn enough credibility with engineers that the gate holds.
• Clear, audit-ready written documentation.
• Willingness to hold a release, and explain the reason, to people who want it shipped.
• US Citizenship Required.
• Active US Secret clearance.
• Willingness to travel up to 25% to customer sites, DEFCON AI HQ, and vendor facilities as required.

Preferred Qualifications

• Active Top Secret clearance.
• Experience in government DevSecOps, RMF, or ATO environments, and producing test evidence for formal accreditation or audit.
• Experience with contract and API testing, test data management, and building test infrastructure others can extend.
• Using language models to generate and maintain test coverage.
• Experience with performance testing tools such as JMeter, k6, Locust, or comparable.
• Experience validating machine learning or model output, including groundedness and citation checking.
• Experience in systems where an incorrect result carries real cost, and where testing had to satisfy an outside reviewer.
• Familiarity with model cards, NIST AI RMF, or comparable responsible AI practice.
• Experience with Section 508 and WCAG verification.

Benefits

• A fully remote, results-based environment.
• Competitive salary, bonus, and equity package.
• 100% employer paid, comprehensive health insurance including medical, dental, and vision for you and your family.
• Unlimited PTO, with your manager’s approval.
• Flexible work environment where you manage your work day.
• 14 weeks of fully-paid parental leave.
• Career track opportunity with potential for rapid advancement with strong performance.
• Opportunities for professional development and continued learning.
• Optional 401K, FSA, and equity incentives available.
• Mental health benefits available through Tara Mind.
• Cost effective GLP-1 solutions available through Crux.