[Hiring] Director, AI Automation Engineering @Impiricus

Role Description

The Director, AI Automation Engineer is a player-coach role: own the quality and reliability layer for everything the AI Organization ships, and stay hands-on building the harnesses, eval frameworks, and release gates that make that real.

• Automated testing infrastructure, evaluation frameworks, and CI/CD guardrails for staging, testing, and managing AI agents and tools at scale.
• Coverage spans internal AI platform work (Sage and the tools built on it) and customer-facing AI across Ascend and DocUpdate.
• Sets the bar, does the hardest technical work early, and grows into leading a small automation team as coverage expands.
• Partnership with AI Engineers, AI Solutions Architects, and product teams to establish baseline quality metrics, automate validation, and make standards usable for engineers and non-engineers building in Sage.

Key Responsibilities

• Testing Harness and Infrastructure:
  • Own the infrastructure that continuously verifies whether autonomous agents are taking the actions we intend, before and after they reach production.
  • Stage, test, and lifecycle-manage bots and agent workflows before they reach production.
  • Establish baseline quality metrics and coverage standards for AI initiatives.
  • Build the infrastructure to run automated research chains and agent evals at scale.
• Evaluation Frameworks:
  • Create automated evaluation frameworks for LLM outputs: accuracy, safety, bias, and regression.
  • Implement RAG evaluation, prompt regression testing, and golden-set benchmarking.
  • Define and enforce quality thresholds that AI Engineers and builders must meet before ship.
  • Partner with AI Researchers and Engineers on eval design for new model and agent patterns.
• CI/CD and Release Quality:
  • Build the release-gating infrastructure that governs when an agent is trusted to act in production, including staged rollouts and rollback authority.
  • Develop AI automation solutions to ensure that agentic coding delivers highly functioning software while maintaining quality across the product portfolio.
  • Integrate testing into GitHub Actions (or equivalent) for AI model and agent releases.
  • Maintain existing test suites and keep them current as models, prompts, and tools change.
• Cross-Team Quality Partnership:
  • Work with AI Solutions Architects on eval criteria for workflow automations they build in the field.
  • Partner with Principal AI Product on launch-readiness and QA for AI projects across the org.
  • Support product teams (Ascend, DocUpdate, Platform) on AI feature validation without owning their roadmaps.
  • Document standards, runbooks, and patterns so non-engineers building in Sage can self-serve basic testing.

Qualifications

• 5+ years building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety.
• Direct experience evaluating LLM outputs: RAG eval, prompt regression, safety and accuracy benchmarking.
• Experience building CI/CD pipelines with integrated test gates (GitHub Actions or similar).
• Comfortable working across AI platform infrastructure and product-facing AI features.
• Operates independently as a senior IC and communicates quality standards to engineers and non-engineers.
• Experience with agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus.
• Familiarity with LLM observability and eval tooling (LangSmith, Braintrust, custom harnesses, etc.) is a strong plus.
• Background in healthcare, pharma, or regulated environments where quality gates matter is a strong plus.
• Prior experience standing up a QA or automation function from scratch, with appetite to grow into leading a team.
• Proficiency in Python and standard testing frameworks (pytest or equivalent).

Benefits

• Healthcare: Medical, dental, and vision coverage for you and your dependents + on-demand healthcare concierge.
• HSA, FSA & DCFSA: Pre-tax savings options for healthcare and dependent care, with monthly employer contributions to HSA (if enrolled in a high-deductible plan).
• Coverage & Protection: 100% paid short- and long-term disability, plus life and AD&D insurance.
• Flexible Time Off: Take the time you need with a flexible vacation policy — recharge your batteries your way.
• Parental Leave: Paid parental leave to spend time with your newborn, adopted, or foster child (available after 6 months).
• Your Work, Your Way: If you’re close to an office, we encourage spending some time in-person to collaborate and connect. If not, remote is always an option.
• 401(k): Save for your future with tax advantages (and company match!).