[Hiring] Senior AI Quality Engineer @Roofr

Role Description

Roofr is building the application foundation that will define how AI is integrated across our entire product — and we need someone to make sure it actually works. As our Senior AI Quality Engineer, you'll sit on the AI Platform team and own the eval frameworks, testing standards, and quality gates that every engineering team at Roofr depends on to ship AI with confidence. This role extends well beyond the platform — you'll work horizontally across the entire testing organization, training teams, embedding best practices, and raising the bar on how Roofr tests everything it builds with AI. This is early, foundational work — you're setting the standard for the whole org, not just one team.

What You'll Get to Do:

• Define the testing standards and patterns for AI at Roofr — establishing how product teams validate AI behaviour when building on top of the application foundation.
• Build and own Roofr's LLM eval framework — selecting and extending the right tooling (e.g. Promptfoo, DeepEval, Braintrust) and designing the methodology that measures whether our AI integrations and agent outputs are performing correctly, consistently, and safely.
• Integrate quality gates into CI/CD pipelines so that regressions in AI behaviour are caught before they reach production.
• Design and implement human-in-the-loop review processes for AI outputs where automated evaluation isn't sufficient.
• Embedded on the AI Platform team — ensuring quality is designed into the integration architecture from day one, not bolted on after the fact.
• Work horizontally across the testing organization — coaching QA engineers and developers on AI eval patterns, embedding best practices into team workflows, and actively raising the quality bar across engineering.
• Stay close to the evolving AI quality landscape — new eval techniques, benchmarking approaches, and tooling like Ragas, Arize Phoenix, or LangSmith — and bring the best of it to Roofr.

Qualifications

• 5–8 years of software engineering or quality assurance experience.
• Hands-on experience building eval frameworks for LLM-powered features.
• Strong engineering fundamentals — you write real code, build real tooling, and aren't reliant on manual testing processes.
• Experience integrating automated quality checks into CI/CD pipelines.
• Familiarity with LLM APIs and agent frameworks (e.g. Anthropic Claude, OpenAI, or similar).
• Experience designing human review workflows to complement automated evaluation.
• Strong collaboration skills — you'll be working across many teams.
• Comfort operating in an early-stage environment where the right approach isn't always obvious.
• Genuine ownership mentality — you care about whether AI at Roofr works well, not just whether the tests pass.

Benefits

• 1st week of employment is mandatory PTO.
• 1 Friday off per month (laundry days).
• Company-wide paid shutdown for the week between Christmas and New Years.
• Flexible time off.
• 80% employer-paid benefits in the U.S. and 100% employer-paid premiums for Extended Healthcare and Dental in Canada.
• RRSP/401k match.
• Generous Parental Leave policy.
• Annual company retreat with great team building activities.
• Ample learning and development opportunities.
• Home office setup stipend.
• Internet and phone allowance.
• Remote first culture.
• Weekly Friday paydays!