[Hiring] Principal QA Engineer @Hive Financial Systems

Role Description

This is a hands-on, high-ownership role. You will build and own the QA function at Peach Pilot — writing test code, designing eval pipelines, and setting the quality bar as we move from early-stage development into full production and enterprise deployment. We are not looking for someone who manages spreadsheets and delegates everything. We are looking for someone who can do the work, knows what good looks like, and raises the bar across the entire engineering team.

This is a fully remote contract role based in Latin America. As the company scales, there is a path to a larger leadership role. For now the focus is getting the product right.

You will work directly with the US-based founding engineering team and must be available during US Eastern business hours with a minimum of 5 hours of daily overlap.

The challenge: QA for AI is a different problem. Traditional QA assumes deterministic outputs. LLMs don't give you that. You will be building a quality function from scratch in an environment where:

• Multi-model routing (Claude, GPT-4o, Grok, Gemini) means the same input can produce different outputs depending on which model handled it.
• Agent orchestration and governance agents must maintain a structurally separate audit trail; any drift between execution and governance is a critical failure.
• The file ingestion pipeline (Word, Excel, PowerPoint, PDF) must survive edge cases that enterprise clients will find within the first week of deployment.
• Your users are CEOs and operations leaders who have never used a terminal; a confusing error state isn't a minor bug, it kills adoption.

Qualifications

• 7+ years of QA engineering experience with at least 3 years in a lead or senior role where you both wrote test code and owned quality outcomes.
• Hands-on experience testing LLM-powered applications.
• You understand prompt sensitivity, output variance, and how to build eval pipelines that catch regressions across model updates.
• You write test code; Python is your primary tool.
• Experience building and maintaining CI/CD-integrated test suites.
• Comfortable testing complex API chains, async/streaming responses, and multi-service workflows.
• Built or significantly improved a QA function in an early-stage or fast-moving environment.
• Strong English communication skills, written and verbal.
• Available during US Eastern business hours with a minimum of 5 hours of daily overlap.

Requirements

• Experience with LLM evaluation frameworks such as LangSmith, PromptFlow, or custom eval pipelines.
• Experience testing agent frameworks such as LangChain or CrewAI.
• Background in enterprise software or regulated industries where audit trail integrity is non-negotiable.
• Insurance industry background is a strong plus.

Benefits

• Competitive contractor rate commensurate with experience.
• Paid monthly via Deel in USD.

The Clincher

Tell us about a quality failure — one you caught before it shipped, or one that got through. What did you build or change after it, and how did you make sure your team could catch the next one without you?