[Hiring] AI QA Trainer @Invisible Agency

Role Description

Are you an AI QA expert eager to shape the future of AI? Large-scale language models are evolving from clever chatbots into enterprise-grade platforms. With rigorous evaluation data, tomorrow’s AI can democratize world-class education, keep pace with cutting-edge research, and streamline workflows for teams everywhere. That quality begins with you—we need your expertise to harden model reasoning and reliability.

We’re looking for AI QA trainers who live and breathe:

• Model evaluation
• LLM safety
• Prompt robustness
• Data quality assurance
• Multilingual and domain-specific testing
• Grounding verification
• Compliance/readiness checks

You’ll challenge advanced language models on tasks like:

• Hallucination detection
• Factual consistency
• Prompt-injection and jailbreak resistance
• Bias/fairness audits
• Chain-of-reasoning reliability
• Tool-use correctness
• Retrieval-augmentation fidelity
• End-to-end workflow validation

On a typical day, you will:

• Converse with the model on real-world scenarios and evaluation prompts
• Verify factual accuracy and logical soundness
• Design and run test plans and regression suites
• Build clear rubrics and pass/fail criteria
• Capture reproducible error traces with root-cause hypotheses
• Suggest improvements to prompt engineering, guardrails, and evaluation metrics (e.g., precision/recall, faithfulness, toxicity, and latency SLOs)
• Partner on adversarial red-teaming, automation (Python/SQL), and dashboarding to track quality deltas over time

Qualifications

• Bachelor’s, master’s, or PhD in computer science, data science, computational linguistics, statistics, or a related field
• Shipped QA for ML/AI systems
• Safety/red-team experience
• Test automation frameworks (e.g., PyTest)
• Hands-on work with LLM eval tooling (e.g., OpenAI Evals, RAG evaluators, W&B)

Requirements

• Evaluation rubric design
• Adversarial testing/red-teaming
• Regression testing at scale
• Bias/fairness auditing
• Grounding verification
• Prompt and system-prompt engineering
• Test automation (Python/SQL)
• High-signal bug reporting
• Clear, metacognitive communication—“showing your work”

Benefits

• Pay range of $6 to $65 per hour, with the exact rate determined after evaluating your experience, expertise, and geographic location
• Final offer amounts may vary from the pay range listed above
• As a contractor, you’ll supply a secure computer and high-speed internet
• Company-sponsored benefits such as health insurance and PTO do not apply

Company Description

• Employment type: Contract
• Workplace type: Remote
• Seniority level: Mid-Senior Level