[Hiring] Medical AI Auditor @Medix

Role Description

We are seeking experienced healthcare clinicians to help train and evaluate the next generation of frontier Artificial Intelligence models. Please note: This is not a direct patient care or clinical hospital position. Instead, you will apply your clinical judgment, real-world case experience, and diagnostic acumen to push AI models to their limits, evaluate complex clinical reasoning, and ensure AI systems are accurate, safe, and aligned with evidence-based standards.

In this role, you will:

• Design realistic clinical scenarios (e.g., complex SOAP notes, medication reconciliations, triage decisions, discharge summaries, or clinical protocols).
• Run them through AI models and critically grade the outputs to identify where the systems succeed or fail.

Key Responsibilities

• Design Realistic Clinical Scenarios: Author authentic, de-identified patient scenarios, prompts, and supporting clinical materials (labs, imaging reports, histories, medication lists) drawn from standard day-to-day practice.
• Model Evaluation & Failure Identification: Review AI-generated clinical plans, summaries, and recommendations against strict professional standards. Identify concrete clinical failures, such as missed drug interactions, unsupported diagnoses, diagnostic anchoring, omitted red flags, or fabricated/hallucinated data.
• Comparative Output Analysis: Compare competing AI model outputs generated from identical clinical prompts and data, select the superior response, and provide clear written rationales for your choice.
• Develop Grading Rubrics: Write detailed, evidence-based evaluation rubrics establishing what a correct clinical response must contain and explaining why an output passes or fails each criterion.
• Collaborative Peer Review: Review, refine, and provide feedback on clinical test cases created by other medical domain contributors across various specialties.

Qualifications

• Active or Recent Clinical Licensure: Licensed MD, DO, NP, PA, PharmD, RN, or allied health professional with independent clinical judgment (e.g., Physical Therapy, Respiratory Therapy, Midwifery). (Active license verification may be requested).
• Clinical Experience: 5+ years of hands-on, direct clinical experience preferred (practical, hands-on individual contributors with fewer years will also be considered).
• Communication & Reasoning: Exceptional written English skills with the ability to articulate complex medical reasoning clearly, explaining why an AI answer is clinically unsound rather than simply stating that it is incorrect.
• Technology Proficiency: Familiarity with general AI/LLM tools (e.g., ChatGPT, Claude) and the ability to differentiate between plausible-sounding text and medically sound conclusions. Comfortable navigating cloud tools (e.g., Google Workspace, Microsoft Office), handling standard files, and managing desktop/browser environments.
• Analytical Thinking: Strong attention to detail, comfort working independently with ambiguous prompts, and the discipline to fact-check statements against underlying medical evidence.

Preferred Specialties

• Primary Care / Family Medicine / Internal Medicine
• Emergency Medicine / Critical Care / Acute Care
• Cardiology, Oncology, Neurology, or Specialty Pharmacy
• Pediatrics, Obstetrics, Behavioral Health, Geriatrics, or Perioperative Care

Benefits

• Total Schedule Flexibility: Work on your own schedule from anywhere within eligible locations; tasks allow for flexible multi-day pacing.
• Direct Impact on Healthcare Tech: Help shape the safety, ethics, and reliability of clinical AI tools before they reach clinicians and patients.
• High-Value Compensation: Competitive hourly project rates with no weekly cap on hours for strong performers.
• Model Access Included: Necessary platform and model tool access are fully provided.

Application & Onboarding Process

• Initial Screening: Review of your clinical background and licensure.
• Clinical Skills Assessment: Complete a self-paced, domain-specific evaluation testing your ability to review AI clinical outputs and identify nuanced medical errors (all assessment work must be your own original thought without generative AI tools).
• Project Onboarding: Upon successful assessment completion, you will receive orientation materials and gain access to paid clinical evaluation projects.