[Hiring] Medical Evaluation Specialist @YO AI Labs

Role Description

We are seeking Medical Evaluation Specialists, including medical students, residents, physicians, and qualified biomedical professionals, to contribute clinical expertise to a project focused on evaluating and improving next-generation AI systems.

In this role, you will:

• Create and validate challenging medical questions and answers designed to test advanced clinical reasoning.
• Help assess whether AI systems can accurately interpret medical evidence, synthesize complex information, and apply nuanced clinical judgment.

No prior AI experience is required—your medical knowledge, research skills, and clinical reasoning are what matter most.

Qualifications

• Current or recent medical training or clinical practice as a medical student, resident, or physician, or equivalent expertise in a relevant biomedical discipline.
• Demonstrated ability to locate, interpret, compare, and synthesize information from primary research and clinical guidelines.
• Strong written English skills with exceptional attention to detail.
• Ability to create engaging, well-structured, challenging, and clinically nuanced questions.
• Strong understanding of evidence-based medicine and clinical reasoning.
• Ability to work independently, manage deadlines, and consistently deliver high-quality work in a remote environment.
• Previous experience with medical question writing, peer review, clinical education, medical content evaluation, or research is a plus.
• Familiarity with AI/ML systems, AI evaluation, or medical AI applications is advantageous but not required.

Requirements

• Create original, high-difficulty medical question-and-answer pairs covering areas such as diagnosis, pathophysiology, pharmacology, clinical guidelines, and clinical decision-making.
• Develop questions that require synthesis, interpretation, and genuine medical reasoning rather than simple factual recall.
• Research and verify answers using primary literature, clinical guidelines, systematic reviews, and authoritative medical references.
• Document the rationale behind answers and provide appropriate supporting citations.
• Evaluate AI-generated responses for clinical accuracy, completeness, reasoning quality, and adherence to available evidence.
• Identify questions that are too straightforward and refine them to increase complexity while maintaining clinical validity.
• Review questions and answers for ambiguity, unsupported assumptions, factual errors, and inconsistencies.
• Ensure all deliverables are written with clarity, precision, and defensibility.
• Incorporate reviewer feedback and adapt work to evolving project guidelines and quality standards.
• Contribute insights that help establish rigorous benchmarks for medical AI evaluation.

Benefits

• Compensation is output-based, with contributors paid per task that meets the applicable project specifications and quality standards.
• The time required to complete each task may vary depending on experience and individual workflow.
• Minimum submission requirements apply, and selected contributors are expected to complete a minimum number of tasks per week.

Start Timeline & Availability

• Roles are typically filled within 48 hours.
• Selected contributors should be prepared to begin their first assignments within 24–48 hours of completing onboarding.