[Hiring] AI/ML Evaluation Reviewer @mpathic

Role Description

mpathic provides independent, multidisciplinary evaluations of healthcare AI technologies seeking deployment in real-world clinical settings. As part of our work supporting Utah's Office of Artificial Intelligence Policy (OAIP) regulatory sandbox, we are recruiting experienced machine learning (ML) and data science professionals to serve as technical reviewers on expert evaluation panels.

In this role, you will independently assess the technical evidence supporting healthcare AI applications, including:

• Model performance
• Validation methodology
• Generalizability
• Reliability
• Proposed monitoring and safeguards

Your assessments will help inform recommendations regarding whether and under what conditions an AI technology may be appropriate for pilot deployment.

You will work alongside clinical and other subject-matter experts to evaluate whether applicants' technical claims are supported by credible evidence and whether their proposed evaluation and monitoring plans are appropriate for the intended healthcare setting and patient population.

This is an evaluation and scientific review role, not a model development position. We are looking for experts who can critically assess existing evidence, identify limitations, and communicate actionable recommendations to technical and non-technical stakeholders.

Qualifications

• At least 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience evaluating or validating predictive models or AI systems.
• Demonstrated understanding of model evaluation methodologies, including appropriate performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error.
• Experience assessing model performance on independent or held-out data and interpreting whether reported performance generalizes beyond the evaluation dataset.
• Hands-on experience evaluating at least one relevant class of AI systems, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models.
• Ability to critically assess technical research, evaluation reports, or validation studies and identify methodological limitations or unsupported conclusions.
• Strong analytical writing skills, including the ability to communicate complex findings clearly to non-technical audiences.
• Ability to work independently while contributing to multidisciplinary expert review and calibration processes.
• Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and reviewer training requirements.

Requirements

• Experience evaluating AI or ML systems used in healthcare, clinical decision support, digital health, or other high-stakes environments.
• Familiarity with clinical AI evaluation and reporting frameworks, such as TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards.
• Experience evaluating generative AI, LLM-based systems, or autonomous AI agents, including multi-turn behavior, reliability, hallucinations, and technical safeguards.
• Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis.
• Knowledge of production monitoring, distribution shift, model drift, version control, and change-management practices.
• Experience designing or evaluating human-in-the-loop systems or comparing model performance against human expert judgments.
• Familiarity with FDA guidance for AI-enabled medical devices, clinical decision support, or software as a medical device.
• Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations.
• Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment.

Benefits

• This is a contract, per-review opportunity.
• Reviewers will be assigned to applications based on their technical expertise and the specific AI systems being evaluated.
• Assignments will involve reviewing applicant materials, documenting findings using mpathic's standardized evaluation framework, and participating in review discussions or calibration activities as needed.
• Reviewers will receive orientation to the evaluation framework and project-specific requirements prior to their first assignment.