[Hiring] AI Agent Evaluator @Hippocratic AI

Role Description

We are seeking Nurses & Nurse Practitioners with clinical experience in one or more of the following areas:

• acute symptom management
• CHF and hypertension management
• diabetes management
• travel health

to serve as a subject matter expert supporting the evaluation of AI-driven clinical agents. In this role, you will evaluate agent conversations and recommendations spanning acute symptom triage and escalation decisions, insulin/diabetes titration, and pre-travel health guidance, ensuring clinical accuracy, guideline adherence, and patient safety. This is a part-time, remote contract position (1099) requiring up to 10 hours per week, with flexible scheduling.

Key Responsibilities

• Review recorded and simulated AI agent interactions involving acute symptom triage and assess the appropriateness of escalation and de-escalation decisions.
• Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence.
• Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment.
• Identify clinical safety concerns, edge cases, and gaps in the agent's triage logic or clinical reasoning.
• Score and annotate agent performance using structured evaluation rubrics.
• Provide clear, actionable written feedback to clinical and product teams to inform agent refinement.
• Participate in periodic case review and calibration sessions with other clinical SMEs to ensure consistent evaluation standards.
• Help develop and refine test scenarios used in ongoing agent quality assurance.

Qualifications

• Registered Nurses: active, unrestricted RN License
• Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (ANCC or AANP).
• Minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a comparable acute-care setting.
• Clinical experience managing patients with diabetes, including insulin titration.
• Strong, current working knowledge of relevant triage protocols and clinical practice guidelines.
• Ability to critically evaluate clinical reasoning and identify inaccuracies or safety risks in written or recorded clinical interactions.
• Comfortable working independently in a remote, asynchronous contract role.
• Strong written communication skills for delivering structured clinical feedback.

Recommended

• Experience in telephone triage or nurse advice line settings.
• Experience in travel medicine, occupational health, or pre-travel consultation.
• Familiarity with CDC and other travel medicine guidelines.
• Prior experience in clinical quality assurance, utilization review, or case review.
• Familiarity with AI-assisted clinical tools or digital health platforms.

Engagement Details

• Contract (1099) position, up to 10 hours per week.
• Fully remote; flexible scheduling.
• Hourly compensation, commensurate with experience.
• Engagement expected to run in support of ongoing agent development and FDA evaluation milestones.

Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.