[Hiring] Subject Matter Experts: ML Model Evaluation @Weekday AI

Role Description

This role is for one of Weekday’s clients.

• Engagement Type: Independent Contractor
• Work Arrangement: Fully Remote
• Estimated Commitment: Approximately 5 Hours
• Schedule: Flexible, Asynchronous
• Payment Frequency: Weekly, based on services rendered
• Pay: $150 / hr
• This requirement is super-urgent & would be awesome if you can start on the project within Friday EOD.

We are seeking highly qualified Subject Matter Experts with PhD-level academic or professional expertise across specialised research and professional domains to participate in a paid Machine Learning (ML) Model Evaluation project. The project focuses on assessing the quality, accuracy, and reliability of AI-generated responses to complex research prompts and domain-specific questions. Your evaluations will help identify where current machine learning models perform effectively and where their outputs require further refinement.

This opportunity is suited to experienced researchers, academics, data analysts, technical writers, and domain specialists who can critically assess complex information, identify factual inaccuracies, and provide structured, evidence-based feedback.

Qualifications

• PhD or equivalent advanced subject matter expertise in a relevant academic, technical, or professional discipline.
• Strong academic or professional background in a specialised domain, with the ability to assess complex information critically.
• Excellent analytical, research, and critical-thinking skills.
• Ability to evaluate technical content, research findings, and domain-specific information for factual correctness and relevance.
• Strong written communication skills, particularly in providing clear and structured evaluation justifications.
• High attention to detail and the ability to identify subtle inaccuracies, inconsistencies, and gaps in AI-generated responses.
• Ability to work independently and complete approximately five hours of asynchronous evaluation tasks.

Requirements

• Applicants must be from any of these geographies: US, UK, Canada, the EU, Norway, Switzerland, and Iceland.
• Should have a finished/running paper in any of these topics: NeurIPS, ICML, ICLR, COLM or RLC.
• We are unable to support H1-B or STEM OPT candidates at this time.

Benefits

• Contract Type: Independent contractor engagement.
• Work Arrangement: Fully remote, with flexible working hours.
• Project Duration: Approximately five hours of evaluation work.
• Payment: Payments are processed weekly based on services rendered.
• Project Continuity: The engagement may be extended, shortened, or concluded early depending on project requirements and individual performance.
• Confidentiality: Participation will not require access to confidential or proprietary information belonging to any current or former employer, client, or academic institution.

Interview and Evaluation Expectations

During the interview process, candidates may be assessed on their review practices, critical thinking, and ability to improve the quality of their evaluations. Responses should address the following three habits:

• Exchanging unfinished review notes with another reviewer.
• Maintaining a separate log of opinions or assessments that have changed during the review process.
• Using review notes and feedback to update personal reading lists.

Ideal Candidate Profile

The ideal candidate combines deep subject matter expertise with strong research judgement and analytical precision. They should be comfortable reviewing complex AI-generated content, distinguishing accurate information from flawed reasoning, and communicating their assessments through clear, structured feedback.