[Hiring] AI Evaluation Guidelines & Rubric Specialist @24-MAG

Role Description

We are sharing a specialised full-time consulting opportunity for US-based linguists, instructional designers, and technical writers experienced in developing clear evaluation guidelines, structured rubrics, and human-rating instructions for generative AI programmes. This role supports a high-impact generative AI initiative focused on translating complex and potentially ambiguous programme requirements into precise, practical guidance for human evaluators. Selected professionals will develop rater-ready instructions across domains such as finance, retail, insurance, legal, and sports while resolving contradictions, defining edge cases, and improving consistency throughout evaluation workflows.

Key Responsibilities

• Rater Guideline Development
  • Translate programme requirements into clear, structured, and actionable instructions for human evaluators
  • Develop guidelines that can be applied consistently across standard scenarios and complex edge cases
  • Define terminology, rating criteria, decision rules, exceptions, and escalation pathways
  • Ensure instructions are accessible to raters while preserving necessary domain-specific precision
• Rubric & Evaluation Framework Design
  • Design detailed scoring rubrics for evaluating generative AI outputs
  • Establish measurable criteria covering correctness, relevance, reasoning quality, completeness, and instruction adherence
  • Create examples and counterexamples illustrating different performance levels
  • Align evaluation frameworks with programme objectives and quality standards
• Ambiguity & Consistency Review
  • Review draft specifications for ambiguity, contradiction, missing information, and inconsistent terminology
  • Identify instructions that may lead to conflicting interpretations across raters
  • Revise guideline sets until they can be applied reliably with minimal escalation
  • Document concrete before-and-after improvements to written requirements and evaluation instructions
• Cross-Domain Instructional Translation
  • Convert specifications from finance, retail, insurance, legal, sports, and other specialist domains into rater-ready guidance
  • Collaborate with subject matter experts to understand domain-specific terminology and professional judgment
  • Preserve important technical nuance while making instructions clear to non-specialist evaluators
  • Maintain consistent structure and quality across multiple domain-specific guideline sets

Qualifications

• At least 3 years of professional experience in linguistics, instructional design, technical writing, content design, or a closely related field
• Direct experience developing or refining guidelines and rubrics for human evaluators in generative AI, RLHF, or model-assessment programmes
• Demonstrated ability to resolve ambiguity and contradiction in complex written specifications
• Experience translating specialist requirements into clear and practical instructions
• Ability to work effectively across multiple subject-matter domains
• A portfolio or concrete examples showing measurable improvements to guidelines, rubrics, or instructional materials
• Demonstrable professional growth and increasing responsibility
• Reliable availability for at least 35 hours per week during weekdays

Requirements

• A degree in linguistics, instructional design, education, communications, technical writing, language studies, or a related field is highly relevant
• Graduate-level education in applied linguistics, learning design, human-computer interaction, or information design may be helpful
• Equivalent professional experience in AI evaluation, technical documentation, or guideline development may also be considered
• Training in assessment design, taxonomy development, content strategy, or quality assurance may be valuable

Nice to Have

• Experience supporting large language model evaluation, reinforcement learning from human feedback, or AI training-data programmes
• Familiarity with annotation platforms, human-feedback workflows, and rater calibration processes
• Experience developing domain-specific guidance for finance, insurance, retail, legal, sports, or comparable fields
• Knowledge of controlled language, information architecture, taxonomy design, or content governance
• Experience conducting guideline usability tests or analysing inter-rater consistency
• Familiarity with version control, documentation systems, and structured authoring tools
• Previous collaboration with researchers, programme managers, engineers, and subject matter experts

Why This Opportunity

• Apply linguistic and instructional-design expertise to an advanced generative AI initiative
• Influence the clarity and reliability of human evaluation processes
• Develop guidelines used across a wide range of professional subject-matter domains
• Solve complex problems involving ambiguity, edge cases, and evaluation consistency
• Join a full-time remote engagement with competitive hourly compensation

Contract Details

• Full-time W-2 contingent employment arrangement
• Fully remote role available to candidates based in the United States
• Expected commitment of at least 35 hours per week during weekdays
• Competitive rates between $40–$60 per hour depending on expertise and project scope
• Direct experience developing rater guidelines or rubrics for generative AI or RLHF programmes is required
• Applicants should be prepared to provide concrete examples of guideline or specification improvements
• Immediate availability is preferred
• Work may include onboarding, calibration, documentation review, and ongoing guideline refinement
• Project scope and duration may be adjusted according to programme requirements and performance