[Hiring] AI Response Quality Evaluator @Gramian Consulting Group

Role Description

We are seeking AI Personalization Evaluators to support Gemini Personalization Evals by reviewing AI-generated responses using personalized prompts and context retrieved from connected Google applications. You will assess response quality, contextual accuracy, and personalization, identifying subtle errors, unsupported assumptions, and opportunities to improve the user experience.

• CONTRACT: Contractor
• LOCATION: Remote — Australia only
• COMMITMENT: Minimum 4 hours per day, up to 40 hours per week
• TIME OVERLAP: 4 hours of overlap with PST
• DURATION: Up to 16 weeks
• EVALUATION PROCESS: Shortlisted candidates will receive a Job Interest Form. Selected candidates will be contacted regarding consent and onboarding requirements.

Key Responsibilities

• Evaluate AI-generated responses using personalized prompts and information retrieved from connected Google applications.
• Assess whether responses are relevant, accurate, helpful, and appropriately personalized.
• Identify incorrect personalization, unsupported assumptions, irrelevant recommendations, and inconsistencies.
• Compare multiple AI-generated responses and evaluate differences in overall user experience.
• Provide clear, detailed, and well-reasoned feedback to support AI model improvements.
• Follow project guidelines to ensure consistent and high-quality evaluations.
• Review nuanced AI responses and identify strengths, weaknesses, and subtle quality issues.
• Maintain data privacy and follow all confidentiality requirements throughout the project.
• Currently based in Australia.
• Willing and able to connect relevant Google apps to Gemini, subject to required consent and permissions.
• Active use of Google applications, with sufficient personal data or history to support personalized retrieval evaluations.
• Strong analytical thinking and attention to detail.
• Excellent written English and ability to provide structured evaluation feedback.
• Ability to work independently in a remote environment.
• Access to a desktop or laptop with a reliable internet connection.
• Bachelor’s degree or equivalent practical experience in any field.