[Hiring] LLM - AI Quality Analyst @Careerflow.ai

Role Description

We're looking for sharp and detail-oriented analysts to join a global team evaluating a personalization feature for a leading AI assistant. In this role, you'll put the AI through its paces, testing how well it uses real personal context from past conversations, Gmail, Google Search, and YouTube activity to give genuinely helpful, tailored responses. This isn't a passive review job; you'll be designing your own prompts drawn from your actual experiences, then rigorously evaluating the AI's responses across multiple quality dimensions. Think of yourself as part product tester, part quality critic.

Responsibilities

• Design and run multi-turn conversational prompts (typically 1-5 turns) that challenge the AI to draw on your personal data and experiences.
• Assess whether the model understood your intent and applied personalization appropriately, or where it missed the mark.
• Evaluate Grounding: flag any claims the model makes about you that aren't supported by evidence, including hallucinations or flawed inferences.
• Evaluate Integration: check whether personal data is woven into responses naturally, or whether the AI comes across as robotic or over-explanatory.
• Conduct side-by-side (SxS) comparisons of two model responses, ranking them on helpfulness, usability, and overall quality.
• Write clear, structured rationales for your rankings citing specific turns in the conversation to back up your assessments.
• Verify "Debug Info" to confirm that chat summaries and external data sources were properly referenced.
• Keep your evaluation data clean by deleting test conversations after each session so they don't influence future results.

Qualifications

• Strong written and reading proficiency in Portuguese, this is the primary language for the project.
• Exceptional analytical thinking, particularly when it comes to evaluating nuanced or ambiguous AI-generated content.
• Willing to use your primary personal Google account (not a test account) and enable personal data sources for evaluation.
• Ability to design creative, context-rich prompts based on your own personal experiences.
• A keen eye for spotting subtle differences between model responses, things like over-narrating, forced connections, or weak personalization.
• Strong written communication skills. You'll need to write clear, defensible evaluation notes regularly.
• Comfortable working independently and staying self-directed in a fully remote setup.
• Reliable desktop or laptop with a solid internet connection.
• Full-time availability in your local time zone, with 4 hours of daily overlap with PST.

Preferred Qualifications

• Prior experience in data annotation, AI quality evaluation, content moderation, or a related field.
• A BS/BA degree (or equivalent experience) in a field like Linguistics, Journalism, Computer Science, Policy, Law, Ethics, or another analytically rigorous discipline.
• Familiarity with personalization concepts and an intuition for when AI responses feel genuine versus when they seem off.

Engagement Details

• Type: Short-term contractor engagement (2 months)
• Location: Remote - open to candidates in LATAM, USA, and Europe (excluding Portugal)
• Hours: Minimum 30 hours per week, with a 40-hour option available
• Overlap requirement: 4 hours per day during PST timezone
• Start date: Immediate

Hiring Process

• Shortlisted candidates will receive a Job Interest Form to confirm availability and fit.
• After profile review, you'll be sent a skills assessment to complete within 24 hours.
• Candidates who pass the assessment will be contacted to walk through pre-onboarding requirements.