[Hiring] AI Evaluator @the Opportunity

Role Description

We are looking for AI Evaluators to assess the accuracy, usefulness, and reasoning of a digital shopping assistant across real-world e-commerce interactions. You will review user conversations, identify errors and weaknesses in the assistant's responses, and develop structured evaluation criteria that can be used to measure and improve response quality.

• Review real user interaction traces with an AI shopping assistant.
• Identify inaccurate, illogical, incomplete, or unhelpful responses.
• Evaluate the quality and relevance of product recommendations.
• Analyze conversations to identify specific reasoning or response failures.
• Create structured rubrics for consistently evaluating future responses.
• Develop verifiers to assess response quality against defined criteria.
• Complete evaluation work through the internal platform for 20+ hours per week.

Qualifications

• Experience in AI evaluation or AI training.
• Experience in data evaluation or annotation.
• Experience in quality assurance.
• Experience in e-commerce.
• Experience in prompt engineering.
• Experience in software testing.
• Strong analytical and attention-to-detail skills.
• Comfortable deeply reviewing text-based interactions.
• Able to develop structured evaluation frameworks.

Requirements

• Experience in data evaluation, quality assurance, AI training, or a related field.
• Strong ability to identify subtle errors and inconsistencies in written responses.
• Familiarity with e-commerce search and digital shopping experiences.
• Ability to analyze AI-generated responses critically.
• Ability to commit to 20+ hours per week on an ongoing basis.

Benefits

• Independent contractor engagement.
• Fully remote, with work completed on your own schedule.
• $40/hour, paid weekly based on services rendered.
• Engagements may be extended, shortened, or concluded early depending on project needs and performance.
• Work must not involve confidential or proprietary information belonging to any employer, client, or institution.
• H-1B and STEM OPT candidates cannot be supported at this time.

Apply

If you have experience in AI evaluation, quality assurance, e-commerce, data annotation, prompt engineering, or software testing and can commit 20+ hours per week, this remote opportunity offers the chance to directly contribute to improving an AI-powered shopping assistant.