[Hiring] Shopify Expert @Mercor

Role Description

Evaluate the Storefront Agent by simulating real shopper interactions, including:

• Product discovery
• Comparisons
• Inquiries about sizing, stock, shipping, and returns

Grade responses based on:

• Relevance
• Accuracy
• Safety

Provide concise feedback on any failures to ensure clarity for team members not present during evaluation.

Work independently to deliver precise and repeatable judgments on agent performance.

Qualifications

• Must-Have:
• Consumer e-commerce fluency — you shop online regularly and understand effective product recommendations.
• Careful, consistent judgment — apply the rubric uniformly across responses.
• Clear, concise written English — articulate reasons for response failures in a sentence or two.
• Preferred:
• Hands-on Shopify merchant experience — familiarity with shopper behavior on Shopify stores.
• Experience using AI shopping assistants or chat agents — understand common failure points.
• Prior work evaluating, red-teaming, or annotating AI model outputs.
• Familiarity with retail operations: catalog and variant structure, inventory, shipping, and returns policy.

Company Description

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include:

• Benchmark
• General Catalyst
• Peter Thiel
• Adam D'Angelo
• Larry Summers
• Jack Dorsey