[Hiring] Customer Support Specialists, AI Response Evaluation @mpathic

Role Description

mpathic is seeking part-time Customer Support Specialists to support a red-teaming and quality assurance (QA) campaign focused on evaluating AI system behavior in customer-facing interactions. In this role, you will review AI-generated responses and multi-turn conversations to assess quality, accuracy, tone, and safety—similar to evaluating customer support agents. This is not a traditional support role. Instead, it focuses on evaluating AI performance, applying your customer support expertise to assess how well AI systems handle real-world customer interactions.

You will help identify, prevent, and characterize risks that emerge when users interact with AI-powered support systems. Responsibilities may include:

• Reviewing AI-generated customer support responses for accuracy, clarity, and helpfulness
• Evaluating tone, empathy, and professionalism in customer interactions
• Identifying poor support behaviors (e.g., dismissiveness, lack of resolution, incorrect guidance)
• Assessing whether AI responses appropriately address customer intent and resolve issues
• Evaluating how AI handles frustration, confusion, or escalating user behavior
• Identifying gaps in responses, including missing steps, unclear instructions, or lack of follow-through
• Detecting overconfident, misleading, or incorrect responses
• Evaluating multi-turn conversations for consistency, escalation handling, and resolution quality
• Identifying patterns of agreement without problem-solving (e.g., empty empathy or unhelpful reassurance)
• Participating in or reviewing red teaming exercises to surface failure modes in support scenarios
• Supporting quality assurance (QA) to ensure consistency across evaluations
• Documenting edge cases, failure patterns, and customer experience risks
• Providing structured written feedback to internal teams
• Collaborating with interdisciplinary teams on AI quality, safety, and evaluation frameworks
• Maintaining strict confidentiality and quality standards

This role requires strong judgment, attention to detail, and the ability to evaluate nuanced customer interactions.

Qualifications

• Detail-oriented and customer-focused
• Experienced in evaluating or delivering high-quality support in fast-paced environments
• Professional experience in one or more of the following:
• Customer support (chat, email, or phone)
• Call centers or contact centers
• Technical support or help desk roles
• Online support environments (SaaS, e-commerce, fintech, etc.)
• Strong understanding of:
• High-quality customer experience and service standards
• Effective communication in customer-facing interactions
• Common support workflows (triaging, troubleshooting, escalation)
• How customers interpret and respond to support interactions
• Ability to identify:
• Poor or incomplete resolutions
• Incorrect or misleading information
• Inappropriate tone (e.g., robotic, dismissive, overly casual)
• Lack of empathy or ineffective de-escalation
• Failure to follow support best practices
• Strong written communication skills and ability to clearly explain reasoning
• Experience with or interest in:
• Evaluating conversations or QA for support teams
• Working with AI tools, chatbots, or automated systems
• Assessing how systems perform under real customer behavior
• Comfort with:
• Tech tools and platforms (Slack, spreadsheets, dashboards)
• Evaluating AI-generated responses (no coding required, but must be tech-comfortable)
• Ambiguity, iteration, and feedback-driven workflows
• Willingness to:
• Sign NDAs and work with sensitive content

Nice to Have (Not Required)

• Experience in QA, coaching, or training within support teams
• Background in trust & safety or content moderation
• Experience with chatbot or AI-assisted support tools
• Familiarity with CX metrics (CSAT, QA scoring, resolution rate)
• Interest in AI, automation, or improving customer experience systems

Compensation

• $30-40/hour, depending on experience and specific project tasks/difficulty

Location

• Seattle, Washington (Remote)

Department

• Experts

Employment Type

• Contractor

Minimum Experience

• Mid-level