[Hiring] AI Operations Specialist @Pavago

Role Description

We are looking for a hands-on AI Operations Specialist to own and improve our production AI conversation systems. Our AI systems interact directly with customers, and their performance directly impacts client conversions.

• Onboard new clients into the AI conversation platform.
• Set up webhook integrations and client knowledge bases.
• Configure campaigns and conversation workflows.
• Ensure new client systems are properly tested before going live.
• Troubleshoot integration and configuration issues during onboarding.
• Monitor live AI-generated conversations and customer interactions.
• Review conversations to identify errors, inconsistencies, and missed opportunities.
• Diagnose AI failure modes and determine their root causes.
• Identify issues affecting conversation quality and conversion.
• Escalate technical issues when necessary and follow through until resolved.
• Write, edit, and improve prompts that generate customer-facing messages.
• Adjust system instructions and conversation logic based on real-world performance.
• Test changes systematically before and after deployment.
• Maintain documentation of prompt and system changes.
• Continuously improve AI behavior while maintaining accuracy and consistency.
• Analyze conversation transcripts and performance data.
• Measure whether system and prompt changes improve conversion.
• Identify patterns across conversations that can inform future improvements.
• Track key performance outcomes and communicate findings to leadership.
• Use data and real customer interactions to guide optimization decisions.
• Work directly with the founder on daily AI operations and improvements.
• Communicate issues, findings, and recommendations clearly.
• Take ownership of problems from identification through resolution.
• Move quickly while maintaining high standards for customer-facing AI output.

Qualifications

• Proven experience running LLM or AI systems in production.
• Hands-on experience monitoring and improving production AI outputs.
• Experience with prompt engineering and system prompt optimization.
• Strong understanding of LLM behavior, failure modes, and iterative testing.
• Experience analyzing AI-generated conversations or customer-facing outputs.
• Ability to work with webhooks, APIs, knowledge bases, or similar integrations.
• Strong analytical and problem-solving skills.
• Excellent written English — required for editing customer-facing AI messages and documenting system changes.
• Strong attention to detail.
• Ability to work independently and take ownership of production systems.

Requirements

• You've actually operated AI systems in production rather than only experimenting with ChatGPT.
• You are comfortable reviewing hundreds or thousands of AI interactions to identify patterns and problems.
• You think in terms of outcomes and conversion, not just whether the AI response sounds good.
• You are methodical about testing changes and measuring their impact.
• You can quickly identify why an AI system is behaving incorrectly and determine how to improve it.
• You communicate clearly and document your work thoroughly.
• You enjoy working directly with a founder in a fast-moving environment.

Interview Process

• Initial Recruiter Screening
• Client Interview
• Offer Stage