[Hiring] Technical Product Manager @CENTRL

Role Description

We are looking for an owner for the accuracy, reliability, and cost-efficiency of the AI behind CentrlX, our agentic platform for Manager Research and Investor Relations teams at investment firms.

You will own the model and prompt layer end to end. Responsibilities include:

• Building evaluation datasets and harnesses to assess changes.
• Running structured comparisons across model providers on accuracy, latency, and cost.
• Driving changes into the product based on evaluations.
• Creating and designing agents and skills.
• Working closely with sales and PS to drive standards for prompting and agents.
• Owning AI Quality and being the point person for improvements.

This is a product role, not a research role. We want the analytical rigor of a data scientist paired with the judgment of a product manager.

Key Responsibilities

• Evaluation & AI Quality
  • Build and own CentrlX's evaluation foundation: golden datasets, grading rubrics, LLM-as-judge pipelines, and regression suites.
  • Define quality metrics for core workflows: document digitization, retrieval, Smart Summary, Smart Response, Smart Evaluation, and multi-step agent runs.
  • Evaluate agent behavior, including tool selection and retrieval quality.
  • Turn client failures into permanent evaluation cases.
• Model Selection & Cost Optimization
  • Benchmark models across providers on accuracy, latency, and cost.
  • Own model migrations end to end, including moving off GPT-4.1.
  • Track and manage AI spend by workflow.
  • Maintain a working view of the model landscape and provide recommendations.
• Instrumentation & Data
  • Define logging and tracing requirements.
  • Partner with Product and Design for in-app feedback capture.
  • Build and maintain datasets, ensuring honesty with holdouts and rotation.
• Ownership & Cross-Functional Partnership
  • Serve as the point of contact for AI quality escalations.
  • Write user stories and acceptance criteria for changes.
  • Publish regular quality and cost readouts for leadership and teams.
  • Work with CENTRL's practitioners to encode domain judgment into rubrics.

Qualifications

• Must have work authorization in the USA.
• 3+ years in product management, data science, or applied AI, with at least 2 years on LLM-based products in production.
• Hands-on experience with Python and SQL.
• Experience building evaluation datasets for LLM systems.
• Working fluency with at least one eval or LLM observability platform.
• Practical understanding of RAG systems.
• Statistical literacy.
• Ability to write clear user stories and acceptance criteria.
• Strong written communication skills.

Preferred Qualifications

• Experience with document AI.
• Experience evaluating agentic systems.
• Experience in financial services or investment management.
• Experience reducing inference cost at scale.
• Experience designing in-product feedback mechanisms.
• Familiarity with agent frameworks and MCP.
• Degree in a quantitative or technical field.