[Hiring] Senior Product Manager @Pencil

Role Description

At Pencil, we are driving innovation in advertising technology through our state-of-the-art SaaS product, which harnesses Generative AI to redefine content creation. Our mission is to make AI the default in advertising without replacing creative people.

The role:

Pencil's agents produce advertising creative at scale - video, adaptation and multiple formats, orchestrated by our agentic system, Scribble. This role owns measuring quality and improving it.

• Own evals end to end, ensuring systems measure whether agents, skills, and workflows produce work a client would ship.
• Demonstrate quality to customers and the market through evidence behind client RFP responses and creative enablement packages.
• Build reliable measurement: rubrics that define quality, calibration between human reviewers and automated judges, and a golden dataset that evolves with the product and client base.
• Make metrics central to how the business operates and sells.

Key responsibilities:

• Own the charter for Evals & Quality: the durable problem, success metrics, and an explicit in/out scope list.
• Define core quality metrics: first-pass-right, coverage, regression detection, and drive them up.
• Own the judging methodology: rubrics for creative quality and the calibration process for automated judges and human raters.
• Design and run the human QC mechanism: review output, feed judgements to the automated layer, and scale the process.
• Own the golden dataset: curation, coverage across formats and client contexts, and versioning.
• Build eval loops for other teams to use independently.
• Own adevals.ai as a product, ensuring high design quality.
• Provide the evidence layer for commercial work: client RFP responses and creative enablement packages.
• Partner with Agent Architects on client-specific quality bars through configuration and rubrics.
• Run the full product lifecycle: Discovery through post-launch audit, with a written hypothesis before every ship and a review 14 days after launch.

Qualifications

• A quality, evals or ML measurement problem you owned as a product, with its own roadmap and users.
• Measurement you shipped in a subjective domain: how you defined quality, and how you kept the signal reliable as it scaled.
• Direct experience with the mechanics: golden datasets, rubric design, judge calibration, inter-rater reliability.
• A product decision derived from evidenced customer need in a domain you had to learn.
• A metric you owned that changed decisions outside your own team.
• Time spent in front of clients: presenting or defending methodology to buyers.

Requirements

• Move Fast: Comfortable making decisions with imperfect information and iterating quickly.
• Stay Human: Care deeply about users and teammates, designing with empathy.
• Act Like an Owner: Take responsibility for outcomes, spotting problems early.
• Keep It Simple: Fight unnecessary complexity and design reusable patterns.
• Love to Surprise: Find unexpected ways to delight users, especially in usability.

Benefits

• 25 days PTO plus public holidays, with a Flexible Time Off scheme.
• Health insurance / private medical cover.
• Monthly stipend towards wellness, fitness, and learning and development.
• Remote work from anywhere in your home country.
• Enhanced parental leave policies for birth, adoption, or surrogacy.
• Access to our Pencil office in The Shard, London for UK employees.
• Flexible working hours.