MODULE 1 · LESSON 2
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolThe AI Product Manager's Job
What carries over, and what does not
Most of conventional product management applies unchanged: understanding users, prioritising, defining scope, working with design and engineering, saying no.
Four things are genuinely new.
1. You own the definition of good. In conventional software, correct is obvious and the argument is about scope. Here, what counts as a good output is a product decision requiring judgement, and if you do not make it, an engineer will make it implicitly by choosing a metric.
2. You own the evaluation set. This is the single biggest departure and the one most often abdicated. The set of examples that defines acceptable behaviour is a product artefact, not a test fixture. It encodes what you promise. Module 3 is entirely about this.
3. You own the failure experience. What the user sees when the system is unsure, wrong, or out of scope is a large part of the product, and nobody else will design it unless you insist.
4. You own the boundary. What the product will not do, and how that is communicated. In conventional software, scope limits are roadmap items. Here they are safety properties.
The evaluation set is your specification
Worth stating directly: if you do not curate the evaluation set, you are not managing the product.
A team optimising against a set that does not represent your users will produce a system that scores well and disappoints. The set determines what gets improved, because teams improve what is measured. Whoever chooses the examples chooses the product.
Practically, this means an AI PM spends real time reading actual outputs on actual inputs — not dashboards, not summaries. There is no substitute, and the PMs who are good at this are the ones who have read a few thousand model outputs and developed an instinct for how their system fails.
Choosing between models that score the same
A recurring decision, and instructive because the tie-break is never technical.
When two options score within noise of each other, choose on:
- Failure mode. Which one fails in the way your users can tolerate? A model that says "I don't know" more often may score lower and serve better.
- Latency. Where the product is conversational, a slower better answer often loses to a faster adequate one.
- Cost per request. Module 4 — this decides your gross margin.
- Consistency. For some products, being reliably the same matters more than being occasionally better.
- Portability. How hard is it to move off this, covered in module 8.
None of these appear on a benchmark. All are product decisions.
Working with an unpredictable team
Managing AI development has one structural difference from conventional software: you cannot reliably estimate whether something will work, only whether it is worth finding out.
Three practices that make this workable:
Time-box investigations rather than scoping features. "Two weeks to find out whether we can get above 80% on this" is honest. "Build the classifier by the 14th" is not, and everyone knows it.
Separate the research question from the product work. Once feasibility is established, the remaining work — integration, interface, evaluation infrastructure, failure handling — is ordinary product engineering and estimable in the ordinary way. Frequently most of the schedule.
Make progress visible through evaluation, not through demos. A demo shows a cherry-picked case. A number on a fixed set shows movement. Weekly evaluation runs against the same set are the honest status report.
The highest-leverage habit in this role is unglamorous. Set aside an hour a week to read raw model outputs on real inputs, unfiltered and unsummarised.
Not the dashboard. Not the ten examples engineering chose. A random sample of what your system actually produced for actual users this week.
What this gives you, and nothing else does:
- You discover failure patterns, not failure rates. A dashboard says 8% of outputs were poor. Reading them tells you that most of those are one thing — the system mishandles messages containing two questions — which is actionable, whereas 8% is not.
- You calibrate your own judgement. After a few hundred outputs you develop an intuition for what this system does badly, which makes every subsequent roadmap decision better.
- You catch the embarrassing before a customer does. The output that is technically fine and tonally catastrophic. No metric detects it. A person reading does, immediately.
- You find the evaluation set's blind spots. Real inputs will include things your curated set never contemplated, and each one is a candidate to add.
- You keep your promises honest. It is very difficult to overclaim in a roadmap meeting about a system whose actual output you read on Friday.
The reason this is rare is that it feels like a low-status activity — it looks like QA rather than strategy. It is the opposite. It is the only direct access you have to your product's real behaviour, and every abstraction between you and it costs you information.
A practical version: sample 30 outputs weekly, sorted by confidence ascending so you see the shaky ones first, and keep a running note of patterns. Ten minutes a day, and it will do more for the product than any amount of dashboard-watching.
Two candidate models score within noise of each other on your evaluation set. On what basis should the choice be made?
The four new responsibilities
Click to flipOwning the definition of good, owning the evaluation set, owning the failure experience, and owning the boundary of what the product will not do.
Click to flip backMost of product management carries over; four things are new. You own the definition of good, the evaluation set — which is a product artefact encoding what you promise — the failure experience, and the boundary of what the product refuses to do. Choose between comparable models on failure mode, latency, cost and portability rather than on scores. And read raw outputs every week, because it is the only unmediated access you have to how your product actually behaves.