MODULE 8 · LESSON 1
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolMetrics and the Operating Cadence
Four families, not one number
Watching only accuracy is the most common operating mistake, and it is what lets a system be technically healthy and commercially useless for a year.
1. Technical performance. Accuracy or its equivalent, on a fixed evaluation set. Also the distribution of what arrives — if inputs change, the number was measured on a world that no longer exists.
2. Operational health. Volume processed, latency, error and failure rates, exception volume. Exception volume is the one to watch: it is your permanent operating cost, and a rise means either the world changed or the system is deteriorating.
3. Adoption and behaviour. The proportion of eligible cases going through the system, the override rate, and — most revealing — who is using it. Adoption concentrated in two enthusiasts is not adoption.
4. Business outcome. The thing the business case promised. Hours, cycle time, error rate, coverage. This is the one that is hardest to measure and the only one that answers whether it was worth doing.
Most organisations measure the first two, because the technical team produces them automatically, and never establish the last two — which are the manager's job and the ones the business case was written in.
Measure against the baseline you captured
Module 4 insisted on a baseline before the project began. This is where it earns its place.
"92% accurate" is not a result. "92% against a human baseline of 89%, measured on the same 300 cases" is. Without the comparison, any number can be presented as success or failure depending on who is presenting.
If you did not capture a baseline, capturing one now is still worth doing — you can measure current human performance on a sample even after deployment, provided some cases still go to people, and the exception path guarantees they do.
Cadence proportionate to risk
Not everything needs the same attention:
Put the review in a calendar with a named owner. Reviews that depend on someone remembering do not happen, and the systems that most need review are precisely the ones nobody is thinking about.
Metrics that mislead
Aggregate accuracy hiding subgroup failure. Covered in AI for Non-Technical Teams: a system at 93% overall may be near-useless for a minority. Break performance down whenever the system affects people.
Adoption measured as logins. People log in and do not use it. Measure completed actions through the system as a proportion of eligible cases.
Override rate read as a single signal. A high rate means the system is wrong or distrusted. A rate near zero may mean it is excellent, or that reviewing has become rubber-stamping. Both extremes need investigating.
Time saved reported by users. People are poor estimators of their own time and tend to report what they think you want. Measure the process, not the perception.
Cost per month without volume. A bill rising because usage is rising is success; a bill rising at flat volume is a problem. They look identical on the invoice.
The whole operating discipline fits on a page. Same shape every month, so changes are visible.
System: Delivery-note extraction · Owner: [name] · Month: [date]
| | This month | Last month | At launch | Baseline | |---|---|---|---|---| | Field accuracy (held-back set) | 93% | 93% | 94% | Human 91% | | Volume processed | 3,410 | 3,180 | 2,900 | — | | Exception rate | 21% | 19% | 18% | — | | Adoption (eligible cases through system) | 88% | 87% | 61% | — | | Override rate | 6% | 6% | 11% | — | | Run cost | £1,240 | £1,150 | £980 | — | | Cost per 1,000 items | £364 | £362 | £338 | — | | Hours saved (modelled) | 290 | 271 | 148 | Case: 265/mo |
Three questions, answered in writing:
- What changed, and do we know why? Exception rate up two points over two months. Suspected cause: one supplier changed layout in the second week.
- Is anything trending the wrong way for three periods running? Exception rate — now on its second period. Watch, and act if it continues.
- Is the business outcome still tracking the case? Yes: 290 hours against 265 promised.
Actions: Confirm the supplier layout change and add samples to the evaluation set. Owner, by [date].
Two things make this work rather than being a ritual. The at-launch and baseline columns never change, so drift is visible without anyone having to remember what good looked like. And cost per unit sits alongside total cost, so a rising bill can be immediately identified as growth or as a problem — the distinction the invoice alone cannot show.
Note also that this page would surface the exception-rate creep long before anyone noticed the operating cost quietly rising, which is exactly the kind of slow failure that otherwise takes a year to detect.
A live system's monthly cost has risen 30% over three months. What single additional figure determines whether this is a problem?
The four metric families
Click to flipTechnical performance, operational health, adoption and behaviour, and business outcome. Most organisations measure the first two automatically and never establish the last two.
Click to flip backMeasure four families, not one number: technical performance, operational health, adoption and business outcome — the last two being the manager's job and the ones the business case was written in. Always report accuracy against the baseline you captured beforehand, review weekly for the first eight weeks and by risk thereafter, and watch exception volume as the earliest signal of drift. Never track cost without volume, because growth and escalation look identical on an invoice.