MODULE 6 · LESSON 1
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolPilots That Produce a Decision
The defining question
Before a pilot starts, answer this: what result would cause us to stop?
If there is no answer, it is not a pilot. It is an implementation with an escape hatch, and it will proceed regardless of what it shows — which is how organisations acquire systems nobody decided to buy.
A pilot needs three things fixed in advance:
A decision rule. "If field-level accuracy on the held-back set exceeds 85% and exception volume stays below 20%, we proceed to phase two. Below either, we stop."
A time box. Six to ten weeks. Long enough to be real, short enough that stopping is not a personal defeat.
A named decider. One person who will say proceed or stop on the date, ideally not the person who proposed it.
Shadow mode first
Introduced in AI for Non-Technical Teams and worth repeating because it is under-used: run the system on live data, record what it would have done, and let humans continue working unchanged.
You get an accurate comparison against real human performance, on real inputs, at no operational risk and with no adoption question confusing the measurement. It costs very little and it settles the argument everyone is having.
Two things shadow mode reveals that a test set never will:
- Input drift you did not know about. The documents arriving this month differ from the historical sample in ways nobody mentioned.
- The genuine human baseline. Frequently uncomfortable. Teams often discover their people disagree with each other more than they expected, which reframes the whole project — you may be replacing inconsistency rather than accuracy.
Pilot patterns that cannot scale
Four ways a pilot succeeds and teaches you nothing, all of which look like progress:
The first is the most common and the most damaging, because the manual data assembly is invisible in the result. A pilot fed by hand proves the model works and says nothing about whether your organisation can supply it, which is the harder question.
Fix: even in a pilot, make the data arrive the way it will in production, even crudely. A scheduled job that fails twice a week is more informative than a perfect manual extract.
Pilot purgatory
The state where an organisation runs pilot after pilot and never puts anything into production. The causes are recognisable:
- No decision rule, so nothing ever conclusively passes.
- Success criteria that drift upward. 87% prompts "can we get 92?" indefinitely.
- No budget line for production. The pilot was funded; the operational deployment was not, so success creates a funding request nobody anticipated.
- No owner for the running system. Everyone wants to build it; nobody wants to be responsible for it in eighteen months.
- Each pilot uses a different vendor and a different approach, so nothing accumulates.
The structural fix is to fund the pilot and the production deployment together, conditionally. "£40,000 for the pilot; £120,000 released automatically if the decision rule is met." This removes the moment where success has to be re-justified from scratch, which is where most pilots quietly die.
A wholesaler piloting automated extraction of delivery-note data, and worth reading as a template.
Week 0. Decision rule written and signed: proceed if field-level accuracy on the held-back set exceeds 90% for supplier, date and total, and if the exception rate is under 25%. The operations director decides on the Friday of week 6. Held-back set of 300 notes assembled and given to nobody.
Weeks 1–2. Vendor configured on 1,200 historical notes. In parallel — and this was the decision that made the pilot useful — an internal developer built a crude scheduled job that pulled new notes from the scanning folder automatically. It broke repeatedly, and every break was information about a production problem they would otherwise have met in month five.
Weeks 3–4. Shadow mode. The system processed every incoming note; staff continued keying manually; both results were stored. This produced the pilot's most valuable finding, which nobody had anticipated: the human baseline was 96% on supplier and date but 91% on totals, because staff transcribed the pre-VAT figure when the layout was unfamiliar. Nobody had known that. The AI project had found a process defect worth fixing regardless of the outcome.
Week 5. Held-back evaluation: 94% on supplier, 97% on date, 92% on totals. Exception rate 18%. The awkward pile clustered in two suppliers whose layouts changed monthly.
Week 6. The operations director proceeded, with two conditions from what the pilot taught: the two volatile suppliers route to humans by default, and the totals field gets a reconciliation check against the purchase order rather than being trusted.
The pilot cost about six weeks and a small licence fee. It produced a decision, an exception design, a genuine baseline, a known production failure mode in the scheduled job, and an unrelated process improvement. Every one of those came from choices made in week 0 — the decision rule, the held-back set, and insisting the data arrive automatically even badly.
A pilot succeeds using data an analyst assembled by hand over two weeks. What has it actually demonstrated?
The defining pilot question
Click to flipWhat result would cause us to stop? Without an answer it is not a pilot but an implementation with an escape hatch.
Click to flip backA pilot is only a pilot if you have written down what result would stop it, by when, decided by whom. Start in shadow mode, which costs almost nothing and produces both a real comparison and an honest human baseline. Insist the data arrives automatically even crudely, because hand-assembled data hides the hardest production dependency. And fund pilot and deployment together, or success will simply create a funding request nobody planned for.