MODULE 1 · LESSON 3
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolDiscovery: What to Validate Before Building
Three questions, in the right order
Desirability. Does anyone want the outcome? Note: the outcome, not the AI. Nobody wants an AI summary; they want to not read the document.
Feasibility. Can it be built to a standard that delivers that outcome? The distinctive AI question, and it has a specific shape covered below.
Viability. Does the economics work at the cost per request? Module 4.
Teams routinely start with feasibility, because a demo is exciting and cheap. That produces impressive capabilities nobody needed — the failure the previous course called an accurate model of something nobody wanted predicted.
The quality threshold
The question conventional discovery never has to ask: how good does this have to be before it is worth anything?
The relationship is rarely linear, and knowing its shape changes what you build:
Establish which you have before committing. Threshold products are dangerous to start, because there is no partial delivery — you either clear the bar or you have spent the budget for nothing. Gradient products can be shipped early and improved with real usage, which also generates the data that improves them.
Test value before building
The strongest discovery technique available, and it is startlingly underused: do it by hand first.
Before building a system that drafts responses, have a person write drafts and put them in front of users as though the system produced them. Before building a system that flags risky contracts, have someone flag them manually and see whether anyone acts on the flag.
This separates two questions that otherwise get answered together and confused:
- Is this valuable if it works perfectly? The manual version answers this at a fraction of the cost, because a human performs above any model you would build.
- Can we build it well enough? Only worth asking if the answer to the first is yes.
The result is often uncomfortable and always cheap: users frequently do not act on the perfect manual version either. In that case there was never a product, and you have learned it in a fortnight rather than after two quarters of engineering.
Impressive is not the same as valuable
A capability demonstration proves something can be done. A product requires that someone's behaviour changes because of it. The gap between those is where most AI products die.
Four questions that expose it:
- What does the user do differently? If the honest answer is "they are more informed", that is usually not a product.
- What were they doing instead, and what did it cost them? If the alternative is fine and cheap, you are competing with fine and cheap.
- Would they pay, or is this a feature of something they already pay for? Many AI capabilities are features, not products, and the distinction determines the whole business model.
- What happens the third time it is wrong? Novelty tolerance is high initially and drops sharply. Products are judged on their behaviour after the novelty has gone.
A team building a tool for accountancy practices proposed a feature that would read client emails and automatically raise the right task in the practice management system — a genuinely appealing idea, well received in early conversations.
Rather than build it, they ran a manual version for three weeks. Two team members watched a shared inbox at three participating firms and manually created the tasks a good system would have created. Users were told a new automated feature was in trial.
Three findings, none of which would have emerged from building it:
The task creation was not the bottleneck. Practice staff created tasks quickly; what took the time was deciding whether something warranted a task, which required knowing the client relationship. The manual operators, who lacked that context, got roughly a third wrong — and these were humans reading carefully. That was strong evidence about feasibility before a line of code.
Users disliked tasks appearing without their involvement. Not because they were wrong, but because the inbox was where they tracked their own thinking, and a task appearing from nowhere broke that. Several turned it off. This is a desirability finding that no accuracy improvement would have addressed.
The valuable version was different. What users responded to was a suggestion inside the email — "create task: chase VAT records for Client X?" — which they accepted or ignored. Same underlying capability, one click, and it preserved their sense of running their own inbox.
Three weeks and two people's part-time attention. The build would have been most of two quarters, and would have produced the version users turned off.
The generalisable move: the manual prototype tests the thing you cannot test with a model, which is whether the outcome is wanted and in what form. It also gives you, almost incidentally, a few hundred labelled examples produced by people trying to do the task well — the beginning of your evaluation set.
A team is considering an AI feature for medical triage, where a wrong classification could delay urgent care. What kind of product is this and what follows?
The order of discovery
Click to flipDesirability, then feasibility, then viability. Teams start with feasibility because a demo is cheap and exciting, which produces impressive capabilities nobody needed.
Click to flip backRun discovery in order — desirability, feasibility, viability — because starting with feasibility produces capabilities nobody wanted. Establish whether you have a threshold, gradient or ceiling product, since threshold products offer no partial delivery and must have feasibility resolved before commitment. Test value with a manual prototype before building anything: it answers whether the outcome is wanted, in what form, at a fraction of the cost, and it produces the beginnings of your evaluation set.