MODULE 4 · LESSON 2
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolThe Capability Ladder
The ladder
Five rungs, ascending in cost, control and commitment. The discipline is to solve the problem on the lowest rung that works.
Rungs 1 to 3 are the working range for the overwhelming majority of AI products, and they compose: a good product usually uses all three together.
What each rung actually buys
Prompting buys behaviour. Format, tone, what to refuse, how to structure an answer. It is also the rung where most quality problems are actually solved, which teams under-appreciate because it feels insufficiently substantial.
Retrieval buys knowledge, freshness and traceability. Your policies, your documents, today's data — with a citation. AI4RE covers the RAG-versus-fine-tuning comparison in depth; the short version is that anything a user might ask you to justify belongs in a retrievable source.
Tool use buys correctness on the things that must be correct. This is the architectural answer to module 1's separation problem: the balance comes from a function call to the system of record, and the model's job is to decide when to call it and how to present the result.
Fine-tuning buys consistency and, sometimes, cost. If you need output in a rigid house format across millions of requests, fine-tuning a smaller model can be cheaper per request than prompting a larger one — which makes it an economic decision as much as a quality one.
Training buys independence, at a price almost no product company should pay.
Why teams skip rungs
Two forces push upward prematurely.
Fine-tuning sounds like engineering; prompting sounds like typing. Moving up the ladder feels like real work and confers more status. This is the same dynamic the managers course names in build-versus-buy, and it responds to being named rather than to argument.
A prompt failure is blamed on the rung rather than the prompt. A poorly specified prompt produces poor output, the team concludes prompting is insufficient, and proposes fine-tuning — when the actual problem was that nobody supplied the context, the format or the examples.
The cost of skipping rungs is real. Fine-tuning locks you to a model, adds a retraining cycle whenever behaviour must change, requires a curated dataset, and makes evaluation harder because you can no longer swap the base model to compare. A change that would have been a prompt edit becomes a training run.
The test before moving up: can you demonstrate that a well-constructed version of the lower rung fails? Not that the first attempt failed — that a serious attempt did.
When moving up is justified
Genuine reasons, all of which produce evidence:
- Consistency of form matters more than flexibility, across high volume, and prompting has not achieved it.
- Cost per request would fall materially by using a smaller fine-tuned model at your volume, and you have modelled it.
- Latency requires a smaller model, and quality only holds with adaptation.
- The behaviour cannot be described, only demonstrated — you have thousands of examples and no way to write the rule.
That last one is the most legitimate and the least often cited. Some tasks are genuinely easier to show than to specify, and that is what fine-tuning is for.
A product that extracts structured data from supplier delivery notes. Watch what each rung costs and buys.
Rung 1 — prompting alone. A prompt describing the fields, the output format and the rules for ambiguity, with three worked examples. Two days of work. Reaches acceptable accuracy on the common layouts and fails on unusual ones. Cost per request is high because the examples and instructions are sent every time.
Rung 2 — add retrieval. For each incoming note, retrieve two or three previously processed examples from the same supplier and include them. Roughly a week of work. Accuracy on unusual layouts improves substantially, because the model now sees how this specific supplier's notes were handled before. Cost per request rises slightly — more input — but the failure rate falls, which reduces the expensive human exception path.
This is the rung most teams never reach, and it is often the one that solves the problem. It also improves automatically as the product is used, because the pool of prior examples grows.
Rung 4 — fine-tune. Take 20,000 processed notes and adapt a smaller model. Several weeks of work plus a data pipeline. Accuracy comparable to rung 2 on common cases, slightly better on rare ones. Cost per request falls meaningfully because the prompt is now short and the model smaller — which at high volume can be the whole justification.
But: the model is now pinned to a base you must maintain, adding a new field requires retraining rather than a prompt edit, and you have a dataset to curate indefinitely.
The honest conclusion. Rung 2 is right for most volumes. Rung 4 becomes right at high volume where the per-request saving exceeds the ongoing cost of maintaining the training pipeline — a calculation, not a preference. The team that jumps from rung 1 to rung 4 because rung 1 disappointed will spend weeks reaching an outcome that a week at rung 2 would have delivered, and will have locked themselves in to get there.
A team's prompted extraction fails on unusual document layouts and they propose fine-tuning. What should be tried first, and why?
The five rungs
Click to flipPrompting, retrieval augmentation, tool use and orchestration, fine-tuning, and training — ascending in cost, control and commitment. Solve on the lowest rung that works.
Click to flip backWork the ladder from the bottom: prompting buys behaviour, retrieval buys knowledge and traceability, tool use buys correctness on facts that must be right — and those three rungs cover most products, composing together. Teams skip upward because fine-tuning feels like real engineering and because bad prompts get blamed on prompting. Before moving up, demonstrate that a serious attempt at the lower rung failed, and be clear whether the reason is quality, consistency, latency or cost per request at your volume.