MODULE 5 · LESSON 1
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolThe Compute Stack
Training and inference are different businesses
The distinction most product people blur, and almost everything downstream depends on it.
Training builds the model. It is enormously compute-intensive, runs for weeks or months on large clusters of tightly interconnected accelerators, happens rarely, and is not time-sensitive to the minute. It is a capital-intensive project.
Inference runs the model to answer a request. Each one is small compared with training, but there are millions of them, they are latency-sensitive, and they happen continuously. It is an operating cost.
For almost every product company, you do not train and you do inference constantly. Which means the infrastructure conversation that matters to you is the inference one: latency, availability, regional placement and cost per request — not the training-cluster discussions that dominate industry coverage.
The layers
Between your product and the physical silicon:
Most product teams operate at the top two layers and can safely ignore the rest — until they cannot. The moments that force you down the stack are: a latency requirement, a data residency requirement, a capacity constraint, or a cost problem. Those are precisely the subjects of this half of the course.
Why inference behaves differently from training
Three practical consequences worth carrying:
Memory is often the binding constraint, not compute. Large models must fit in accelerator memory, and the model's size plus the working memory for the request determines what hardware can serve it. This is why smaller models are disproportionately cheaper — they fit on less expensive hardware and more requests share a device.
Batching creates a latency–cost trade-off. Serving many requests together uses hardware far more efficiently, but a request may wait for the batch. Providers manage this for you, and it is part of why a cheaper tier is often slower — you are being batched more aggressively.
Inference capacity is regional and finite. A provider has so much serving capacity in a given region. Under load, you experience it as rate limits, queueing or elevated latency. This is the origin of the reliability work in module 8.
Which layer is your problem actually in?
A diagnostic worth applying before any infrastructure work:
- "Answers are wrong" — the top layer. A prompt, retrieval or evaluation problem. Nothing below helps.
- "Answers are slow" — could be model size, could be batching tier, could be geography. Measure where the time goes before acting; teams routinely optimise a prompt when the latency was network round trips.
- "We hit rate limits at peak" — provider capacity allocation. A commercial conversation and a fallback design problem, not an engineering one.
- "Cost per request is too high" — model choice, input size, routing. Module 4's levers.
- "We cannot serve this customer" — data residency. Module 6, and no amount of engineering resolves it.
The common error is treating every problem as an engineering problem at your own layer. Several of these are commercial or legal problems wearing technical clothing.
Teams reach for self-hosting for three reasons — cost, control, and residency — and it is worth being precise about when it delivers.
On cost. Self-hosting replaces a per-request price with a fixed cost for hardware or reserved capacity that you pay whether or not you use it. That is a good trade only at consistently high utilisation. Most products have peaky demand: busy weekday afternoons, quiet nights and weekends. Paying for a GPU at 3am to save on inference you were not doing is a worse deal than it appears, and utilisation is the number that decides it. The calculation is real and worth doing — it just usually comes out against you until volume is both large and steady.
On control. Self-hosting genuinely gives you version stability — the model cannot change beneath you, which module 3 identified as a distinctive risk. That is a real benefit. But you have exchanged it for the obligation to run inference infrastructure: capacity planning, failover, upgrades, monitoring, on-call. That is a team, and it is a team not building your product.
On residency. This is the strongest case, and often the only one that survives scrutiny. If you must process data in a jurisdiction where no provider offers a suitable endpoint, hosting is the answer — and module 6 covers when that genuinely applies rather than when it is assumed.
The middle path most teams should take: use a provider, but architect so the model is swappable — an internal interface with the provider behind it, evaluation that runs against any backend, and prompts not entangled with one vendor's quirks. That preserves the option to self-host later without paying for it now, and it is the same portability argument module 8 makes about dependency.
A product team reports that answers are slow. What should be established before any optimisation work?
Training versus inference
Click to flipTraining is a rare capital-intensive project on large clusters; inference is a continuous latency-sensitive operating cost. For most product companies the infrastructure conversation that matters is the inference one.
Click to flip backTraining and inference are different businesses, and yours is inference: latency-sensitive, continuous, regional and finite. Know the layers between your application and the silicon, and recognise that only four things force you down the stack — latency, residency, capacity and cost. Diagnose which layer a problem actually lives in before working on it, because rate limits are commercial, residency is legal, and both routinely arrive disguised as engineering problems.