MODULE 5 · LESSON 2
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolLatency, Regions and Physics
The floor you cannot engineer past
Light travels at about 300,000 km/s in vacuum, and roughly 200,000 km/s in optical fibre because glass slows it. That gives a simple and unavoidable rule:
Every 1,000 km of distance costs about 5 ms one way, or 10 ms round trip — before any equipment does anything at all.
Worked through for real distances, as a theoretical minimum on a perfectly straight route:
| Route | Approx. distance | Theoretical round trip |
|---|---|---|
| London → Frankfurt | ~640 km | ~6 ms |
| London → New York | ~5,600 km | ~56 ms |
| London → Mumbai | ~7,200 km | ~72 ms |
| London → Sydney | ~17,000 km | ~170 ms |
Real-world figures are meaningfully higher, because fibre does not follow great circles, routes detour around geography and political boundaries, and every switch and router adds delay. A practical rule is to expect somewhere around 1.5 times the theoretical figure.
No engineering removes this. You cannot cache your way past it for a request that must reach a model, and no amount of optimisation makes Sydney closer to London. The only remedy is to move the compute nearer the user.
What makes up an AI request's latency
Total latency is a sum, and knowing the components tells you which are addressable:
Two practical consequences follow immediately.
Long prompts cost latency as well as money. The module 4 lever of sending less context improves both, which makes it unusually high-value.
Streaming changes the experience without changing the total. If you stream, the user sees the first token after network plus queueing plus time-to-first-token, rather than waiting for the whole generation. Perceived latency falls dramatically while measured total latency is unchanged. For any conversational surface this is close to mandatory.
Placing inference
The decision is about where your users are, weighed against where capacity and permission exist.
One region is right when your users are geographically concentrated, or when the interaction is not latency-sensitive — a document processed in the background does not care about 150 ms.
Multiple regions become necessary when a latency-sensitive surface serves users on several continents, or when residency requires it, which module 6 covers.
Multi-region is not free, and the costs are mostly not the compute:
- Your knowledge base must exist in each region, or retrieval crosses the ocean and you have reintroduced the latency you moved to avoid.
- Model availability differs by region — a provider may not offer the same models everywhere, so behaviour can differ between your regions.
- Evaluation must run per region if the models differ, because otherwise you are testing one product and shipping another.
- Operational complexity rises: deployments, monitoring, incidents and cost tracking all multiply.
Co-locating retrieval with inference is the point most often missed. A team moves inference to Singapore for Asian users, leaves the vector database in Europe, and every request now crosses the world twice. The measured improvement is negative and the cause is invisible unless you break latency down by component.
A team is building a voice interface. Natural conversation tolerates roughly 300 ms of gap before it feels broken, and the pipeline is speech-to-text, then language model, then text-to-speech — three stages, each taking time.
Their users are in Australia. Their infrastructure and provider capacity are in the United States.
The physics alone rules it out: Sydney to the US west coast is on the order of 12,000 km, giving a theoretical round trip near 120 ms and a realistic one meaningfully higher. Add three model stages and the budget is gone before anything has been computed.
Options, in the order a team should consider them:
Move the compute. Serve from an Australian or nearby region if the provider offers the models there. This is the only option that addresses the actual cause, and its availability is a provider question rather than an engineering one.
Shorten the pipeline. A model handling speech directly, without separate transcription and synthesis stages, removes two hops. Fewer stages is often a bigger win than optimising any one of them.
Change the interaction so the latency is not felt. Start speaking before the full answer is generated. Use a short acknowledgement while the substantive answer is produced. This is a design solution to a physics problem, and it is frequently the most effective one available.
Change the product. If none of the above suffices, the honest conclusion is that a real-time voice interface is not deliverable to this market from this infrastructure, and an asynchronous interaction — a message rather than a conversation — is the product that can actually be shipped.
The generalisable lesson: when a requirement conflicts with physics, the fix is almost never optimisation. It is relocating the compute, removing a stage, redesigning the interaction, or changing the promise. Teams lose months to optimisation before reaching that conclusion, and the calculation that would have shown it takes ten minutes.
A team moves inference to a Singapore region to serve Asian users, but leaves the vector database in Europe. What happens and why is it easy to miss?
The latency floor
Click to flipLight travels about 200,000 km/s in fibre, so every 1,000 km costs roughly 5 ms one way and 10 ms round trip before any equipment acts. Real routes run around 1.5 times the theoretical minimum.
Click to flip backDistance imposes a latency floor no engineering removes: about 10 ms round trip per 1,000 km in fibre, and roughly 1.5 times that in practice. Break total latency into its components, because only some are addressable and teams routinely optimise the wrong one. Stream output to change perceived latency, send less context to improve both cost and time-to-first-token, and if you move inference closer to users, move retrieval with it — otherwise you have relocated the problem rather than solved it.