MODULE 7 · LESSON 1
Free — no login requiredSign in to track progress, save quiz attempts and enrol in the full course.
Sign in to track progress / enrolLanguage and Locale
Performance is not uniform
The assumption that a model performing well in English performs comparably elsewhere is wrong, and the gap is large enough to change product decisions.
Published multilingual benchmark work has documented performance differences of up to around 24 percentage points between high-resource and low-resource languages on the same questions — the same content, translated, answered less well. Benchmarks such as MMLU-ProX, introduced in 2025, exist specifically to make these disparities measurable, covering thousands of parallel questions across dozens of languages.
The cause is training data composition. Analysis of GPT-3 found roughly 92.65% of its training tokens were English, leaving about 7.35% for every other language combined. Later models are more multilingual, but the fundamental asymmetry persists: the volume of high-quality text available in English vastly exceeds what exists in most other languages, and instruction-tuning has historically concentrated in English too.
The practical statement to carry into product decisions: an English evaluation score does not predict performance in French, Arabic or Swahili, and the gap widens as language resources thin.
The tokenisation disparity
A second, less discussed effect with direct commercial consequences.
Models process text as tokens, and tokenisers are typically optimised for English. The same meaning expressed in another language frequently requires substantially more tokens — research has noted multipliers of around three times for languages such as Arabic.
Three consequences follow directly:
Cost. Per-token pricing means the same conversation costs several times more in some languages. Your unit economics from module 4 are language-dependent, and a flat global price can be unprofitable in exactly the markets you are expanding into.
Latency. More tokens means longer time to first token and longer generation. The module 5 latency budget is tighter in those languages, on top of being further from your compute.
Context capacity. A document that fits comfortably in a context window in English may not in another language, so your retrieval and chunking strategy behaves differently.
Evaluate per language
The operational consequence for module 3's discipline: an evaluation suite in one language measures the product in one language.
If you serve multiple languages, you need per-language evaluation. Practically:
- A core set per supported language, drawn from real usage in that language, not translated from English. Translated test sets miss the constructions, idioms and topics native users actually produce.
- Criteria reviewed by native speakers. Quality criteria written in English and applied to another language by a non-speaker will miss tone failures that are obvious to a native reader.
- Per-language reporting. A blended multilingual score conceals exactly the disparity you need to see, in the same way an aggregate accuracy figure conceals subgroup failure.
- A language-specific decision on what to support. It is more honest and better product management to support four languages well than eleven badly.
That last point deserves emphasis. Listing a language as supported when the experience in it is materially worse is a promise you are not keeping, and users in that market will conclude the product is poor rather than that the language is unsupported.
Translation is not localisation
Translating your interface is the smallest part of serving a market.
Localisation for an AI product additionally involves: the language the model responds in and how naturally; formality and register, which differ sharply by language and context; date, number, currency and address formats; names that the system must not mangle; regulatory language obligations in some sectors; and the retrieval corpus, which must exist in that language or answers will silently come from English sources.
That last one is a common and subtle failure: a product answering in French from a knowledge base that only exists in English produces a fluent French answer derived from a source the user cannot check, and citation — module 2's verification mechanism — breaks, because the cited document is in a language the reader may not have.
A team runs a customer-service assistant in English. They plan a Spanish launch, expecting it to be a translation task of a couple of sprints.
What they find, in the order they find it:
The model answers Spanish fluently. This is reassuring and misleading — fluency is the surface layer, and it is the strongest thing these systems do. It says nothing about correctness.
Evaluation on their translated test set scores well. Also misleading. Translated tests contain English-shaped questions rendered in Spanish. Real Spanish-speaking customers phrase things differently, raise different issues, and use regional variation the translated set has none of.
The knowledge base is entirely in English. The assistant retrieves English policy documents and answers in Spanish. Answers are largely right and completely unverifiable for the customer, and every citation links to a document they cannot read. Module 2's verification design has quietly stopped working.
Cost per conversation is meaningfully higher, because Spanish uses more tokens for the same content than English. Nobody modelled it, and the market with lower average revenue is also the one with higher unit cost.
Formality is wrong. The assistant uses an informal register with customers who expect a formal one, and the effect on perceived professionalism is significant and invisible to the English-speaking team.
Escalation to a human breaks. Tickets escalate to an English-speaking support team, so the one thing that should catch the assistant's failures cannot.
What the launch actually required: a Spanish evaluation set built from real Spanish enquiries, the knowledge base translated and maintained in Spanish, revised unit economics, register decided with native speakers, and Spanish-speaking escalation capacity.
The generalisable point: the model speaking the language is the easiest part, and it is the part teams mistake for the whole job.
A team's assistant answers fluently in a new language, and their translated evaluation set scores well. Why is this weak evidence of readiness?
Performance is not uniform
Click to flipPublished multilingual benchmarks have documented gaps of up to around 24 percentage points between high- and low-resource languages on the same translated questions. An English score does not predict performance elsewhere.
Click to flip backModel quality varies substantially by language, with documented gaps of up to around 24 points on identical translated questions, driven by training data composition. Tokenisation disparities raise cost, latency and context consumption simultaneously, making your unit economics language-dependent. Evaluate per language using sets built from real usage rather than translations, and remember that the model speaking the language is the easy part — the knowledge base, register, economics and escalation path are the work.