Prediction, Not Understanding

This is the single most consequential idea in the entire AI4RE syllabus, and educational objective 2.2 asks you to be able to explain it: LLMs work by probabilistic next-word prediction, not by reasoning or understanding.

When people try a Large Language Model for the first time, the answers can feel human. The text is fluent, the tone fits the question, and the result often seems to show comprehension. The real story is simpler. LLMs are statistical text prediction systems that generate one word at a time based on patterns learned from training data. They do not understand and they do not reason. They match patterns and continue writing.

Given some words, the model picks the next small chunk that is most likely. The syllabus offers the mental model to hold onto: a very powerful autocomplete. That one idea explains most of the behaviour you will encounter.

How the mechanism works

The model is trained on an enormous quantity of text. During training it sees many examples of how words follow each other in different situations, from casual chat to technical writing. Its goal never changes: for any given context, guess the next token — a small piece of text that may be a whole word, part of a word, or punctuation.

Later, when you ask a question, the model uses what it learned to continue your text one token at a time, each choice conditioned on everything that came before.

This works well because language is full of patterns. Grammar creates structure. Common phrases repeat across topics. Many documents follow a predictable shape — and requirements documents are among the most predictable of all. A system that is very good at continuing these patterns produces output that feels coherent and relevant.

Modern models use an architecture called a transformer. A key part of it, self-attention, helps the model decide which parts of your input matter most for the next step. That is why a model can tie a pronoun back to the right name, or keep track of a topic introduced many sentences earlier, without any explicit rule telling it to.

What the system is not doing

The syllabus is careful about the negative claim, and so should you be, because this is where exam questions live.

The model is not thinking like a person. It has no inner picture of the world. There is no intent or understanding behind the sentences. It recognises and reproduces patterns it has seen across many texts. Because those patterns are rich and varied, the result can look like understanding — but the mechanism remains statistical prediction.

The clearest demonstration is arithmetic. Ask a model to perform a long multiplication and it may output a number that looks entirely reasonable and is wrong. It did not calculate anything. It predicted a number that statistically follows the pattern of a multiplication problem. To handle deterministic tasks — tasks that follow a clear algorithm — some models call a tool in the background, such as a calculator or a code runner, while the language model handles the instructions and the explanation. The model continues to predict text; the tool does the deterministic work.

People sometimes ask a model to show its steps. This can genuinely improve results on tasks that benefit from structure, and the reason is worth understanding: the model has seen many examples of step-by-step explanations in its training data — textbooks, tutorials, question-and-answer material — so it can reproduce that format. Reasoning systems go further and run private planning steps before answering, drafting and revising internally before presenting a result. This can raise quality. It does not change the core. The system is still predicting text, just arranged into several rounds before you see it.