01

A word does not travel alone

Consider “bank” in a sentence about a river, then in a sentence about a loan. The surrounding words change the interpretation. Self-attention is a mathematical mechanism that combines information from positions within a sequence. It can let a representation incorporate relevant context, including words that are not its immediate neighbours. This is a useful way into the idea, rather than a claim that a model reads as a person does.

02

Queries, keys and values

The Transformer introduced in Attention Is All You Need forms queries, keys and values from representations. Comparisons between queries and keys produce scores; a normalisation turns them into weights used to combine values. Several attention heads can learn different relationships. The important distinction is between a word printed on a page and its internal numerical representation. The toy bars below illustrate weighting; they do not reveal the internals of a live AI system.

03

Text generation adds another step

Language models work with tokens, which may be words, parts of words or other pieces of text. Decoder-only models generate a continuation using the preceding context, with causal attention preventing access to future tokens. Other configurations can look in both directions across an input. Attention is part of a larger architecture with additional transformations; it is not a complete explanation of every model behaviour.

04

Inspect the demonstration honestly

Switch between the river and lending examples. We deliberately assign a higher illustrative weight to the context word that distinguishes the meaning of “bank”. The three displayed weights sum to 100%, but they cover only three selected words, not a full sentence or a trained attention head. They are teaching values, not measured probabilities of a correct answer. Strong-looking connections do not establish factual accuracy. The useful question is what information a representation draws on—and what evidence supports the resulting output.

FOLLOW THE EVIDENCE

Sources & further reading

Sources checked 14 November 2026. Educational models are labelled with their assumptions. These organisations do not endorse this website.