How AI Works

Attention

Every token decides which others matter

Scroll

Every token decides which others matter

Words change meaning with context. In the trophy didn’t fit in the suitcase because it was too big, the thing that was too big is the trophy; change the last word to small and it becomes the suitcase. The model only learns which one was meant when it reaches the final word.

Attention is the mechanism that handles this. At each position, a token scores every earlier token for relevance and blends in information from the high scorers. The scores are worked out fresh for every input, so big leans on trophy while small, in the same slot, leans on suitcase.

Real models run many attention operations side by side, called heads, each free to track a different kind of relation — which noun a pronoun refers to, which verb goes with which subject. The cost is that every token compares itself with every earlier token, which is why long inputs get expensive and why the KV cache exists.

Foundations
WHERE THE LAST WORD LOOKS trophy fit suitcase it was big small each row is a word, each tower is how much it attends to an earlier one the last row is the last word weights are illustrative big: trophy 0.58 · it 0.21 · suitcase 0.09 small: suitcase 0.55 · it 0.22 · trophy 0.10 Swap one word at the end and the same slot attends to a different noun. Attention is how each token pulls in the context it needs from the tokens before it.
Towers showing how much each word attends to the words before it. Swap big for small and the last word's tallest tower moves from trophy to suitcase.