How AI Works
Every token decides which others matter
Scroll
Every token decides which others matter
Words change meaning with context. In the trophy didn’t fit in the suitcase because it was too big, the thing that was too big is the trophy; change the last word to small and it becomes the suitcase. The model only learns which one was meant when it reaches the final word.
Attention is the mechanism that handles this. At each position, a token scores every earlier token for relevance and blends in information from the high scorers. The scores are worked out fresh for every input, so big leans on trophy while small, in the same slot, leans on suitcase.
Real models run many attention operations side by side, called heads, each free to track a different kind of relation — which noun a pronoun refers to, which verb goes with which subject. The cost is that every token compares itself with every earlier token, which is why long inputs get expensive and why the KV cache exists.
Foundations