How AI Works
Attention and a feed-forward step, stacked
Scroll
Attention and a feed-forward step, stacked
The transformer is the architecture behind essentially every current LLM. It has two ingredients, and both have already appeared: attention, where tokens exchange information, and a small neural network that processes each token on its own, called the feed-forward step.
One attention step plus one feed-forward step makes a block. A model is that block repeated dozens of times, each copy with its own weights. Roughly speaking, earlier blocks pick up surface patterns and later ones build more abstract features. The last block's output becomes a score for every token in the vocabulary, and that is the next-token prediction.
What it had over older designs was parallelism. During training, attention lets every position in a passage be processed at once instead of one after another, which is what made training on vast datasets practical. Much of how LLMs are built and served — caching, batching, context limits — follows from this structure.
Foundations