How AI Works

Transformers

Attention and a feed-forward step, stacked

Scroll

Attention and a feed-forward step, stacked

The transformer is the architecture behind essentially every current LLM. It has two ingredients, and both have already appeared: attention, where tokens exchange information, and a small neural network that processes each token on its own, called the feed-forward step.

One attention step plus one feed-forward step makes a block. A model is that block repeated dozens of times, each copy with its own weights. Roughly speaking, earlier blocks pick up surface patterns and later ones build more abstract features. The last block's output becomes a score for every token in the vocabulary, and that is the next-token prediction.

What it had over older designs was parallelism. During training, attention lets every position in a passage be processed at once instead of one after another, which is what made training on vast datasets practical. Much of how LLMs are built and served — caching, batching, context limits — follows from this structure.

Foundations
ONE BLOCK, REPEATED attention tokens share information feed-forward each token, on its own every block is this one, with its own weights block 1 block 2 block 3 block N tokens in next-token scores Attention lets tokens share context; feed-forward works on each token alone. Stack that pair dozens of times and the last block's output becomes the prediction.
One block opened up beside a tower of the same block stacked many times.