Inside the Model
How a model knows word order, and why it struggles past its training length
Scroll
How a model knows word order, and why it struggles past its training length
Attention compares tokens by their content, so on its own it has no sense of order: shuffle the words and each pair of tokens gets exactly the same score. Position has to be added. The original transformer added a fixed wave pattern to each token's embedding, and many later models learned a separate position embedding for each slot.
Most current open LLMs use rotary position embeddings instead. Before attention, each query and key vector is rotated by an angle proportional to its position, at a different speed in each pair of dimensions. When two rotated vectors are compared, the result depends only on the distance between the two tokens, which gives the model a sense of relative position.
Past the length it was trained on, a model meets distances it has never seen, and quality usually drops fast. Context extension methods rescale the rotation frequencies so that longer inputs map onto familiar angles, then fine-tune briefly on long documents. Even then, a long window is not used evenly: models tend to use material near the start and end of a long input better than material in the middle. Test retrieval from the middle of your own long inputs before you rely on an advertised context length.
Architecture