Inside the Model

Attention variants: MHA, GQA, MQA

Fewer key-value heads, a smaller cache

Scroll

Fewer key-value heads, a smaller cache

Attention runs many heads side by side. In its original form, multi-head attention, every head has its own query, key and value projections. During generation the keys and values for every past token stay in the KV cache, so the cache grows with the number of key-value heads, the number of layers and the length of the context.

Multi-query attention keeps all the query heads but makes them share one key head and one value head. Grouped-query attention sits in between: the query heads are split into groups, and each group shares one key-value head. Take a model with 32 layers and 32 query heads of size 128, storing 16-bit values. Multi-head attention needs 512 KiB of cache per token, or 4 GiB for a single 8,192-token sequence. With 8 key-value heads that falls to 1 GiB, and with one it is 128 MiB.

Generation is usually limited by how fast the cache can be read from memory rather than by arithmetic, so a smaller cache means faster tokens and more concurrent requests on the same hardware. The cost is capacity. Multi-query attention can lose quality and make training less stable; grouped-query attention keeps most of the multi-head quality at close to multi-query speed, which is why many recent open models use it. Some newer designs go further and compress keys and values into a smaller latent vector before caching them.

Architecture
HOW MANY KEY-VALUE HEADS THE QUERIES SHARE multi-query 1 key-value head KV cache ⅛× grouped-query 2 key-value heads KV cache ¼× multi-head 8 key-value heads KV cache 1× 8 query heads in each Fewer key-value heads, smaller cache per token: more requests and longer contexts fit in the same memory. Grouped-query usually keeps most of the quality.
Eight query heads sharing eight, two or one key-value heads, with the cache drawn to scale.