Inference Performance
Weights are fixed; the cache decides how many users fit
Scroll
Weights are fixed; the cache decides how many users fit
Serving a model is mostly a memory problem. An accelerator's memory has to hold three things: the weights, the KV cache for every request in flight, and working space for activations and the serving runtime. The weights are a fixed cost. The cache grows with every request and every token.
Take an 8-billion-parameter model stored in 16-bit precision. At 2 bytes per parameter, the weights take 16 GB. With 32 layers and 8 key-value heads of size 128, the cache costs 2 × 8 × 128 × 2 bytes per layer, times 32 layers: 128 KiB per token, or about 1.07 GB for one 8,192-token sequence. On an 80 GB accelerator, after the weights and a reserve of about 6 GB for activations and overhead, roughly 58 GB is left. That holds about 54 such sequences at once.
That figure sets how many users one device can serve at this context length, and every lever moves it. Storing the weights in 8 bits frees 8 GB, for about 61 sequences. Storing the cache in 8 bits halves its cost per token and allows about 108. A shorter context limit raises the count directly. The arithmetic also shows when one device stops being enough: at 70 billion parameters, the weights alone take 140 GB in 16-bit, before a single token of cache. The page on distributed inference covers splitting a model across devices.
Inference