Inference Performance

GPU memory math: what fits where

Weights are fixed; the cache decides how many users fit

Scroll

Weights are fixed; the cache decides how many users fit

Serving a model is mostly a memory problem. An accelerator's memory has to hold three things: the weights, the KV cache for every request in flight, and working space for activations and the serving runtime. The weights are a fixed cost. The cache grows with every request and every token.

Take an 8-billion-parameter model stored in 16-bit precision. At 2 bytes per parameter, the weights take 16 GB. With 32 layers and 8 key-value heads of size 128, the cache costs 2 × 8 × 128 × 2 bytes per layer, times 32 layers: 128 KiB per token, or about 1.07 GB for one 8,192-token sequence. On an 80 GB accelerator, after the weights and a reserve of about 6 GB for activations and overhead, roughly 58 GB is left. That holds about 54 such sequences at once.

That figure sets how many users one device can serve at this context length, and every lever moves it. Storing the weights in 8 bits frees 8 GB, for about 61 sequences. Storing the cache in 8 bits halves its cost per token and allows about 108. A shorter context limit raises the count directly. The arithmetic also shows when one device stops being enough: at 70 billion parameters, the weights alone take 140 GB in 16-bit, before a single token of cache. The page on distributed inference covers splitting a model across devices.

Inference
WHAT FITS ON ONE 80 GB ACCELERATOR 8B model · 16-bit · 8,192-token sequences · 6 GB reserve assumed weights 16 GB reserve 6 GB KV cache: 54 sequences, 58 GB 8-bit weights → 61 sequences 8-bit KV cache → 108 sequences 70B model, 16-bit → 140 GB of weights: does not fit Capacity is whatever the weights leave for the cache. Shrink either one and more requests fit; past one device's memory, the model has to be split.
An 80 GB accelerator filled with weights, a reserve, and the KV cache of 54 sequences.