Inference Performance
Splitting one model across many devices
Scroll
Splitting one model across many devices
When a model's weights and cache do not fit on one accelerator, the model has to be split across several. There are two basic ways to cut it, and they have different costs.
Tensor parallelism splits every layer. Each device holds a slice of every weight matrix and a share of the attention heads, and all the devices compute their part of each layer at the same time. They then exchange partial results before the next step can start, which happens twice in every transformer block. That traffic needs a very fast link between devices, so tensor parallelism usually stays inside one server. In return, several devices work on each token at once, which cuts latency.
Pipeline parallelism splits the stack instead. The first device holds the first group of layers, the next device the next group, and each hands its activations on. The traffic is light, so the stages can sit in different servers. But one request still passes through the stages one after another, so it gets no faster, and devices sit idle unless enough requests are in flight to keep every stage busy. Large deployments often combine the two: tensor parallelism inside each server, pipeline parallelism across servers, and full copies of the whole arrangement to handle more traffic.
Inference