Inference Performance
Keep the GPU busy
Scroll
Keep the GPU busy
GPUs are efficient when they work on many things at once, so requests are batched. The naive version waits for a whole batch to finish before starting the next one.
Continuous batching keeps the batch open. As soon as one request completes its slot is freed and a queued request drops straight into it, mid-flight.
Without this, a batch runs at the speed of its slowest member and the hardware idles while short requests wait for a long one. With it, requests of wildly different lengths can share the same run — which is most of the difference between a demo and a service.
Serving