What is pipeline parallelism?

A model too large for one GPU can be cut in 2 ways. Tensor parallelismTensor ParallelismSplitting each layer's weight matrices across several GPUs, so every GPU does part of every layer and they combine their results after each one. slices every layer across the GPUs. Pipeline parallelism keeps each layer whole and gives each GPU a run of consecutive layers instead: layers 1 to 20 on the first, 21 to 40 on the second, and so on. The output of one GPU is the input of the next.

The idle problem

Done naively, this wastes most of the hardware. The first GPU works on a batch while the others wait. It passes the batch on, and then it waits while the second GPU works. With 4 GPUs, each is busy only about a quarter of the time. The idle stretches are called the pipeline "bubble".

Micro-batches fill the line

The fix, made popular by Google's GPipe paper in 2018, is to cut each batch into many small micro-batches and feed them in one after another. While GPU 2 works on micro-batch 1, GPU 1 has already started micro-batch 2. After a short start-up, every stage is busy at once, like an assembly line where every station has a car.

The bubble does not vanish. There is still a ramp at the start and a drain at the end. But with many more micro-batches than stages, it becomes a small share of the time. Training adds a second complication, because each micro-batch must also flow backwards through the stages for the gradient step, and later schedules interleave forward and backward passes to keep the bubble small.

Why use it instead of tensor parallelism

The big advantage is how little the GPUs talk. Between stages, only the activations at one boundary move, once per micro-batch. Tensor parallelism exchanges data after every layer. So pipeline parallelism copes with slower links, such as the network between separate servers.

The usual large setup combines all 3: tensor parallelismTensor ParallelismSplitting each layer's weight matrices across several GPUs, so every GPU does part of every layer and they combine their results after each one. across the 8 GPUs inside each server, pipeline parallelism across a few servers, and data parallelismData ParallelismTraining on several GPUs by giving each a full copy of the model and a different slice of each batch, then averaging their gradients so every copy stays the same. over copies of the whole arrangement.

When serving a model

For running a model, pipeline parallelism is mostly a way to make it fit across servers. It does not make a single answer faster, because each token still passes through every stage in turn. It helps total throughput when many requests keep all stages busy. In vLLMvLLMAn open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer. it is --pipeline-parallel-size, often set to the number of servers, with tensor parallelism inside each.