# What is pipeline parallelism?

Putting different layers of a model on different GPUs and passing work from one to the next like an assembly line, with small batches keeping every stage busy.

- Page: https://tecyfy.com/concepts/pipeline-parallelism
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: PP, Inter-Layer Model Parallelism

A model too large for one GPU can be cut in 2 ways. [Tensor parallelism](https://tecyfy.com/concepts/tensor-parallelism) slices every layer across the GPUs. Pipeline parallelism keeps each layer whole and gives each GPU a run of consecutive layers instead: layers 1 to 20 on the first, 21 to 40 on the second, and so on. The output of one GPU is the input of the next.

## The idle problem

Done naively, this wastes most of the hardware. The first GPU works on a batch while the others wait. It passes the batch on, and then it waits while the second GPU works. With 4 GPUs, each is busy only about a quarter of the time. The idle stretches are called the pipeline "bubble".

## Micro-batches fill the line

The fix, made popular by Google's GPipe paper in 2018, is to cut each batch into many small micro-batches and feed them in one after another. While GPU 2 works on micro-batch 1, GPU 1 has already started micro-batch 2. After a short start-up, every stage is busy at once, like an assembly line where every station has a car.

The bubble does not vanish. There is still a ramp at the start and a drain at the end. But with many more micro-batches than stages, it becomes a small share of the time. Training adds a second complication, because each micro-batch must also flow backwards through the stages for the gradient step, and later schedules interleave forward and backward passes to keep the bubble small.

## Why use it instead of tensor parallelism

The big advantage is how little the GPUs talk. Between stages, only the activations at one boundary move, once per micro-batch. Tensor parallelism exchanges data after every layer. So pipeline parallelism copes with slower links, such as the network between separate servers.

The usual large setup combines all 3: [tensor parallelism](https://tecyfy.com/concepts/tensor-parallelism) across the 8 GPUs inside each server, pipeline parallelism across a few servers, and [data parallelism](https://tecyfy.com/concepts/data-parallelism) over copies of the whole arrangement.

## When serving a model

For running a model, pipeline parallelism is mostly a way to make it fit across servers. It does not make a single answer faster, because each token still passes through every stage in turn. It helps total throughput when many requests keep all stages busy. In [vLLM](https://tecyfy.com/concepts/vllm) it is `--pipeline-parallel-size`, often set to the number of servers, with tensor parallelism inside each.

## Related

- [Tensor Parallelism](https://tecyfy.com/concepts/tensor-parallelism): Splitting each layer's weight matrices across several GPUs, so every GPU does part of every layer and they combine their results after each one.
- [Data Parallelism](https://tecyfy.com/concepts/data-parallelism): Training on several GPUs by giving each a full copy of the model and a different slice of each batch, then averaging their gradients so every copy stays the same.
- [Sharding](https://tecyfy.com/concepts#sharding): Partitioning large model parameters or optimizer states across devices to fit in memory.
- [vLLM](https://tecyfy.com/concepts/vllm): An open-source engine for serving language models fast, built on PagedAttention, that answers requests through the same kinds of API the big model makers offer.

## Sources

- [GPipe, efficient training of giant neural networks using pipeline parallelism](https://arxiv.org/abs/1811.06965)
- [Parallelism and scaling](https://docs.vllm.ai/en/latest/serving/parallelism_scaling/)