# What is Proximal Policy Optimization (PPO)?

A reinforcement learning method that improves a model in small, capped steps. It was the algorithm behind the RLHF stage of the first ChatGPT-style assistants.

- Page: https://tecyfy.com/concepts/proximal-policy-optimization
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: PPO

Supervised training shows a model the right answer and has it copy. That does not work when there is no single right answer, only better and worse ones: a more helpful reply, a safer refusal, a working proof. Reinforcement learning handles that case. The model tries, gets a score, and is nudged towards whatever scored well. PPO is the method OpenAI published in 2017 for doing that nudging without wrecking the model.

## The problem it solves

In reinforcement learning the model is called the policy: the thing that picks actions, or for a language model, the next [Token](https://tecyfy.com/concepts/token). Each update changes the policy based on answers it just produced and how they scored. If one update goes too far, the policy starts producing very different answers, the scores on them are unreliable, and training can collapse.

PPO's answer is a cap. For each update it compares how likely the new policy is to give an answer with how likely the old one was. If that ratio moves outside a small band (the paper uses 20% either side), the extra change earns no further reward. The model is allowed to improve, but only "proximally", close to where it was. That simple clip made PPO stable, easy to implement, and the default choice for years.

## PPO in RLHF

For the first instruction-following assistants, PPO was the last stage of [reinforcement learning from human feedback](https://tecyfy.com/concepts/reinforcement-learning-from-human-feedback):

1. A [reward model](https://tecyfy.com/concepts#rlhf-reward-model) is trained on human rankings of answers.
2. The language model writes answers to prompts.
3. The reward model scores them, and PPO updates the language model towards higher scores, with a penalty for drifting too far from where it started.

OpenAI's InstructGPT paper in 2022 used this recipe. People preferred the answers of its 1.3B model to those of the 175B GPT-3, a model more than 100 times larger.

## What came after

PPO for language models is heavy. It runs the model being trained, a frozen copy of it, the reward model and a value model that estimates expected scores, all at once. Two simpler routes became popular:

- **DPO** (Direct Preference Optimization, 2023) skips the reward model and the sampling and learns directly from pairs of preferred and rejected answers.
- **GRPO**, from DeepSeek in 2024, keeps PPO's clipped update but drops the value model, scoring each answer against a group of others for the same prompt. It is widely used for training reasoning models on maths and code, where an answer can be checked automatically.

The core idea of PPO, improve in small capped steps, survives in most of them.

## Related

- [Reinforcement Learning from Human Feedback](https://tecyfy.com/concepts/reinforcement-learning-from-human-feedback): A training step where people rank a model's answers, a reward model learns those preferences, and the language model is tuned to score well on it.
- [RLHF Reward Model](https://tecyfy.com/concepts#rlhf-reward-model): A model trained to score outputs according to human preferences.
- [Alignment](https://tecyfy.com/concepts#alignment): Ensuring model objectives match human values and intent
- [Supervised Fine‑Tuning](https://tecyfy.com/concepts#supervised-fine-tuning): Standard fine‑tuning on labeled input‑output pairs.
- [Gradient Descent](https://tecyfy.com/concepts/gradient-descent): The way a neural network learns. Measure how wrong it is, work out which way each weight should move to be less wrong, take a small step that way, repeat.

## Sources

- [Proximal policy optimization algorithms](https://arxiv.org/abs/1707.06347)
- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)
- [Direct preference optimization, your language model is secretly a reward model](https://arxiv.org/abs/2305.18290)
- [DeepSeekMath, pushing the limits of mathematical reasoning in open language models](https://arxiv.org/abs/2402.03300)