What is RLHF (reinforcement learning from human feedback), and what does it do to a model?

RLHF is how a model learns what people prefer, not just what text looks like. It comes after pre-training and instruction tuningInstruction TuningFine-tuning a pre-trained model on many tasks written as instructions with good answers, so it follows a request instead of just continuing the text., and it is a large part of why an assistant answers politely, declines some requests and writes in the shape people like.

The 3 steps

OpenAI's 2022 InstructGPT paper laid out the recipe most assistants were built on:

1. Demonstrations. People write good answers to prompts, and the model is fine-tuned on them (supervised fine-tuningSupervised Fine‑TuningStandard fine‑tuning on labeled input‑output pairs.).
2. Rankings. The model writes several answers to the same prompt, and people rank them from best to worst. A separate reward model learns to predict those rankings.
3. Reinforcement learning. The language model is tuned to write answers the reward model scores highly, many more than people could rank by hand.

The result surprised people: answers from a 1.3 billion parameter InstructGPT were preferred to those of the 175 billion parameter GPT-3, a model with about 100 times more parameters. Making a model bigger had not made it follow instructions; training on what people preferred did.

What it does not fix

The same paper says InstructGPT still made simple mistakes. A reward model learns what raters like, and raters can like a confident, fluent answer more than a correct one. OpenAI's 2025 study of hallucinationsHallucinationWhen a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist. makes the wider point: training and evaluation that reward a guess over "I don't know" teach a model to guess, the way a multiple-choice exam with no penalty for wrong answers does.

Later methods replace some of the human ranking. Anthropic's Constitutional AI, for one, has a model judge answers against a written list of principles, with no human labels for harmful outputs. The idea is the same: the model is tuned toward what a preference signal rewards. As an engineer you do not run RLHF on an API model. You live with its effects, and you measure them with your own evalsEval SetThe list of test cases an LLM app is graded on after every change. Each case is an input and what a good answer must do, so a score going up or down shows whether the change helped..