# What is RLHF (reinforcement learning from human feedback), and what does it do to a model?

A training step where people rank a model's answers, a reward model learns those preferences, and the language model is tuned to score well on it.

- Page: https://tecyfy.com/concepts/reinforcement-learning-from-human-feedback
- Subject: How AI works (https://tecyfy.com/concepts)
- Level: advanced
- Also called: RLHF

RLHF is how a model learns what people prefer, not just what text looks like. It comes after pre-training and [instruction tuning](https://tecyfy.com/concepts/instruction-tuning), and it is a large part of why an assistant answers politely, declines some requests and writes in the shape people like.

## The 3 steps

OpenAI's 2022 InstructGPT paper laid out the recipe most assistants were built on:

1. **Demonstrations.** People write good answers to prompts, and the model is fine-tuned on them ([supervised fine-tuning](https://tecyfy.com/concepts#supervised-fine-tuning)).
2. **Rankings.** The model writes several answers to the same prompt, and people rank them from best to worst. A separate reward model learns to predict those rankings.
3. **Reinforcement learning.** The language model is tuned to write answers the reward model scores highly, many more than people could rank by hand.

The result surprised people: answers from a 1.3 billion parameter InstructGPT were preferred to those of the 175 billion parameter GPT-3, a model with about 100 times more parameters. Making a model bigger had not made it follow instructions; training on what people preferred did.

## What it does not fix

The same paper says InstructGPT still made simple mistakes. A reward model learns what raters like, and raters can like a confident, fluent answer more than a correct one. OpenAI's 2025 study of [hallucinations](https://tecyfy.com/concepts/hallucination) makes the wider point: training and evaluation that reward a guess over "I don't know" teach a model to guess, the way a multiple-choice exam with no penalty for wrong answers does.

Later methods replace some of the human ranking. Anthropic's Constitutional AI, for one, has a model judge answers against a written list of principles, with no human labels for harmful outputs. The idea is the same: the model is tuned toward what a preference signal rewards. As an engineer you do not run RLHF on an API model. You live with its effects, and you measure them with your own [evals](https://tecyfy.com/concepts/eval-set).

## Related

- [Instruction Tuning](https://tecyfy.com/concepts/instruction-tuning): Fine-tuning a pre-trained model on many tasks written as instructions with good answers, so it follows a request instead of just continuing the text.
- [Supervised Fine‑Tuning](https://tecyfy.com/concepts#supervised-fine-tuning): Standard fine‑tuning on labeled input‑output pairs.
- [Alignment](https://tecyfy.com/concepts#alignment): Ensuring model objectives match human values and intent
- [Large Language Model](https://tecyfy.com/concepts/large-language-model): A model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions.
- [Hallucination](https://tecyfy.com/concepts/hallucination): When a model states something false as if it were true, such as a wrong date, a made-up quote or a source that does not exist.
- [Red Teaming](https://tecyfy.com/concepts/red-teaming): Attacking your own AI system on purpose, the way a real attacker or a careless user would, to find where it fails before they do.

## Sources

- [Training language models to follow instructions with human feedback](https://arxiv.org/abs/2203.02155)
- [Constitutional AI: Harmlessness from AI Feedback](https://arxiv.org/abs/2212.08073)
- [Why Language Models Hallucinate](https://arxiv.org/abs/2509.04664)