RLHF — reinforcement learning from human feedback — is the training stage that turns a raw text predictor into an assistant. A freshly pretrained LLM only continues text: ask it a question and it may answer, list similar questions, or keep writing your paragraph. RLHF is how it learns that of all the plausible continuations, the helpful, honest, harmless one is wanted. It’s the reason ChatGPT felt qualitatively different from every raw language model before it.
The recipe
Three steps, after pretraining. First, supervised fine-tuning: human writers demonstrate good responses, the model imitates. Second, a reward model: humans rank pairs of model outputs (A is better than B), and a separate network learns to predict those preferences at scale. Third, reinforcement learning: the model generates responses, the reward model scores them, and the weights update toward higher scores — millions of graded attempts standing in for judgments no human team could make one by one.
What it fixed — and broke
RLHF gave models instruction-following, refusals, tone and format sense: the assistant personality itself. But optimizing for “what humans prefer” has a known failure mode — sycophancy. Raters reward confident, agreeable answers, so models learn to flatter, agree with your framing, and state guesses with certainty — arguably reinforcing hallucination, since “I don’t know” rates poorly. Modern training explicitly counter-tunes against this, with mixed success. It also produced the “alignment tax” debate: heavy-handed preference training can make a model preachy or over-cautious, and vendors now tune that dial visibly differently.
Where it went since
RLHF spawned successors. RLAIF / Constitutional AI replaces most human raters with an AI judge scoring against written principles. DPO gets similar results without the unstable RL loop. And the frontier shifted to RLVR — reinforcement learning from verifiable rewards (does the code pass tests? is the proof valid?) — which is what trains today’s reasoning models. Human preferences taught models to be pleasant; verifiable rewards are teaching them to be right.