Reinforcement Learning from Human Feedback
A post-training alignment workflow that collects human preference signals, fits a reward model, and optimizes the language-model policy toward preferred behavior.
Reinforcement Learning from Human Feedback, usually shortened to RLHF, is a post-training workflow that steers model behavior using human preference signals after a base or instruction-tuned checkpoint already exists.
At a glance
Released
March 2022
Authors
Long Ouyang, Jeff Wu, Xu Jiang, et al.
Regime type
Training Alignment
Related modules
No related modules listed yet.
What It Is
Reinforcement Learning from Human Feedback is a post-training alignment method. Teams start from a pretrained or instruction-tuned language model, collect human rankings or scores over its outputs, train a reward model that predicts those preferences, and then run reinforcement learning so the policy earns higher reward on behavior people prefer.Why It Exists
Instruction-tuned models can follow demonstrations, but product teams often need behavior shaped by comparative judgments: which answer is more helpful, honest, or safe. RLHF exists to turn those preference judgments into a training signal that nudges the model toward aligned behavior for instruction following, preference alignment, and safety-policy shaping.How It Works
The workflow starts from a pretrained or instruction-tuned model that generates candidate answers. Human annotators rank or score those outputs side by side, producing preference data. A reward model learns to predict those human judgments and turns them into a scalar reward or preference signal. A reinforcement-learning optimizer, often Proximal Policy Optimization or PPO, updates the policy so preferred answers earn higher reward while a KL penalty keeps updates close to a reference model. The result is aligned model behavior that reflects the collected preference signal.RLHF feedback-and-optimization loop
Pretrained or
instruction-tuned modelHuman preference
data collectionReward model
or preference signalPolicy optimization
(PPO / RL loop)Aligned model
behaviorPretrained or
instruction-tuned model to Human preference
data collectionHuman preference
data collection to Reward model
or preference signalReward model
or preference signal to Policy optimization
(PPO / RL loop)Policy optimization
(PPO / RL loop) to Aligned model
behavior
Preference and policy flow
\\max_{\\pi_\\theta} \\; \\mathbb{E}_{x,y \\sim \\pi_\\theta}\\big[ r_\\phi(x,y) - \\beta \\, \\mathrm{KL}(\\pi_\\theta \\| \\pi_{\\mathrm{ref}}) \\big]