Proximal Policy Optimization

A post-training regime that updates a model with clipped reinforcement-learning steps so RLHF behavior improves without each update moving too far at once.

Proximal Policy Optimization, usually shortened to PPO, is the clipped policy-update method that made early RLHF systems workable. It improves a model against reward signals, but it does so in small guarded steps because aggressive reinforcement-learning updates can damage language behavior quickly.

At a glance

Released

July 2017

Authors

John Schulman, Filip Wolski, Prafulla Dhariwal, et al.

Regime type

Training Alignment

Related modules

No related modules listed yet.

What It Is

Proximal Policy Optimization is a reinforcement-learning training regime. In language-model use, it usually appears after supervised fine-tuning, where the model samples responses, a reward model scores them, and Proximal Policy Optimization nudges the policy toward higher-reward behavior without letting one update swing too far.

Why It Exists

Early RLHF pipelines needed a way to optimize against learned preference signals while keeping the language model stable. Proximal Policy Optimization was attractive because its clipped update rule is meant to limit abrupt policy shifts, which helped teams improve helpfulness or harmlessness without fully trusting raw reward maximization.

How It Works

The current policy generates candidate answers, a reward model or preference-derived score judges them, and Proximal Policy Optimization compares the new policy against the old one before taking the update. The clipped objective keeps the policy ratio inside a bounded range, so the optimizer only accepts gains that do not come from moving the model too far in one step.
PPO keeps sampling, scoring, and clipped policy updates inside one repeated RLHF-style loop.
LmathrmCLIP(theta)=mathbbEtleft[minleft(rt(theta)At,operatornameclip(rt(theta),1−epsilon,1+epsilon)Atright)right]L^{\\mathrm{CLIP}}(\\theta)=\\mathbb{E}_t\\left[\\min\\left(r_t(\\theta)A_t,\\operatorname{clip}(r_t(\\theta),1-\\epsilon,1+\\epsilon)A_t\\right)\\right]
LCLIP(θ)
the clipped PPO objective for the current policy parameters

rt(θ)

the ratio between the new policy probability and the old policy probability for sample t

At

the advantage estimate for sample t
ε
the clip width that limits how far the ratio can move in one update

Et

the average over sampled training steps t

Compared To Nearby Regimes

Proximal Policy Optimization is heavier than direct preference optimization (DPO) because it needs online rollouts and repeated policy updates instead of only fitting preference pairs offline. It is closer to RLHF's classic reinforcement-learning loop than group relative policy optimization (GRPO), which tries to simplify the scoring story by comparing grouped outputs more directly.

Limitations And Failure Modes

Proximal Policy Optimization is operationally heavy because teams must keep a sampling loop, reward-model scoring path, advantage estimation, and repeated policy-training passes all working together. It can also become unstable if rewards are noisy, if the reward model is exploitable, or if the clipping guard is too weak to stop reward hacking and language-quality collapse.

Tags

References

  1. Schulman, John, et al. "Proximal Policy Optimization Algorithms." arXiv, 2017, https://arxiv.org/abs/1707.06347.
  2. Ouyang, Long, et al. "Training language models to follow instructions with human feedback." arXiv, 2022, https://arxiv.org/abs/2203.02155.

Instruction Tuning

Previous Page

Reinforcement Learning from Human Feedback

Next Page

On this page

What It IsWhy It ExistsHow It WorksCompared To Nearby RegimesLimitations And Failure ModesRelated ToTagsReferences