Reinforcement Learning with Verifiable Rewards

A post-training reinforcement learning regime where rewards come from externally checkable task outcomes rather than human preference labels alone.

Reinforcement Learning with Verifiable Rewards, usually shortened to RLVR, is a post-training method that uses reinforcement learning when a task outcome can be checked by an external verifier instead of relying only on human preference labels.

At a glance

Released

January 2025

Authors

DeepSeek-AI

Regime type

Training

Related modules

No related modules listed yet.

What It Is

Reinforcement Learning with Verifiable Rewards is a post-training regime for language models. The model acts as a policy, samples candidate answers or reasoning traces, and receives reward from deterministic checks on whether those outputs satisfy task rules. RLVR is most natural when the answer or behavior can be checked by an external verifier, such as exact-answer math grading, code unit tests, formal proof checkers, or other rule-based validators.

Why It Exists

Many reasoning-heavy tasks have objectively checkable success criteria. A math problem may have one exact final answer, a programming task may pass or fail a test suite, and a logic puzzle may satisfy a rule-based grader. Those checks provide reward signals without asking humans to rank every model output. RLVR exists so teams can optimize models on those tasks with reinforcement learning while keeping the reward grounded in verifiable outcomes rather than subjective preference alone. Public work such as DeepSeek-R1 shows how verifier-based reinforcement learning can strengthen math and code reasoning when checkable rewards are available.

How It Works

Training usually starts from a capable checkpoint, often after supervised fine-tuning. Each training step follows the same observable loop. First, a task prompt is drawn from a verifiable task distribution. The model samples a candidate response for that prompt. An external verifier checks whether the response passes the task rules, such as exact-answer grading, unit tests, or formal checks. The verifier outcome is turned into a reward signal, often a pass-or-fail score or a shaped numeric reward. A reinforcement learning optimizer then applies a policy update so higher-reward responses become more likely. The loop repeats across many prompts until the model improves on the verifiable task distribution.
RLVR training flow
Request and reward flow
RLVR samples model outputs, checks them with an external verifier, assigns reward from the check outcome, and updates the policy from that reward.
R=mathbb1[textverifier(y,x)=textpass]R = \\mathbb{1}[\\text{verifier}(y, x) = \\text{pass}]

Compared To Nearby Regimes

Supervised fine-tuning, often shortened to SFT, teaches the model to imitate labeled demonstrations directly. RLVR instead samples multiple candidate outputs and keeps updates that score well under external checks rather than copying one fixed demonstration per prompt. Reinforcement learning from human feedback, often shortened to RLHF, relies on human preference signals or learned reward models that approximate what people like. RLVR relies on externally checkable outcomes where a verifier can score success without ranking subjective quality. Group Relative Policy Optimization, or GRPO, is a nearby reinforcement-learning optimizer that compares rewards across a group of sampled candidates. GRPO can be used with human-preference rewards or with verifier-based rewards, but RLVR names the reward source—verifiable task outcomes—not the optimizer family.

Limitations And Failure Modes

RLVR only applies where verifier coverage is strong enough to trust the reward signal. Tasks without checkable outcomes still need human feedback or other reward sources. Narrow verifiers can encourage reward hacking, where the model satisfies the checked rule without learning the broader behavior teams want. Task narrowness is another limit: strong scores on one verifiable benchmark do not guarantee transfer to open-ended reasoning, safety, or style. Verifiable success can also miss qualities humans care about, such as helpful explanations, honest uncertainty, or refusal of unsafe requests, because those traits may not appear in the verifier at all.

Tags

References

  1. DeepSeek-AI. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv, 2025, https://arxiv.org/abs/2501.12948.

Post-Training

Previous Page

Memory

Next Page

On this page

What It IsWhy It ExistsHow It WorksCompared To Nearby RegimesLimitations And Failure ModesRelated ToTagsReferences