Dropout

A training-time regularization method that randomly disables a subset of activations during learning and uses the full network, with appropriate scaling, at inference.

Dropout is a training-time regularization method. During learning it randomly turns off a fraction of activations or units so the network cannot rely on fixed co-adapted pathways, then at inference it runs the full network with scaling so expected outputs stay aligned.

At a glance

Released

June 2014

Authors

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, et al.

Regime type

Training

Related modules

No related modules listed yet.

What It Is

Dropout is a training-time technique that randomly zeros out a chosen fraction of activations or units on each forward pass. The mask changes from step to step, so no single neuron or pathway can always be present. At inference the same weights are used, but dropout is turned off and surviving activations are scaled so the model's expected output matches what training prepared it for.

Why It Exists

Large neural networks can memorize training detail by letting small groups of units co-adapt—each unit only useful when its partners are active. Dropout breaks that dependency by forcing many partial networks to share the work. The hope is a more robust ensemble effect inside one model: pathways learn features that still help when neighbors disappear, which can improve generalization on held-out data.

How It Works

Each training step samples a dropout mask, applies it to activations inside selected layers, runs forward and backward through the thinned network, and updates weights. The dropout rate controls how many units are removed. Common implementations scale surviving activations during training—often by dividing by one minus the dropout rate—so inference can use the full network without an extra scaling pass. The flow below contrasts the noisy training path with the deterministic inference path teams serve.
Dropout training flow
Activation and weight flow
Training randomly drops activations and updates weights on partial networks; inference uses the full network with scaling so outputs stay calibrated.
mathbbE[textinferenceoutput]approxtexttrainingoutputwithinverteddropoutscaling\\mathbb{E}[\\text{inference output}] \\approx \\text{training output with inverted dropout scaling}

Compared To Nearby Regimes

Dropout is one regularization lever, not a replacement for the whole training recipe. Weight decay and similar penalties shrink weights in the loss instead of randomly removing activations. Data augmentation changes inputs; dropout changes which internal pathways are active. Early stopping ends training before memorization dominates, while dropout changes every step inside an ongoing run. Teams often combine dropout with other regularizers and tune the dropout rate against validation behavior rather than treating it as always beneficial.

Limitations And Failure Modes

Dropout is not free regularization. Too much dropout can slow convergence or underfit by removing too much signal each step. Very small models, some recurrent setups, and certain low-data regimes may see little benefit or unstable training. It also adds a train-versus-inference mismatch that implementations must handle with correct scaling and mode switches. When validation does not improve, teams reduce the rate, move dropout to different layers, or rely on other regularizers instead of assuming dropout always helps.

Tags

References

  1. Srivastava, Nitish, et al. "Dropout: A Simple Way to Prevent Neural Networks from Overfitting." Journal of Machine Learning Research, vol. 15, no. 56, 2014, pp. 1929-1958, https://jmlr.org/papers/v15/srivastava14a.html.

On-Policy Distillation

Previous Page

FP4 Quantization-Aware Training

Next Page

On this page

What It IsWhy It ExistsHow It WorksCompared To Nearby RegimesLimitations And Failure ModesRelated ToTagsReferences