---
title: Retrieved Dropout Models in Neural Networks
url: https://www.emergentmind.com/topics/retrieved-dropout-models
type: topic
---

# Retrieved Dropout Models in Neural Networks

Searching arXiv for papers on retrieved dropout models and closely related dropout interpretations.
arxiv_search(query="1805.09208 dropout retrieved models power mean deterministic dropout", max_results=5)
arxiv_search(query="dropout family power mean deterministic dropout Monte Carlo dropout arXiv", max_results=10)
Retrieved dropout models are a family of predictive models that can be selected *after* a network has been trained once with ordinary stochastic dropout. In the formulation introduced in "Pushing the bounds of dropout" [1805.09208], standard dropout training is reinterpreted as simultaneously performing MAP estimation for multiple conditional models that share the same trained parameters $\Theta$ but differ in how they aggregate predictions over dropout masks and how much stochasticity they retain at evaluation. Under this view, arithmetic Monte Carlo averaging, geometric averaging, power-mean aggregation, reduced-dropout subvariants, and the deterministic limit are not separate training procedures; they are post-training retrieval choices from a common dropout family [1805.09208].

## 1. Origins and conceptual setting

The broader dropout literature treats dropout methods as stochastic techniques used in neural network training or inference for regularization, uncertainty estimation, and related purposes [1904.13310]. Standard activation dropout multiplies hidden activations by Bernoulli masks during training, and its conventional interpretation combines two ideas: discouraging co-adaptation and approximating an ensemble of subnetworks. Retrieved dropout models preserve that stochastic-training background, but alter the inference-time interpretation.

In the retrieved-model view, the stochastic training objective is not tied uniquely to one predictive rule such as arithmetic Monte Carlo averaging. Instead, the same dropout-trained parameters may support a family of conditional models whose MAP objectives are all lower bounded by the original stochastic dropout objective [1805.09208]. This shifts the status of evaluation-time choice: selecting a predictive rule after training is not treated as an approximation heuristic alone, but as choosing one member of a broader family already implicated by training.

A useful boundary condition is that retrieved dropout models concern **post-training selection among predictive rules derived from standard dropout training**. They are therefore distinct from later dropout variants that change the training-time stochastic process itself, such as time-scheduled retain probabilities in Curriculum Dropout [1703.06229], sample-dependent mask distributions in Contextual Dropout [2103.04181], or learned structured masks in AutoDropout [2101.01761].

## 2. Conditional MAP reformulation of dropout

The central formal move in [1805.09208] is to cast dropout as a conditional MAP problem. Let \(X=\{x_i\}_{i=1}^N\), \(Y=\{y_i\}_{i=1}^N\), \(\Theta\) be the trainable parameters, and \(w\) the dropout-masked weights. The model uses a prior
\[
p(\Theta)=N(\Theta\mid 0,\sigma_p^2 I)
\]
and a dropout-induced conditional distribution over masked weights. For a row \(w_r\) of a weight matrix, the paper writes
\[
p(w_r\mid \Theta)=p\,N(w_r\mid 0,\sigma^2 I)+(1-p)\,N(w_r\mid \Theta_r,\sigma^2 I).
\]

With
\[
p(y,w\mid x,\Theta)=p(y\mid x,w)\,p(w\mid \Theta),
\]
the posterior admits the lower bound
\[
\ln p(\Theta\mid X,Y)\geq \sum_{i=1}^N \int p(w\mid \Theta)\ln p(y_i\mid x_i,w)\,dw+\ln p(\Theta)-C_{MAP}.
\]
Approximating the expectation with one sampled mask \(\hat w\sim p(w\mid \Theta)\) yields
\[
L_{MAP}=-\sum_{i=1}^N \ln p(y_i\mid x_i,\hat w_i)-\ln p(\Theta),
\]
which the paper treats as effectively the standard dropout objective with weight decay [1805.09208].

This reformulation matters because it supplies a common lower bound shared by several different conditional models. The training procedure fixes \(\Theta\), but it does not force a single evaluation-time aggregation rule. A plausible implication is that dropout training can be viewed as preparing parameters for a structured model-selection problem at inference rather than only for one canonical test-time approximation.

## 3. The retrievable family of dropout models

The retrieved family is organized by two axes: how predictive probabilities are aggregated over dropout masks, and how much dropout stochasticity is retained at evaluation [1805.09208]. The basic members are the arithmetic model, the geometric model, and the power-mean continuum between them.

The **arithmetic model** is the direct latent-variable conditional model,
\[
p(y\mid x,\Theta)=\mathbb{E}_{\hat w\sim p(w\mid \Theta)}[p(y\mid x,\hat w)].
\]
In practice this is arithmetic Monte Carlo dropout, denoted AMC.

The **geometric model** uses the renormalized geometric mean over masks,
\[
p(y\mid x,\Theta)=\frac{\exp\!\big(\mathbb{E}_{\hat w\sim p(w\mid \Theta)} \ln p(y\mid x,\hat w)\big)}{Z(x,\Theta)},
\qquad
Z(x,\Theta)=\sum_{c=1}^C \exp\!\big(\mathbb{E}_{\hat w\sim p(w\mid \Theta)} \ln p(c\mid x,\hat w)\big).
\]
This is geometric Monte Carlo dropout, or GMC.

The **power-mean family** interpolates between these two. Using
\[
M_\alpha(x_1,\dots,x_n)=\left(\frac{1}{n}\sum_{i=1}^n x_i^\alpha\right)^{1/\alpha},
\]
the corresponding conditional model is
\[
p(y\mid x,\Theta)=
\frac{\left(\mathbb{E}_{\hat w\sim p(w\mid \Theta)} p(y\mid x,\hat w)^\alpha\right)^{1/\alpha}}{Z(x,\Theta)},
\qquad \alpha\in[0,1].
\]
Here \(\alpha=1\) gives the arithmetic mean and \(\alpha\to 0\) gives the geometric mean.

The paper then introduces less stochastic **subvariants** by scaling the evaluation-time dropout rate with a multiplier \(\lambda\in[0,1]\). This produces the **extended dropout family**, parameterized by \((\alpha,\lambda)\): \(\lambda=1\) recovers the original stochastic model, \(0<\lambda<1\) gives reduced-variance versions, and \(\lambda=0\) gives the deterministic limit [1805.09208]. In this precise sense, a retrieved dropout model is the member of this family selected after training.

## 4. Lower bounds, objective ordering, and the deterministic limit

A defining claim of [1805.09208] is that the standard stochastic dropout objective is a common lower bound for the MAP objectives of the retrieved family. For the power-mean family,
\[
\ln p(\Theta\mid X,Y)
=
\sum_{i=1}^N
\left[
\ln \left(\mathbb{E}_{\hat w} p(y_i\mid x_i,\hat w)^\alpha\right)^{1/\alpha}
-\ln Z(x_i,\Theta)
\right]
+\ln p(\Theta)-C_{MAP},
\]
and for \(\alpha\in[0,1]\), the monotonicity of the power mean gives \(Z(x,\Theta)\leq 1\). Jensen’s inequality then yields the common lower bound based on
\[
\sum_{i=1}^N \mathbb{E}_{\hat w\sim p(w\mid \Theta)} \ln p(y_i\mid x_i,\hat w)+\ln p(\Theta)-C_{MAP}.
\]

This objective ordering supports the paper’s most distinctive interpretive claim: reducing stochasticity at evaluation tends to tighten the lower bound. The mechanism is the **Jensen gap**, which shrinks as predictive variance decreases. Using the result cited from Maddison et al.,
\[
\ln(\mathbb{E}[L])-\mathbb{E}[\ln L]
=
\frac{\operatorname{var}(L)}{2(\mathbb{E}[L])^2}
+\mathcal{O}\!\left(\sqrt{\mathbb{E}\!\left[(L-\mathbb{E}[L])^6\right]}\right),
\]
the paper argues that less variance implies a tighter approximation to the true objective [1805.09208].

The endpoint of this logic is **deterministic dropout**. In the deterministic limit, the lower-bound gap disappears: the bound equals the objective, and the paper describes this as the highest and tightest objective in the family. A common misconception is therefore addressed directly. Deterministic dropout is not treated merely as a cheap approximation to Monte Carlo averaging; it is presented as the best available approximation to the true objective induced by dropout training [1805.09208].

This does not mean deterministic retrieval is always best for test-set generalization. It means the deterministic member has a privileged status with respect to objective fit. The paper explicitly separates objective tightness from downstream predictive behavior.

## 5. Empirical behavior and task dependence

The empirical study in [1805.09208] evaluates retrieved models on MNIST classification and on language modeling benchmarks including Enwik8, PTB, and Wikitext-2. The principal experimental finding is that reducing evaluation-time dropout improves training fit monotonically. On PTB, changing only the evaluation-time dropout multiplier from \(\lambda=1.0\) to \(\lambda=0.0\) reduced cross-entropy from \(2.878\) to \(2.731\) [1805.09208]. This is the direct empirical counterpart of the bound-tightening argument.

Validation and test behavior were more task dependent. On **MNIST**, deterministic dropout was best, matching the theoretical expectation that the tightest objective approximation could also yield the best generalization [1805.09208]. In **word-level language modeling**, however, the best results often came from **arithmetic averaging**, not from deterministic retrieval. On PTB test, the best reported direct retrieval choice was arithmetic averaging with reduced dropout at
\[
4.012 \text{ nats} \ (\approx 55.2 \text{ ppl}),
\]
improving on the baseline deterministic result
\[
4.071.
\]
On Wikitext-2, the best test result was about
\[
4.154
\]
versus the baseline deterministic
\[
4.188
\]
[1805.09208].

The explanation given is **smoothing**, especially for rare words in highly imbalanced output spaces. Deterministic retrieval dominates on the training set, but arithmetic MC retrieval can generalize better because it smooths predictive probabilities. This interpretation is consistent with the broader recurrent-language-model literature, where dropout choice, schedule, and placement interact strongly with rare-word behavior and sequence robustness [1811.00998].

The paper also reports that tuning a **softmax temperature** improved deterministic and geometric retrieval considerably, often bringing them close to AMC, while AMC itself benefited little. This supports the claim that arithmetic retrieval already contributes substantial smoothing [1805.09208]. A plausible implication is that deterministic retrieval plus temperature tuning functions as a computationally cheap proxy when full Monte Carlo evaluation is undesirable.

## 6. Conceptual boundaries, misconceptions, and relation to later dropout work

Retrieved dropout models occupy a specific position in the dropout design space. They do **not** change the training-time corruption law; they change the interpretation and selection of the predictive model after ordinary dropout training. In that sense they are orthogonal to later methods that adapt dropout *during* training.

This distinction is sharp in relation to several later developments. Curriculum Dropout changes the retain probability over optimization time through
\[
\theta_{\rm curriculum}(t)=(1-\overline{\theta})e^{-\gamma t}+\overline{\theta},
\]
so its contribution is adaptive regularization during training rather than post-training retrieval [1703.06229]. Contextual Dropout replaces fixed dropout rates with sample-dependent distributions \(q_\phi(z\mid x)\), making masks local random variables conditioned on the input [2103.04181]. AutoDropout treats dropout structure itself as a search problem over learned mask patterns [2101.01761]. The stochastic delta rule redefines each weight as a random variable with learned mean and standard deviation, making standard dropout a coarse, fixed, non-adaptive instance in a broader stochastic family [1808.03578]. None of these are retrieved dropout models in the strict sense of [1805.09208].

The same caution applies to newer uses of the word *dropout* that are not regularization in the original sense. In PEFT for LLMs, "LoRA Meets Dropout under a Unified Framework" organizes transformer-specific dropout by dropping position, structural pattern, and compensation measure, and introduces HiddenKey as a preferred LoRA regularizer [2403.00812]. In diffusion language models, DPad’s "suffix dropout" removes future suffix tokens at inference time before attention computation, and the paper explicitly classifies this as structured token pruning rather than canonical dropout regularization [2508.14148]. Such methods share the vocabulary of dropout, but not the retrieval-based conditional-model interpretation.

Two misconceptions are therefore particularly important. First, retrieved dropout models should not be collapsed into ordinary MC-dropout uncertainty estimation: their central claim concerns a family of MAP objectives sharing a common lower bound, not merely stochastic test-time sampling [1805.09208]. Second, deterministic dropout should not be described only as an approximation to arithmetic MC averaging. In the retrieved-model framework, deterministic dropout is the zero-variance endpoint of the extended family and the member with bound equal to objective [1805.09208].

Taken together, these points define retrieved dropout models as a post-training model-selection theory for ordinary dropout. The framework recasts a single dropout training run as preparation for choosing among arithmetic, geometric, power-mean, reduced-stochasticity, and deterministic predictive rules. Its enduring significance lies less in proposing another masking scheme than in reassigning where model choice occurs: not exclusively in training, but also in the retrieval of the predictive model after training has finished.

Source: https://www.emergentmind.com/topics/retrieved-dropout-models