High-Rate Mixout Fine-Tuning

• High-rate Mixout is a fine-tuning method that stochastically reverts 70%–90% of parameters to pretrained anchors to stabilize learning on small datasets.
• It acts as an adaptive quadratic regularizer that pulls weights toward the pretrained model, reducing catastrophic forgetting and mimicking ensemble benefits.
• Empirical evidence shows that high-rate Mixout improves out-of-domain performance while reducing computational cost, particularly in domain generalization and small-data scenarios.

Searching arXiv for the cited Mixout and High-Rate Mixout papers to ground the article in the primary literature. High-rate Mixout is the regime of Mixout in which a large fraction of parameters are stochastically reverted to a reference set of pretrained weights during training, so that fine-tuning remains strongly anchored to the initialization rather than drifting freely toward a task-specific solution. The original Mixout formulation was introduced for stabilizing fine-tuning of large pretrained LLMs, with practically useful rates in the interval $p\in[0.7,0.9]$ on small-data GLUE tasks (Lee et al., 2019). Subsequent work transplanted the same high-rate regime to vision and domain generalization, using swap or masking rates of $0.8$ for ResNet50 and $0.9$ for ViT-S/16, and later extended the method through adaptive anchors and episodic mask resampling in GMixout (Aminbeidokhti et al., 8 Oct 2025, Aminbeidokhti et al., 8 Oct 2025).

1. Core construction

In its standard parameter-space form, Mixout is defined by a current parameter vector $\theta\in\mathbb{R}^d$, a fixed target parameter $\theta_0\in\mathbb{R}^d$, and a Bernoulli mask $m\in\{0,1\}^d$ with independent entries

$m_i\sim \mathrm{Bernoulli}(1-p),$

where $p$ is the mixout probability. The mixed parameter is

$\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$

Each coordinate therefore either stays at $\theta_i$ with probability $0.8$0 or is reset to $0.8$1 with probability $0.8$2. Training uses $0.8$3 in the forward pass and evaluates $0.8$4; in the original formulation, test-time inference uses $0.8$5 without mixing, analogously to inverted dropout (Lee et al., 2019).

A later parameterization rewrites the fine-tuned model as a pretrained anchor plus a learnable residual,

$0.8$6

and samples an element-wise mask $0.8$7 from $0.8$8. To preserve the expectation of each neuron’s output, the residual is rescaled by $0.8$9, giving the stochastic objective

$0.9$0

Under this view, $0.9$1 is the fraction of coordinates reverted to $0.9$2 at each step, and the sparsity $0.9$3 is the fraction of updated coordinates (Aminbeidokhti et al., 8 Oct 2025).

The term “high-rate” refers specifically to operating this mechanism with large swap or masking probabilities. In the NLP setting this denotes $0.9$4; in the domain-generalization setting it denotes very aggressive anchoring, empirically $0.9$5 for ResNet50 and $0.9$6 for ViT-S/16 (Lee et al., 2019, Aminbeidokhti et al., 8 Oct 2025).

2. Regularization interpretation

The theoretical appeal of Mixout is that it can be interpreted as an adaptive quadratic pull toward a nonzero reference model rather than toward the origin. In the original analysis, classical inverted DropConnect or Dropout is described as adding, in expectation, an adaptive $0.9$7-penalty that pulls parameters toward the zero vector. Mixout replaces that origin-centered bias with a pull toward $0.9$8, the pretrained parameter vector (Lee et al., 2019).

More precisely, let $0.9$9 be strongly convex with modulus $\theta\in\mathbb{R}^d$0, and define

$\theta\in\mathbb{R}^d$1

When $\theta\in\mathbb{R}^d$2, the paper states

$\theta\in\mathbb{R}^d$3

Accordingly, minimizing the mixout-augmented objective is, up to constants, equivalent to minimizing

$\theta\in\mathbb{R}^d$4

Because $\theta\in\mathbb{R}^d$5 depends on the local curvature of $\theta\in\mathbb{R}^d$6, the resulting pull toward $\theta\in\mathbb{R}^d$7 is adaptive along the optimization trajectory (Lee et al., 2019).

The residual-based analysis reaches the same qualitative conclusion. A first-order expansion yields an implicit weight-decay term

$\theta\in\mathbb{R}^d$8

so increasing $\theta\in\mathbb{R}^d$9 monotonically increases the effective regularization strength $\theta_0\in\mathbb{R}^d$0. This gives a precise sense in which high-rate Mixout biases the solution toward the pretrained anchor (Aminbeidokhti et al., 8 Oct 2025).

3. Behavior in the high-rate regime

The high-rate regime is characterized by a rapid increase in regularization strength as $\theta_0\in\mathbb{R}^d$1 approaches $\theta_0\in\mathbb{R}^d$2. In the original formulation, the effective coefficient scales roughly like $\theta_0\in\mathbb{R}^d$3: at $\theta_0\in\mathbb{R}^d$4, $\theta_0\in\mathbb{R}^d$5; at $\theta_0\in\mathbb{R}^d$6, $\theta_0\in\mathbb{R}^d$7; at $\theta_0\in\mathbb{R}^d$8, $\theta_0\in\mathbb{R}^d$9. As $m\in\{0,1\}^d$0, $m\in\{0,1\}^d$1, so $m\in\{0,1\}^d$2 is forced to remain extremely close to $m\in\{0,1\}^d$3; as $m\in\{0,1\}^d$4, $m\in\{0,1\}^d$5, recovering unregularized fine-tuning (Lee et al., 2019).

Optimization effects follow directly from this scaling. Higher $m\in\{0,1\}^d$6 slows the drift away from pretrained initialization, which reduces catastrophic forgetting and improves stability when the downstream dataset is small. The same mechanism can also slow convergence to the downstream minimum and produce under-fitting if the anchoring becomes too strong. The interval $m\in\{0,1\}^d$7 is therefore presented as a stability–adaptation compromise: strong enough to avoid degenerate runs, but not so strong that adaptation is entirely suppressed (Lee et al., 2019).

The vision literature extends this interpretation with an implicit-ensemble perspective. If $m\in\{0,1\}^d$8 denotes subnetworks obtained from random masks, their average satisfies

$m\in\{0,1\}^d$9

A first-order Taylor expansion around $m_i\sim \mathrm{Bernoulli}(1-p),$0 shows that the prediction of the mean network approximates the average over subnetworks up to second-order terms in $m_i\sim \mathrm{Bernoulli}(1-p),$1. This suggests that high-rate Mixout can be read not only as strong anchor regularization but also as a single-run surrogate for an ensemble centered near the pretrained model (Aminbeidokhti et al., 8 Oct 2025).

A common misunderstanding is that higher masking is uniformly beneficial. The reported results do not support that interpretation. In the domain-generalization study, out-of-domain accuracy improves with increasing $m_i\sim \mathrm{Bernoulli}(1-p),$2 up to an architecture-specific optimum, but for ResNets performance drops sharply beyond $m_i\sim \mathrm{Bernoulli}(1-p),$3 because too many kernels are frozen each step; in the GMixout study, higher $m_i\sim \mathrm{Bernoulli}(1-p),$4 improves out-of-domain robustness while sacrificing some in-domain accuracy, yielding an explicit ID–OOD trade-off (Aminbeidokhti et al., 8 Oct 2025, Aminbeidokhti et al., 8 Oct 2025).

4. Evidence from small-data language-model fine-tuning

The original empirical case for high-rate Mixout comes from fine-tuning BERT-large, a $m_i\sim \mathrm{Bernoulli}(1-p),$5-parameter model, on GLUE tasks known to be unstable under vanilla fine-tuning. The evaluation considered RTE with $m_i\sim \mathrm{Bernoulli}(1-p),$6 examples, MRPC with $m_i\sim \mathrm{Bernoulli}(1-p),$7, CoLA with $m_i\sim \mathrm{Bernoulli}(1-p),$8, and STS-B with $m_i\sim \mathrm{Bernoulli}(1-p),$9. Baselines were Dropout$p$0+weight-decay$p$1, corresponding to the Devlin et al. default, and weight-decay-only with $p$2. Mixout variants used $p$3, with or without weight-decay$p$4, and each setting was run with $p$5 random restarts; the study reported mean $p$6 standard deviation, best dev scores, and failed runs at chance level (Lee et al., 2019).

On RTE, the baseline mean dev score of $p$7 improved to $p$8 with Mixout$p$9, $\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$0 with Mixout$\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$1, and $\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$2 with Mixout$\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$3. On MRPC, the baseline $\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$4 improved to $\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$5, $\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$6, and $\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$7 respectively. On CoLA, the baseline $\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$8 increased to $\tilde\theta = m\odot\theta + (1-m)\odot\theta_0.$9, $\theta_i$0, and $\theta_i$1. On STS-B, the baseline $\theta_i$2 increased to $\theta_i$3, $\theta_i$4, and $\theta_i$5. Best dev scores also improved in most cases; for example, MRPC improved from $\theta_i$6 under the baseline to $\theta_i$7 for weight-decay-only and above $\theta_i$8 across the Mixout variants, while CoLA best dev reached $\theta_i$9 under Mixout$0.8$00 (Lee et al., 2019).

The stability effect was as important as the mean-score effect. High-rate Mixout, especially around $0.8$01, raised the mean dev score by large margins and reduced failed-run counts from approximately $0.8$02–$0.8$03 to $0.8$04–$0.8$05. By contrast, increasing $0.8$06 in dropout did not improve stability: larger dropout rates hurt average dev performance and increased failure rates. This contrast underlies one of the central distinctions between dropout and Mixout in transfer learning: dropout removes activations or weights toward zero, whereas Mixout reverts parameters toward a pretrained reference (Lee et al., 2019).

The same study also reported a large-data counterexample. On SST-2, with $0.8$07 examples, the mean and max dev scores were $0.8$08 for Dropout$0.8$09+W$0.8$10 and $0.8$11 for Mixout$0.8$12+W$0.8$13. The result was summarized as essentially no harm when data are plentiful, which argues against the view that strong anchoring is only useful at the cost of large-data degradation (Lee et al., 2019).

5. Domain generalization and computational properties

The 2025 domain-generalization formulation reframed high-rate Mixout as a way to approximate ensemble robustness without training multiple models. The setting trains on multiple source domains $0.8$14 and generalizes to unseen target domains $0.8$15. The objective is written as

$0.8$16

with

$0.8$17

For ResNet50, the implementation swaps entire convolutional kernels rather than individual weights; for ViT-S/16, it applies unstructured Mixout in all MLP and attention-projection layers. Validation-domain grid search selected $0.8$18 for ResNet50 and $0.8$19 for ViT-S/16 (Aminbeidokhti et al., 8 Oct 2025).

The experimental program used leave-one-out cross-validation on PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet. Training employed Adam, a batch size of $0.8$20 examples per domain per mini-batch, learning-rate grid $0.8$21, weight decay $0.8$22, swap-rate grid $0.8$23, and $0.8$24 steps for small datasets or $0.8$25 for DomainNet. Average out-of-domain accuracy for ResNet50 was $0.8$26 for ERM, $0.8$27 for Large Dropout, $0.8$28 for High-Rate Mixout, $0.8$29 for Deep Ensemble $0.8$30, and $0.8$31 for Model Soup $0.8$32. For ViT-S/16, ERM reached $0.8$33, High-Rate Mixout $0.8$34, Deep Ensemble $0.8$35, and Model Soup $0.8$36 (Aminbeidokhti et al., 8 Oct 2025).

A second contribution of this study was a computational argument. If the forward pass costs $0.8$37, a standard training iteration uses approximately $0.8$38 for forward plus backward weight gradients plus backward input gradients. With Mixout rate $0.8$39, gradients are neither computed nor stored for the frozen parameters, so weight-gradient FLOPs become $0.8$40, input-gradient FLOPs remain $0.8$41, and the total is approximately

$0.8$42

The reported consequence is up to $0.8$43 reduction in gradient-computation FLOPs and up to $0.8$44 reduction in gradient memory when $0.8$45. In the headline summary, ResNet50 training cost was $0.8$46 ERM and ViT-S/16 training cost was $0.8$47 ERM, while inference cost remained $0.8$48 (Aminbeidokhti et al., 8 Oct 2025).

6. Extensions, implementation practice, and limitations

Practical guidance in the original NLP study is explicit. High-rate Mixout is recommended for downstream tasks with fewer than $0.8$49 training examples; $0.8$50 is the starting point, with movement toward $0.8$51 if degenerate runs persist and movement toward $0.8$52 or lower if under-fitting appears. The method is applied to all pretrained layers. For a newly initialized classification head, one may either skip Mixout or apply Mixout toward the head’s random initialization, denoted mixout$0.8$53, with $0.8$54; the summary states that mixout$0.8$55 often outperforms simple dropout on that layer. A small weight decay, approximately $0.8$56, may be retained, but Mixout already accounts for most of the gain, and no extra hyperparameters beyond replacing dropout are required; the same learning-rate schedule, batch size, and optimizer can be kept, with Adam and learning rate approximately $0.8$57 for BERT (Lee et al., 2019).

GMixout generalizes high-rate Mixout by replacing the fixed anchor with an exponential moving-average anchor and by introducing a resampling-frequency hyperparameter $0.8$58. At the start of episode $0.8$59, the anchor is updated as

$0.8$60

or, in the practical update rule, $0.8$61 followed by $0.8$62. The mask is held fixed for $0.8$63 optimizer steps, so if $0.8$64 is the total number of steps, the number of episodes is $0.8$65. A sparse-kernel implementation stores only the unmasked indices $0.8$66 and the corresponding residual values, materializing $0.8$67 on the fly; the summary reports zero inference-time overhead after merging $0.8$68 into $0.8$69, and peak GPU memory reduced by approximately $0.8$70–$0.8$71 compared to dense Mixout (Aminbeidokhti et al., 8 Oct 2025).

The GMixout results sharpen the limitations of fixed-anchor high-rate Mixout rather than negating them. On DomainNet, Mixout with $0.8$72 achieved average OOD $0.8$73 and ID $0.8$74, whereas GMixout with $0.8$75, $0.8$76, and $0.8$77 achieved average OOD $0.8$78 and ID $0.8$79. Figure-level ablations summarized in the paper indicate that larger $0.8$80 improves OOD while costing ID, and that increasing $0.8$81 up to about $0.8$82 steps per mask improves both ID and OOD before diversity begins to fall. The same study recommends $0.8$83, $0.8$84, and $0.8$85–$0.8$86 episodes as a robust default across covariate shift, corruption, and class-imbalance benchmarks (Aminbeidokhti et al., 8 Oct 2025).

Taken together, these results delimit the present understanding of high-rate Mixout. It is not merely “dropout with a larger probability,” since larger dropout probabilities harmed stability in the original BERT study. It is also not a synonym for parameter freezing, because the method preserves adaptation through stochastic residual updates and, in the domain-generalization setting, approaches ensemble-level robustness at a fraction of ensemble cost. The main technical tension is consistent across the literature: increasing $0.8$87 strengthens anchoring and robustness, but beyond an architecture- and task-dependent threshold it can under-train the downstream model (Lee et al., 2019, Aminbeidokhti et al., 8 Oct 2025, Aminbeidokhti et al., 8 Oct 2025).