Diversity Regularizer (DiRe) Methods

• Diversity Regularizer (DiRe) is a family of regularization strategies that penalize redundant learned components to encourage complementary behavior in ensembles, representations, and synthetic datasets.
• Negative-correlation DiRe in deep ensembles aligns individual model confidence with ensemble accuracy by minimizing cross-entropy with a diversity penalty based on deviations from the ensemble mean.
• DiRe methods extend across multiple domains—including regression, adversarial robustness, and dataset condensation—balancing estimation and approximation errors to optimize performance.

Diversity Regularizer (DiRe) denotes a family of regularization strategies that explicitly penalize redundancy among learned components so that an ensemble, representation space, policy population, cluster assignment, or synthetic dataset does not collapse onto near-duplicate solutions. In the 2018 deep-ensemble formulation that is most directly associated with calibration, the method appears as Negative-Correlation (NC) regularization: each ensemble member minimizes cross-entropy plus a diversity penalty defined on its deviation from the ensemble mean probability vector, with the stated effect of improving calibration while maintaining similar prediction accuracy (Shui et al., 2018). In later literature, the label “DiRe” and closely related names were reused for information-theoretic hidden-unit regularization, out-of-distribution ensemble diversification, multimodal geometric regularization, and diversity-promoting dataset condensation, so the term is best understood as a recurring regularization principle rather than a single universally fixed algorithm (Zhang et al., 2020, Mehrtens et al., 2022, Xia et al., 29 Jan 2026, Mohanty et al., 15 Dec 2025).

1. Terminological scope and recurring design pattern

Across the literature summarized here, “diversity regularization” refers to an auxiliary loss that encourages complementary behavior while preserving task performance. What changes from paper to paper is the object being diversified: softmax outputs, regression functions, hidden-unit weight vectors, hidden representations, convolutional filters, inverse-dynamics distributions, cluster centroids, or synthetic-sample embeddings.

Setting	What is diversified	Representative paper
Deep ensembles for calibration	Member probability outputs	(Shui et al., 2018)
Regression ensembles	Member predictions and ensemble capacity	(Reeve et al., 2018)
Hidden-unit generalization	Weight directions or label-based redundancy	(Xie et al., 2015, Zhang et al., 2020)
Ensemble robustness under shift	OOD logits or feature redundancy	(Mehrtens et al., 2022, Rame et al., 2021)
Multimodal learning	Intra-modal spread and inter-modal drift	(Xia et al., 29 Jan 2026)
Dataset condensation	Synthetic embeddings within and across classes	(Mohanty et al., 15 Dec 2025)

A common misconception is to treat all DiRe methods as decorrelation penalties of the same kind. The papers do not support that simplification. Some methods act directly on prediction vectors, some on feature geometry, some on class-conditional mutual information, and some on synthetic data embeddings. A plausible unifying description is that DiRe methods convert diversity from an emergent by-product of optimization into an explicit inductive bias.

2. Negative-correlation DiRe in deep ensembles

In the ensemble-calibration setting, the training set is $\mathcal{D}=\{(x_n,y_n)\}_{n=1}^N$, the ensemble size is $M$, member $m$ produces logits $f_m(x;\theta_m)$ and probabilities $$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$, and the ensemble average is

$\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$

With cross-entropy

$$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$

each member minimizes

$$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$

where $\lambda\ge 0$ weights diversity against accuracy. The NC diversity term is

$$Div_m(x)=\bigl(p_m(x)-\bar p(x)\bigr)^T\sum_{j\ne m}\bigl(p_j(x)-\bar p(x)\bigr).$$

An equivalent minibatch form is

$M$0

Its stated intuition is that, if two members’ output deviations from the ensemble mean are positively correlated, they make similar mistakes; penalizing that correlation pushes members toward complementary predictions (Shui et al., 2018).

The training procedure is correspondingly direct. For each minibatch, all members perform a forward pass, the ensemble mean $M$1 is computed, and each member’s total loss is formed as its average classification loss plus $M$2 times its diversity loss. Parameters are then updated by standard gradient descent on that member-specific total loss. Standard random minibatches suffice, and no special data partitioning is required. The reported tuning practice uses a grid such as $M$3 for $M$4, with $M$5 identified as a good default on CIFAR-100; ensemble sizes $M$6 are reported to show clearer calibration gains than very small ensembles such as $M$7 (Shui et al., 2018).

This formulation is notable because it regularizes the relationship among members rather than only the loss of the averaged ensemble. The regularizer is applied on softmax outputs, not on weights, filters, or intermediate features. That distinction separates NC-regularization from later DiRe variants that target representation geometry or mutual information.

3. Calibration metrics and empirical behavior

The calibration analysis associated with NC-regularization uses reliability diagrams and Expected Calibration Error (ECE). Test-time predictions $M$8 are partitioned into $M$9 equally spaced confidence bins $m$0, and each bin is summarized by

$m$1

ECE is then

$m$2

The reported empirical pattern is that a standard deep ensemble (“pure”) tends to be under-confident, with $m$3 across bins, and that increasing $m$4 can worsen ECE. Adding NC-regularization drives confidence and accuracy closer together, reducing ECE with little or no loss in accuracy (Shui et al., 2018).

Setting	Baseline(s)	NC-regularized outcome
CIFAR-100, VGG-11+BN, $m$5, $m$6	Pure ensemble: accuracy 70.88%, ECE 4.3%	accuracy 70.96%, ECE 2.5%
CIFAR-100, people superclass of 5 classes	Single net: $m$7; Pure ensemble: $m$8	NC-ensemble: $m$9
CIFAR-10, modified AlexNet, $f_m(x;\theta_m)$0	Single net: acc 76.07%, ECE 3.9%; Pure ensemble: acc 77.51%, ECE 5.9%	NC-ensemble: acc 78.65%, ECE 1.4%

The accompanying reliability diagrams and prediction-confidence histograms are described as showing two simultaneous effects: the confidence distribution shifts toward the diagonal associated with ideal calibration, and the prediction histograms become sharper, corresponding to improved accuracy. Within the ensemble-calibration literature, this is an important point because many post hoc calibration procedures improve confidence alignment without altering the decision function, whereas NC-regularization modifies the training dynamics so that accuracy and calibration move together (Shui et al., 2018).

4. Theoretical interpretations and bias–variance trade-offs

One theoretical strand analyzes diversity at the level of hidden-unit weight directions through the Mutual Angular Regularizer (MAR). For a layer with incoming-weight vectors $f_m(x;\theta_m)$1, the non-obtuse pairwise angle is

$f_m(x;\theta_m)$2

and the regularizer is

$f_m(x;\theta_m)$3

The network objective becomes

$f_m(x;\theta_m)$4

The analysis shows that increasing diversity of hidden units reduces estimation error and increases approximation error, yielding an intermediate optimum rather than a monotone preference for maximal spreading. Empirically, on TIMIT phoneme classification with one hidden layer and $f_m(x;\theta_m)$5 sigmoid units, the best test accuracy occurs at a nonzero, moderate $f_m(x;\theta_m)$6; the reported curve rises from about $f_m(x;\theta_m)$7 at $f_m(x;\theta_m)$8 to about $f_m(x;\theta_m)$9 at $$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$0 (Xie et al., 2015).

A second strand studies Negative Correlation Learning (NCL) in regression ensembles and connects diversity to effective degrees of freedom. With $$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$1 regressors and squared error, the diversity penalty can be written as

$$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$2

For fixed basis functions, the effective degrees of freedom admit the closed form

$$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$3

and the paper establishes a connection to Tikhonov regularisation. It further states that, with an appropriately chosen diversity parameter, an NCL ensemble can always outperform the unregularised ensemble in the presence of noise, and it derives SURE-based tuning together with a Monte-Carlo estimator extending the connection to deep neural-network ensembles (Reeve et al., 2018).

A third strand frames diversity through information theory. In the label-based hidden-unit regularizer, the key quantity is

$$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$4

which is added to the main objective as

$$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$5

The stated interpretation is that lowering this redundancy tightens an information-theoretic generalization bound, since the decrease of redundancy generally improves generalization capacity. On CIFAR-10 and CIFAR-100, the reported effect is a markedly smaller train–test gap than NONE, Dropout, Decov, or the unsupervised MI regularizer UDM (Zhang et al., 2020).

Taken together, these analyses support a consistent caution: diversity is not a free good. Theoretical and empirical results repeatedly indicate that moderate, task-preserving diversity is beneficial, whereas excessive diversification can degrade approximation quality, hurt individual accuracy, or interfere with the primary structural objective.

5. Domain-specific extensions

In ensemble robustness and uncertainty estimation, later methods diversified outputs in more specialized ways than NC-regularization. One line uses out-of-distribution samples: an ensemble of $$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$6 classifiers is trained with the standard in-distribution cross-entropy on the averaged prediction plus a diversity term on uniformly sampled OOD noise,

$$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$7

where $$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$8 stacks $$p_m(x)=\mathrm{softmax}(f_m(x;\theta_m))\in \Delta^K$$9-normalized logit vectors for the $\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$0-th OOD sample. Because the determinant of the Gram matrix is maximized when the columns are orthogonal, the method pushes ensemble members apart on inputs outside the training distribution while leaving in-distribution predictions unchanged. On CIFAR-10 corruptions, the paper reports that even $\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$1 with Sample Diversity in TreeNet can outperform $\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$2 independently trained TreeNet; it also reports strong gains in ECE, NLL, and OOD AUC-ROC, especially in partially shared architectures (Mehrtens et al., 2022). A related adversarial-robustness method, ADP, defines diversity among non-maximal predictions through a determinant-based squared volume,

$\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$3

and adds

$\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$4

to the training objective; on CIFAR-10, the reported robust accuracy under attacks such as FGSM, BIM, PGD, MIM, C–W, and EAD improves substantially while maintaining strong clean accuracy (Pang et al., 2019). Another ensemble method, DICE, moves the regularizer into feature space and adversarially minimizes conditional redundancy $\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$5 while preserving class information via VCEB; the paper states that an ensemble of 5 networks trained with DICE matches an ensemble of 7 networks trained independently on CIFAR-10/100 (Rame et al., 2021).

In generative and representation-learning settings, diversity regularization is applied directly to learned filters or intermediate features. Diversity Regularized Adversarial Learning for GANs defines, for each convolutional layer, a Gram matrix $\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$6 over $\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$7-normalized filters and penalizes masked squared correlations above a threshold $\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$8 through

$\bar p(x)=\frac{1}{M}\sum_{j=1}^M p_j(x).$9

With $$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$0, $$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$1, and $$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$2, the paper reports improved stability and, for CIFAR-10 with spectral normalization, an Inception Score increase from $$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$3 to $$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$4 (Ayinde et al., 2019). Guided layer-wise learning introduces diversifying regularization for generative models via Hellinger divergence between variational posteriors of differently labeled examples and for discriminative models via hidden-feature distances across such pairs; the reported effect is faster convergence, smaller generalization errors, and mitigation of vanishing gradients (Sulimov et al., 2019). In multimodal learning, the Dispersive-and-Anchoring Geometric Regularizer defines an intra-modal dispersive loss

$$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$5

with $$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$6, and an inter-modal anchoring loss

$$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$7

The combined objective is

$$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$8

and the reported overhead is about $$\mathcal{L}_{CE}(y,p)=-\sum_{k=1}^K 1\{y=k\}\log p_k,$$9–$$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$0 extra per epoch with zero cost at inference (Xia et al., 29 Jan 2026).

In structured-data, control, and synthetic-data settings, diversity regularization is again reinterpreted rather than merely transplanted. In reinforcement learning, Diversity in Regulation trains one policy at a time and augments reward with inverse-dynamics disagreement under a filtered state representation,

$$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$1

with PPO updates and frozen inverse models for previously learned policies; the paper reports improvements in adaptation to damage, dynamics shifts, and sensor failures (Xu et al., 2022). In graph clustering, DMoN-DPR adds distance-based, variance-based, and entropy-based regularizers to the DMoN objective to encourage feature-space diversity among clusters; on Coauthor Physics, the DVE configuration is reported to improve NMI from $$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$2 to $$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$3 and F1 from $$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$4 to $$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$5 (Salehi et al., 23 Jan 2025). In dataset condensation, DiRe defines cosine diversity, cosine distribution matching, and Euclidean distribution matching over synthetic and real embeddings,

$$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$6

and adds this term to the base condensation loss. For ImageNet-1K with SRe$$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$7L and IPC $$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$8, the reported top-1 accuracy rises from $$E_m(\theta_m)=\mathbb{E}_{(x,y)\sim\mathcal{D}}\bigl[\mathcal{L}_{CE}(y,p_m(x))+\lambda\cdot Div_m(x)\bigr],$$9 to $\lambda\ge 0$0, while Coverage increases from $\lambda\ge 0$1 to $\lambda\ge 0$2 (Mohanty et al., 15 Dec 2025).

6. Practical interpretation, common misunderstandings, and limitations

A central misunderstanding is to equate diversity with arbitrary disagreement. The ensemble-calibration formulation does not reward disagreement for its own sake; it penalizes positively correlated deviations from the ensemble mean while still optimizing each member’s cross-entropy (Shui et al., 2018). DICE makes this even more explicit by attempting to remove only conditionally redundant information, not task-relevant information shared through $\lambda\ge 0$3 (Rame et al., 2021). The multimodal dispersive-and-anchoring formulation likewise avoids total cross-modal collapse by imposing a tolerance $\lambda\ge 0$4 and ceasing to pull paired embeddings together once they are within radius $\lambda\ge 0$5 (Xia et al., 29 Jan 2026).

A second misunderstanding is that more diversity is always better. The MAR analysis states that increasing diversity reduces estimation error and increases approximation error, and the reported TIMIT curves peak at a nonzero, moderate regularization strength rather than at the largest possible value (Xie et al., 2015). The original NC-regularization summary recommends starting at $\lambda\ge 0$6 and reducing $\lambda\ge 0$7 if under-fitting emerges; it also notes that smaller ensembles still benefit but to a lesser extent (Shui et al., 2018). DICE reports that large $\lambda\ge 0$8 can over-diversify and hurt accuracy, and DMoN-DPR notes that overly large weights on distance, variance, or entropy terms can compromise pure modularity (Rame et al., 2021, Salehi et al., 23 Jan 2025).

Operationally, DiRe methods differ significantly in cost and implementation burden. NC-regularized deep ensembles require essentially $\lambda\ge 0$9 the cost of a single model plus negligible extra cost for the diversity term (Shui et al., 2018). OOD log-determinant regularization requires a Gram determinant per OOD sample and, in high-class-count regimes, the condition $$Div_m(x)=\bigl(p_m(x)-\bar p(x)\bigr)^T\sum_{j\ne m}\bigl(p_j(x)-\bar p(x)\bigr).$$0 or an alternative such as the $$Div_m(x)=\bigl(p_m(x)-\bar p(x)\bigr)^T\sum_{j\ne m}\bigl(p_j(x)-\bar p(x)\bigr).$$1 variant to avoid singularity (Mehrtens et al., 2022). Information-theoretic methods such as the label-based redundancy estimator and DICE add auxiliary networks and adversarial optimization steps (Zhang et al., 2020, Rame et al., 2021). Synthetic-data and multimodal variants rely on dense pairwise similarity or distance computations, but DAGR reports zero inference-time cost, and the dataset-condensation variant can exploit cached real embeddings and vectorized pairwise operations (Xia et al., 29 Jan 2026, Mohanty et al., 15 Dec 2025).

This literature suggests a stable high-level conclusion. Diversity regularization is most effective when the regularized signal is tightly matched to the failure mode of interest: output covariance for calibration, OOD logit geometry for robustness under shift, conditional redundancy for simultaneous ensemble errors, angular separation for hidden-unit specialization, dispersive geometry for multimodal collapse, or embedding diversity for synthetic-data redundancy. Under that reading, “DiRe” is less a single named method than a broad research program that treats complementarity as a trainable property of modern machine-learning systems.