Distance-Based Diversity Regularization

• Distance-based diversity regularization is a set of techniques that add geometry-sensitive terms to model objectives, ensuring outputs or components remain distinct.
• It leverages various distance measures—including Euclidean, cosine, and determinant metrics—to balance repulsion and representativeness across applications like deep ensembles, clustering, and dataset condensation.
• Empirical findings demonstrate that tuning the diversity strength enhances calibration and robustness while mitigating the risk of over-dispersion.

Searching arXiv for recent and foundational papers on distance-based diversity regularization. Distance-based diversity regularization denotes a family of methods that encourage outputs, components, samples, or trajectories to be sufficiently different from the rest under an explicit distance or dissimilarity, rather than optimizing task fit alone. In the literature, this idea appears in diverse K-nearest-neighbor retrieval, deep ensembles, graph clustering, weakly supervised segmentation, dataset condensation, generative adversarial learning, latent variable modeling, recommendation, video attention, kernel methods, and reinforcement learning, with diversity expressed through Euclidean distances, cosine dissimilarities, mutual angles, determinant-based volume terms, Hellinger distance, Jeffrey’s divergence, and related surrogates 0310028.

1. Canonical objective forms

A recurring pattern is additive regularization. In deep ensembles with negative correlation, the per-network objective is

$$E_i(\mathbf{x}, y; h_i) = L(y, h_i(\mathbf{x})) + \lambda \,\mathrm{div}\big(h_i(\mathbf{x}); h_{1:M}\big),$$

with ensemble prediction

$$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$

and a diversity term defined on output deviations from the ensemble mean (Shui et al., 2018). In graph clustering, DMoN–DPR augments the DMoN loss with a distance-based centroid-separation term,

$$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$

where

$$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$

so pairs of centroids contribute only when their squared Euclidean distance falls below a margin $\epsilon$ (Salehi et al., 23 Jan 2025).

A second pattern is repulsion combined with representativeness. In dataset condensation, DiRe defines a synthetic regularization term

$$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$

where cosine diversity penalizes synthetic–synthetic cosine similarity, while cosine and Euclidean distribution matching pull synthetic embeddings toward real ones (Mohanty et al., 15 Dec 2025). This pairing of repulsive and attractive terms is one of the clearest formulations of diversity as a regularizer rather than a standalone objective.

A third pattern replaces entropy or likelihood-based regularization with sample-based distance terms. For intractable multimodal policies, DrAC uses

$$J(\pi)=\mathbb{E}_\pi\Big[\sum_{t=0}^{\infty}\gamma^t\big(R_t+\alpha D^\pi(S_t)\big)\Big],$$

with

$$D^\pi(s)=\mathbb{E}_{x\sim \pi(\cdot|s),\,y\sim \pi(\cdot|s)}[\log \delta(x,y)],$$

and $\delta(x,y)=\|x-y\|_2$ in action space, so diversity is optimized directly from sampled decisions without requiring $\log \pi_\theta(a|s)$ (Wang et al., 3 Nov 2025).

These formulations suggest a common interpretation: distance-based diversity regularization typically adds a geometry-sensitive term that discourages collapse while preserving the primary task objective.

2. Distance notions and geometric surrogates

The family is heterogeneous chiefly because “distance” is instantiated in different spaces.

Setting	Object diversified	Distance or surrogate
Diverse KNN retrieval	query results	sufficiently different answers under a user-tunable diversity notion [0310028]
Deep ensembles	member outputs or logits	negative correlation, log-determinant, pairwise $$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$0 distance (Shui et al., 2018, Mehrtens et al., 2022)
Graph clustering	cluster centroids	squared Euclidean distance with margin $$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$1 (Salehi et al., 23 Jan 2025)
Histopathology prototypes	intra-class activation distributions	Jeffrey’s divergence over spatial distributions (Le et al., 5 Dec 2025)
Dataset condensation	synthetic embeddings	cosine similarity and Euclidean distance (Mohanty et al., 15 Dec 2025)
GANs and latent variable models	filters or latent components	squared cosine similarity, mutual angular separation (Ayinde et al., 2019, Xie et al., 2015)
Video attention	spatial attention distributions	Hellinger distance via $$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$2 (Li et al., 2018)
RL reasoning trajectories	correct reasoning trajectories	cosine dissimilarity of trajectory embeddings plus formula uniqueness (Wan et al., 23 Feb 2026)
Kernel Nyström sampling	landmark subsets	determinant of kernel submatrices, equivalent to spread in RKHS (Fanuel et al., 2020)

Function-space distances are especially common. In video-based person re-identification, each attention map is a probability mass function over spatial cells, and diversity is enforced through

$$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$3

where $$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$4 elementwise; the off-diagonal terms equal overlaps of square-rooted attention distributions and correspond to squared Hellinger distance (Li et al., 2018). In weakly supervised histopathology segmentation, each prototype induces a spatial probability distribution

$$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$5

and intra-class diversity is regularized through

$$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$6

with $$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$7 (Le et al., 5 Dec 2025).

Parameter-space distances remain important in settings where components themselves define the model’s representational basis. DiReAL penalizes positively and negatively correlated filters through thresholded squared cosine similarity, while mutual angular regularization defines

$$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$8

and rewards larger mean pairwise angles with lower variance of angles across components (Ayinde et al., 2019, Xie et al., 2015).

3. Representative mechanisms across application areas

In retrieval and recommender systems, diversity regularization corrects the tendency of nearest-neighbor or metric models to collapse onto dominant modes. The KNN diversity problem is posed as returning “the closest result set such that each answer is sufficiently different from the rest,” and MOTLEY is presented as providing a “seamless interface between diversity and distance” [0310028]. In collaborative metric learning, DPCML assigns multiple embeddings to each user and scores an item by the minimum user–item distance,

$$\bar{h}(\mathbf{x}) = \frac{1}{M} \sum_{i=1}^M h_i(\mathbf{x}),$$9

while controlling intra-user diversity through

$$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$0

and the banded penalty

$$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$1

so embeddings are neither collapsed nor excessively dispersed (Bao et al., 2022).

In ensemble prediction and uncertainty estimation, the central target is disagreement among functions rather than among parameters. Negative correlation learning regularizes output deviations around the ensemble mean (Shui et al., 2018). A later ensemble study defines “Sample Diversity” on OOD inputs by normalizing logits, stacking them into $$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$2, and maximizing

$$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$3

which encourages orthogonality among member logits specifically on unfamiliar data (Mehrtens et al., 2022). This is a distinct design choice: diversity is not enforced on labeled in-distribution samples, but on auxiliary inputs used to shape uncertainty.

In clustering, segmentation, and prototype methods, the diversified objects are centroids, prototypes, or assignment distributions. DMoN–DPR explicitly separates cluster centroids in feature space through squared Euclidean margins (Salehi et al., 23 Jan 2025). LPD promotes complementary intra-class prototypes by penalizing similarity between their induced spatial distributions via Jeffrey’s divergence (Le et al., 5 Dec 2025). In video re-identification, multiple spatial attention heads are encouraged to discover different body regions through Hellinger-distance-based repulsion among attention maps (Li et al., 2018).

In optimization, sampling, and search, diversity is often encoded through determinants or genealogical distances. DPP-based Nyström sampling makes subset probability proportional to $$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$4, so landmark sets with larger volume in RKHS are favored; the paper shows that diversity sampling acts as implicit regularization for kernel methods (Fanuel et al., 2020). In evolutionary algorithms, genealogical diversity is approximated with trash genes and normalized Hamming distance,

$$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$5

and then added directly to the modified fitness

$$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$6

making genealogical spread an explicit regularizer in the search objective (Gabor et al., 2017).

4. Optimization and computational properties

Distance-based diversity regularizers differ sharply in what they require from the optimizer. Some are fully differentiable and add only modest overhead. DMoN–DPR computes soft centroids, pairwise centroid distances, and a ReLU margin term; the authors state that the overall overhead is negligible relative to adjacency-based GNN operations (Salehi et al., 23 Jan 2025). Negative-correlation ensembles backpropagate through per-network losses while treating the ensemble mean as constant for the member being updated (Shui et al., 2018). LPD computes softmax-normalized activation distributions and Jeffrey’s divergence over prototype pairs after a short classification warm-up for stability (Le et al., 5 Dec 2025).

Other methods rely on matrix objectives whose geometry is global over a set of components. Log-determinant regularizers in OOD ensemble diversification and determinant-based DPP sampling both reward volume, but their computational roles differ: in ensembles the log-determinant is optimized directly as a training loss on predictions, while in kernel methods determinants define the sampling distribution over landmark subsets (Mehrtens et al., 2022, Fanuel et al., 2020). Mutual angular regularization addresses the non-convexity and non-smoothness of direct angle optimization by replacing it with a smooth lower bound based on $$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$7 (Xie et al., 2015).

A notable recent development is sample-based diversity without tractable densities. DrAC shows that stochastic-mapping actors

$$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$8

admit policy gradients through reparameterization,

$$L_{\text{DMoN--DPR}} = L_{\text{DMoN}} + W_{\text{dist}} \, L_{\text{DPR}}^{\text{Distance}} + W_{\text{var}} \, L_{\text{DPR}}^{\text{variance}} + W_{\text{entropy}} \, L_{\text{DPR}}^{\text{entropy}},$$9

and that the diversity estimator

$$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$0

is optimized by ordinary automatic differentiation (Wang et al., 3 Nov 2025). This removes the dependence on $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$1, KL terms, or closed-form entropy.

Sampling-based variants regularize optimization indirectly by changing the minibatch distribution. Diversity-based sampling for domain alignment replaces random minibatches with k-DPP or k-means++ minibatches in feature space, leaving the training loss unchanged but improving the representativeness of each stochastic estimate (Napoli et al., 2024). This suggests that distance-based diversity regularization need not appear as an explicit penalty; it can also enter through the sampling mechanism that determines which distances the optimizer sees.

5. Empirical effects and characteristic trade-offs

The empirical record is consistently favorable, but not monotone in regularization strength. In deep ensembles, negative correlation improves calibration while largely preserving accuracy. On CIFAR-100 with $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$2, the reported accuracy/ECE pairs are $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$3 for the pure ensemble and $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$4 for the NC-regularized ensemble, with the effect becoming more pronounced as ensemble size grows (Shui et al., 2018). In partially shared ensembles, OOD “Sample Diversity” improves corrupted-data accuracy and calibration, and in some settings a 2-member ensemble with diversity regularization matches or exceeds the robustness of a 5-member unregularized ensemble (Mehrtens et al., 2022).

In graph clustering, the gains are strongest when feature richness makes pure structural separation insufficient. On Coauthor Physics, DMoN has F1 $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$5, DMoN–DPR(D) reaches $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$6, and DMoN–DPR(DVE) reaches $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$7, while Coauthor CS improves from $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$8 to $$L_{\text{DPR}}^{\text{Distance}} = \frac{1}{k(k-1)} \sum_{i=1}^{k} \sum_{j\neq i} \mathrm{ReLU}\!\left(\epsilon - \|\mu_i-\mu_j\|_2^2\right),$$9 under DVE (Salehi et al., 23 Jan 2025). In histopathology weak supervision, adding the prototype diversity regularizer improves over PBIP by $\epsilon$0 mIoU and $\epsilon$1 mDice on BCSS-WSSS, while overly strong diversity ($\epsilon$2) slightly degrades performance relative to moderate values (Le et al., 5 Dec 2025).

In dataset condensation, DiRe simultaneously improves test accuracy and diversity metrics. On ImageNet-1K with SRe$\epsilon$3L at IPC $\epsilon$4, the reported accuracy rises from $\epsilon$5 to $\epsilon$6, coverage from $\epsilon$7 to $\epsilon$8, intra-class cosine similarity drops from $\epsilon$9 to $$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$0, and the Vendi score rises from $$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$1 to $$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$2 (Mohanty et al., 15 Dec 2025). In distribution alignment, diverse minibatches reduce the mean absolute percentage error of MMD estimates from $$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$3 under random sampling to $$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$4 under k-DPP, and the same samplers improve out-of-distribution accuracy for CORAL, DANN, and ERM (Napoli et al., 2024).

These results point to a pervasive trade-off. Stronger repulsion can improve coverage, uncertainty, and minority-mode retention, but it can also conflict with task fit. DPCML addresses this explicitly by penalizing both insufficient and excessive pairwise separation among user embeddings (Bao et al., 2022). MDR in metric learning adopts a different strategy: it “explicitly disturbs a learning procedure” by forcing pairwise distances into multiple levels, so the regularizer and the base loss “interfere with the objective of each other,” which the authors attribute to better generalization (Kim et al., 2021).

6. Misconceptions, relations, and open directions

A common misconception is that diversity regularization is equivalent to entropy regularization. Recent RLVR work rejects this directly: token-level entropy “induces only local stochasticity and fails to induce meaningful path-level diversity,” whereas DSDR adds explicit global diversity among correct reasoning trajectories through cosine dissimilarity of trajectory embeddings and formula-level uniqueness, coupled to a correct-only local entropy term (Wan et al., 23 Feb 2026). A related point appears in few-shot image generation: mixup-based distance learning enforces consistency between latent mixing weights and feature-space similarities, thereby smoothing the latent space and resisting memorization, which is not the same as merely raising entropy (Kong et al., 2021).

A second misconception is that any parameter-space repulsion suffices. The literature is divided. DiReAL and mutual angular regularization show that weight- or component-space geometry can stabilize learning, reduce redundancy, and improve interpretability (Ayinde et al., 2019, Xie et al., 2015). By contrast, the ensemble study on robustness reports that functional diversity matters much more than parameter distance in partially shared architectures, and that weight-space regularization brings little improvement compared with prediction-space regularization on OOD samples (Mehrtens et al., 2022). This suggests that the appropriate distance space is task dependent.

A third misconception is that more separation is always better. Several papers reject that conclusion explicitly. DPCML uses a banded distance penalty rather than pure maximization (Bao et al., 2022). LPD reports that $$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$5–$$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$6 works best and that $$L_{\text{syn}}= \sum_{c=1}^C \Big[r_c\big(l_{\text{cos\_div}}^c+l_{\text{cos\_dm}}^c\big)+r_e\,l_{\text{euc\_dm}}^c\Big],$$7 slightly degrades performance (Le et al., 5 Dec 2025). DMoN–DPR observes that small margins are preferred on Cora and CiteSeer, whereas richer feature spaces such as Coauthor Physics benefit from larger margins (Salehi et al., 23 Jan 2025). The empirical pattern is therefore not “maximize distance,” but “shape geometry to match the task.”

Open directions are consistent across domains. DivHF points to active selection of informative triples and other query-efficient ways to learn human-aligned behavior spaces (Wang et al., 2023). DSDR explicitly notes the possibility of richer trajectory metrics beyond cosine distance and formula uniqueness (Wan et al., 23 Feb 2026). Prototype-based segmentation identifies alternative divergences, including Jensen–Shannon or optimal-transport distances, as plausible replacements for Jeffrey’s divergence (Le et al., 5 Dec 2025). This suggests that the next stage of the field is less about introducing diversity per se than about learning or adapting the distance geometry in which diversity is defined.