Sample-Agnostic Diversity Enhancement (SADE)

• SADE is a set of techniques that enforces diversity among model experts by regularizing inter-expert similarity at the parameter level.
• It leverages sample-agnostic regularization to promote specialized subspace learning, improving accuracy and parameter efficiency in tasks like low-rank adaptation.
• Applied in collaborative low-rank adaptation and long-tailed recognition, SADE achieves notable accuracy gains and significant reductions in computational overhead.

Sample-Agnostic Diversity Enhancement (SADE) refers to a family of techniques in modern deep learning that enforce or leverage diversity among model components—specifically, among parameterized experts or low-rank modules—in a manner independent of individual data samples. The primary objective is to expand the effective expressiveness of adapted models without introducing excessive parameter overhead or sample-dependent regularization terms. This principle underlies several recent innovations: SADE serves as a critical regularization mechanism within collaborative low-rank adaptation frameworks for pre-trained vision transformers and as a test-time adaptation mechanism via self-supervised aggregation of diverse experts in long-tailed visual recognition settings (Liu et al., 31 Dec 2025, Zhang et al., 2021).

1. Motivations for Sample-Agnostic Diversity Enhancement

In many settings, modular adaptation or transfer learning involves injecting multiple experts (parameter subsets or low-rank matrices) into a larger architecture. As these experts co-adapt, they may become redundant, extracting similar directions or representations, thereby wasting model capacity available for adaptation. Sample-agnostic diversity enhancement is designed to encourage experts to specialize on distinct subspaces, maximizing the cumulative benefit of their joint contribution to model updates. This is achieved by regularizing inter-expert similarity at the parameter level, irrespective of any individual input or batch statistics (Liu et al., 31 Dec 2025).

In test-agnostic long-tailed recognition, diversity enhancement via multiple skill-diverse experts enables robust performance even when the test class distribution is unknown and may differ arbitrarily from the training distribution. Expert diversification improves the likelihood that at least one expert is well-aligned with the true, latent test distribution (Zhang et al., 2021).

2. Formulations and Algorithmic Foundations

2.1 Parameter-Space Regularization in CLoRA

Within collaborative low-rank adaptation (CLoRA), each low-rank module (LRM) generates its weight update as a sum of $p$ experts:

$$\Delta W_j = \sum_{h=1}^{p} M_h^j, \quad \text{where} \quad M_h^j = D_h Q_h^j U_h$$

with $D_h, U_h$ as shared down/up-projection matrices and $Q_h^j$ as per-LRM mixing matrices. Without enforced diversity, multiple $M_h^j$ may converge to similar directions.

SADE regularizes the LRM update by penalizing sample-agnostic inter-expert correlations:

$$\mathrm{RSR}^j = \sum_{h<r} \| M_h^j (M_r^j)^\top \|_F^2$$

resulting in a training objective:

$$L = L_\mathrm{task} + \frac{\alpha}{d^2} \sum_{j=1}^{\#\text{LRM}} \mathrm{RSR}^j$$

where $\alpha$ is a hyperparameter setting the strength of the regularization (Liu et al., 31 Dec 2025).

2.2 Skill-Diverse Expert Aggregation in Long-Tailed Recognition

In test-agnostic long-tailed recognition, a single backbone shares $K=3$ heads, each “expert” $v_k$ being trained to optimize for a specific simulated marginal: head-heavy, balanced, or tail-heavy. These experts are jointly optimized via their respective loss terms:

• Cross-entropy under empirical distribution for the forward expert,
• Balanced softmax loss for the uniform expert,
• Inverse softmax for the backward expert.

At test time, aggregation weights $w_k$ for the experts are inferred by maximizing prediction stability under input augmentation, without knowledge of the true test distribution. The objective is:

$$\max_{w} \mathcal{S}(w) = \frac{1}{n_t} \sum_{x \in D_t} \hat y^1(x) \cdot \hat y^2(x)$$

where $\hat y^j(x)$ are normalized predictions across two augmentations of $x$, and $\cdot$ denotes the inner product (Zhang et al., 2021).

3. Empirical Evaluation and Performance Impact

Empirical studies in CLoRA demonstrate that integrating SADE yields consistent accuracy improvements and increased parameter efficiency. On VTAB-1K (ViT-Base), enabling SADE led to a +1.7 percentage point (ppt) gain in mean accuracy (from 73.4% to 75.1%) compared to base-space sharing alone. Additionally, switching to the sample-agnostic SADE regularizer achieved nearly identical accuracy compared to a sample-dependent variant while reducing GFLOPs by approximately 93.9% (Liu et al., 31 Dec 2025).

In test-agnostic long-tailed recognition, SADE’s self-supervised expert aggregation approach consistently outperformed previous state-of-the-art models such as softmax, balanced softmax, and RIDE under uniform, forward-long-tailed, and backward-long-tailed test distributions. For instance, on ImageNet-LT under uniform test distribution, the top-1 accuracy of SADE reached 58.8%, compared to 56.3% for RIDE. Notably, this improvement is achieved without any supervision on the test marginal (Zhang et al., 2021).

4. Hyperparameters and Implementation Considerations

The effectiveness of SADE is governed by a set of tunable hyperparameters:

• $\alpha$ (regularization weight): Typically drawn from $\{0.1, 1.0, 10.0\}$ for image classification and $\{0.01, 0.1, 1.0, 10.0\}$ for point cloud tasks; optimal performance often achieved near $\alpha=1.0$ for images and $\alpha=0.1$ or $1.0$ for point clouds.
• $p$ (number of experts): Typical values are $p \in \{4, 6, 8, 10\}$, with $p=8$ yielding strong results.
• $r$ (expert rank): Usually fixed small (such as $r=8$) to maintain parameter efficiency.
• For test-agnostic long-tailed recognition, the number of experts is three, with aggregation weights learned at inference via the self-supervised stability objective.

Implementation of SADE in CLoRA involves injecting LRMs before Multi-Head Attention (MHA) and Feed-Forward Network (FFN) modules in each transformer layer, with all computation related to expert diversity handled during training; at inference, the expert sum is folded into the adapted weights, ensuring zero overhead (Liu et al., 31 Dec 2025).

5. Theoretical and Intuitive Justification

From a mixture-of-experts perspective, having $p$ diverse experts within an LRM allows the update $\Delta W_j$ to span a subspace of dimension up to $p\,r$. If experts are redundant, the effective rank is decreased and the model adaptation capacity is wasted. By penalizing inter-expert correlations so that $M_h^j (M_r^j)^\top \rightarrow 0$, SADE encourages each expert to specialize in complementary directions, maximizing representational diversity while retaining parameter efficiency.

In self-supervised aggregation (test-agnostic long-tailed recognition), maximizing prediction stability aligns the aggregated expert’s label distribution to the true, unknown test distribution and encourages lower entropy (more confident) predictions. Theoretically, the stability objective $\mathcal{S}(w)$ is proportional to $I(\hat Y; Y) - H(\hat Y)$, connecting it to mutual information between predictions and true labels (Zhang et al., 2021).

6. Comparison to Sample-Dependent and Other Diversity Mechanisms

SADE regularization is explicitly sample-agnostic, operating on the parameters of the model rather than intermediate activations dependent on inputs or batches. This results in computational savings, as sample-dependent measures incur a higher computational burden (e.g., $O(bp^2 n d^2)$ per LRM per batch versus $O(p^2 d^3)$ for SADE), with SADE being fully independent of batch size.

Empirically, SADE achieves nearly identical generalization performance compared to sample-dependent regularizers but at a fraction of the computational cost—e.g., a 93.9% reduction in GFLOPs in vision tasks (Liu et al., 31 Dec 2025). A plausible implication is that sample-agnostic approaches of this type will be favored in resource-constrained adaptation regimes.

Aspect	Sample-Agnostic (SADE)	Sample-Dependent
Regularization target	Parameter-level inter-expert similarity	Token/activation-level similarity
Computational complexity per batch	$O(p^2 d^3)$	$O(bp^2 n d^2)$
Dependence on input	None	High
Empirical accuracy (e.g. on VTAB-1K)	Similar	Similar
GFLOPs cost	Significantly lower	Higher

7. Practical Significance and Broader Impacts

Sample-Agnostic Diversity Enhancement facilitates scalable, efficient model adaptation by maximizing intra-module diversity without introducing excessive computational burden, especially for large-scale transformer architectures and in regimes of low-data or distribution shift. In collaborative low-rank adaptation, it enables parameter-efficient fine-tuning with enhanced downstream accuracy. In long-tailed recognition, it enables robust, test-time adaptation without requiring knowledge of the test label distribution (Liu et al., 31 Dec 2025, Zhang et al., 2021). The mechanisms underlying SADE may generalize to other mixture-of-expert systems and diverse representation learning settings. Empirical gains and reductions in computational cost suggest these techniques are likely to guide future research in efficient and adaptive learning with expert ensembles.