Distribution-aware Scaling Module (DSM)

• DSM enhances DFER by combining a robust supervised contrastive loss with adaptive, gradient-sensitive loss scaling to mitigate heterogeneity challenges.
• It dynamically adjusts the balance between classification and contrastive objectives using gradient norms as a proxy for batch difficulty, ensuring stable and discriminative learning.
• By incorporating an information bottleneck penalty, DSM compacts embeddings and reduces redundancy, improving robustness against multi-source data variations and expression variability.

Distribution-aware Scaling Module (DSM) is an adaptive optimization module introduced within the Heterogeneity-aware Distributional Framework (HDF) for Dynamic Facial Expression Recognition (DFER) to stabilize and improve discriminative learning under sample heterogeneity (Cui et al., 21 Jul 2025). In the formulation reported in "Learning from Heterogeneity: Generalizing Dynamic Facial Expression Recognition via Distributionally Robust Optimization" (Cui et al., 21 Jul 2025), DSM addresses optimization imbalance induced by multi-source data and individual expression variability through two coupled mechanisms: information-constrained, distributionally robust supervised contrastive learning, and gradient-driven adaptive balancing of classification and contrastive objectives. Within HDF, DSM operates at the loss and optimization level, complementing the Time-Frequency Distributional Attention Module (DAM), which handles robustness at the feature-modeling level.

1. Problem setting and role within HDF

The DSM is motivated by three heterogeneity challenges in DFER. First, in-the-wild DFER datasets aggregate videos from movies, news, interviews, and related sources, producing large variation in style, resolution, compression, and motion continuity. Second, individual variability in identity, age, intensity, and idiosyncratic expression patterns makes the same emotion heterogeneously expressed, inflating intra-class variance and yielding many hard or ambiguous samples. Third, under such heterogeneity, training gradients fluctuate and convergence becomes unstable; hard samples can dominate the signal, causing the classification objective to overwhelm feature learning and degrade generalization (Cui et al., 21 Jul 2025).

HDF comprises two plug-and-play components. DAM improves feature robustness against temporal inconsistency and frequency or style shifts via dual-branch temporal and frequency attention, guided by distributionally robust principles. DSM stabilizes and improves discriminative learning under heterogeneity by combining an information-constrained, distributionally robust supervised contrastive objective with gradient-sensitive objective balancing. The division of labor is explicit: DAM handles distributional robustness at the feature-modeling level, whereas DSM complements it at the optimization level.

The module is also framed in relation to Distributionally Robust Optimization (DRO). DSM aligns with DRO by reweighting negative pairs in the contrastive objective to reduce the undue influence of distributional outliers, described as heterogeneous hard negatives, and by adaptively scaling loss terms under batch-level difficulty, measured through gradient sharpness as a proxy for worst-case training scenarios. This suggests that DSM is intended not merely as a task-weighting heuristic, but as a training mechanism for heterogeneous batches in which difficult or atypical samples intermittently dominate the optimization dynamics.

2. Theoretical principles

DSM is built on two stated principles: a gradient sensitivity principle and an information bottleneck (IB) principle (Cui et al., 21 Jul 2025).

Under the gradient sensitivity principle, DSM estimates batch difficulty via a gradient norm proxy. Large norms indicate sharp or unstable regions, or hard batches; small norms indicate easier regimes. The module then shifts emphasis toward classification loss when gradients are large, described as stability first, and toward contrastive refinement when gradients are small, described as discriminability. The sensitivity metric is the $L_2$ norm of the gradient of the total loss:

$g = \left\|\nabla_w L_{\text{total}}\right\|_2.$

The paper relates this estimate to sharpness-aware ideas: larger $g$ implies sharper or unstable landscapes, or harder batches.

Under the information bottleneck principle, DSM constrains the representation to be compact and task-relevant by penalizing redundancy in the learned features. The implementation uses an IB surrogate defined as the total covariance energy of the batch embeddings $Z$. Let normalized embeddings be $Z = \{z_i \in \mathbb{R}^d\}$ with batch mean

$\bar z = \frac{1}{N}\sum_i z_i$

and covariance

$$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$

The IB penalty is the trace of the covariance:

$L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$

This surrogate is stated to operationalize an information bottleneck-style compression by discouraging excessive variance in the representation space. The paper further characterizes this as compressing $I(X;Z)$ while preserving $I(Z;Y)$ through supervision, thereby encouraging compact, discriminative clusters.

3. Loss construction and adaptive scaling

DSM combines a standard classification loss with a robust supervised contrastive term regularized by the IB penalty (Cui et al., 21 Jul 2025).

For classification, the module uses standard multi-class cross-entropy over a batch of size $g = \left\|\nabla_w L_{\text{total}}\right\|_2.$0. Let logits be $g = \left\|\nabla_w L_{\text{total}}\right\|_2.$1, class posterior $g = \left\|\nabla_w L_{\text{total}}\right\|_2.$2, and $g = \left\|\nabla_w L_{\text{total}}\right\|_2.$3 the ground-truth label:

$g = \left\|\nabla_w L_{\text{total}}\right\|_2.$4

For supervised contrastive learning, the similarity between normalized embeddings $g = \left\|\nabla_w L_{\text{total}}\right\|_2.$5 and $g = \left\|\nabla_w L_{\text{total}}\right\|_2.$6 is

$g = \left\|\nabla_w L_{\text{total}}\right\|_2.$7

where $g = \left\|\nabla_w L_{\text{total}}\right\|_2.$8 is a temperature. For each anchor $g = \left\|\nabla_w L_{\text{total}}\right\|_2.$9, the positive set $g$0 is defined as label-equal samples excluding $g$1, and negatives are all $g$2 not in $g$3.

DSM introduces distributionally robust negative weighting using a Gaussian kernel:

$g$4

where $g$5 is the Gaussian center controlling robust weighting of negatives. This weighting downweights outlier negatives by similarity deviation from $g$6. The robust supervised contrastive loss for anchor $g$7 is then

$g$8

Averaging over the batch and adding the IB penalty yields the IB-regularized robust contrastive loss:

$g$9

where $Z$0 is the IB trade-off inside the contrastive term.

The defining feature of DSM is its gradient-driven adaptive loss scaling. Given the batch difficulty proxy $Z$1, the module dynamically scales the two losses as

$Z$2

where $Z$3 and $Z$4 are base scaling coefficients. The final DSM training objective is

$Z$5

The stated intuition is direct: large $Z$6 emphasizes $Z$7 for stability, while small $Z$8 emphasizes $Z$9 to refine representation.

4. Training procedure, placement, and implementation details

DSM is placed after feature extraction. In the reported instantiation, fused features are produced by the backbone and DAM, and DSM then consumes logits for $Z = \{z_i \in \mathbb{R}^d\}$0 and normalized embeddings for $Z = \{z_i \in \mathbb{R}^d\}$1, returning $Z = \{z_i \in \mathbb{R}^d\}$2 for backpropagation (Cui et al., 21 Jul 2025). The input pipeline is described with a backbone or encoder $Z = \{z_i \in \mathbb{R}^d\}$3 such as X3D, a projection head $Z = \{z_i \in \mathbb{R}^d\}$4 for contrastive embeddings, and a classifier head $Z = \{z_i \in \mathbb{R}^d\}$5 for logits.

For each training iteration with batch $Z = \{z_i \in \mathbb{R}^d\}$6, the reported training routine proceeds as follows. First, compute representation vectors $Z = \{z_i \in \mathbb{R}^d\}$7 from DAM-fused features, logits $Z = \{z_i \in \mathbb{R}^d\}$8, and normalized contrastive embeddings

$Z = \{z_i \in \mathbb{R}^d\}$9

Second, compute the unweighted loss terms: $\bar z = \frac{1}{N}\sum_i z_i$0, the robust supervised contrastive loss, the covariance-based IB penalty, and then $\bar z = \frac{1}{N}\sum_i z_i$1. Third, form a provisional sum

$\bar z = \frac{1}{N}\sum_i z_i$2

and compute

$\bar z = \frac{1}{N}\sum_i z_i$3

where $\bar z = \frac{1}{N}\sum_i z_i$4 denotes trainable parameters $\bar z = \frac{1}{N}\sum_i z_i$5. The text notes that this operationalization avoids circularity when setting $\bar z = \frac{1}{N}\sum_i z_i$6 and $\bar z = \frac{1}{N}\sum_i z_i$7. Fourth, apply the adaptive loss scaling. Fifth, backpropagate $\bar z = \frac{1}{N}\sum_i z_i$8 and update parameters using AdamW and the cosine schedule.

Several implementation details are explicitly specified. The similarity temperature $\bar z = \frac{1}{N}\sum_i z_i$9 is set to $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$0 in the paper for stable or effective representation learning, and the Gaussian center $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$1 is set to $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$2. The paper does not numerically specify $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$3, $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$4, or $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$5; it states that $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$6 controls compactness, while $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$7 and $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$8 scale the dynamic weights $$\mathrm{Cov}(Z) = \frac{1}{N}\sum_{i=1}^N (z_i-\bar z)(z_i-\bar z)^\top.$$9 and $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$0. Training is reported with AdamW, and although the paper draws inspiration from SAM for “sharpness-aware” signals, no mandatory SAM two-step is required for DSM.

The module is described as backbone-agnostic and contrastive-agnostic. The paper instantiates it with X3D and supervised contrastive learning, while stating that any 3D CNN or transformer backbone, and other contrastive formulations including other supervised or semi-supervised variants, are compatible. The computational overhead involves pairwise similarities with $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$1 complexity, Gaussian weights for negatives, covariance and trace computation, and one provisional backward pass to measure $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$2. This is described as modest with standard DFER batch sizes. Numerical stability guidance includes normalizing embeddings before similarities, using $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$3 when computing norms, and caution with very small batches because contrastive statistics may be noisy.

5. Empirical evidence

The reported empirical evidence situates DSM within the broader HDF results and also isolates its contribution through ablation (Cui et al., 21 Jul 2025). On DFEW under 5-fold cross-validation, HDF achieves WAR $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$4 and UAR $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$5, surpassing prior methods including transformer- and vision-language-based models. On FERV39k, HDF achieves WAR $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$6 and UAR $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$7, reported as best among compared methods.

Ablation on DFEW fd5 provides the clearest isolated evidence for DSM:

Setting	WAR	UAR
Baseline (X3D + CE)	68.62%	58.21%
DAM only	69.73%	59.02%
DSM only	71.79%	59.93%
DAM + DSM (full HDF)	73.24%	61.31%

In this ablation, DSM alone improves over the baseline by $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$8 WAR and $L_{\mathrm{IB}} = \mathrm{Tr}(\mathrm{Cov}(Z)).$9 UAR. Combined with DAM, the total gains are $I(X;Z)$0 WAR and $I(X;Z)$1 UAR on fd5. These results are presented as evidence that DSM contributes materially even without DAM, while the full framework benefits from their complementarity.

Qualitative evidence is also reported. t-SNE plots show tighter intra-class clusters and clearer inter-class separation when DSM is used. Loss landscape visualization indicates a flatter, smoother surface under DSM, consistent with reduced gradient instability and better generalization in heterogeneous conditions. This suggests that the module’s combination of robust contrastive weighting, covariance regularization, and gradient-sensitive balancing alters both representation geometry and optimization behavior.

6. Relation to other balancing strategies, limitations, and extensions

The paper compares DSM conceptually with several families of balancing strategies (Cui et al., 21 Jul 2025). Uncertainty weighting learns task weights from predicted uncertainty; GradNorm adjusts by equalizing gradient norms across tasks; Dynamic Weight Averaging weighs tasks by loss rates; focal mechanisms reweight samples by difficulty at the instance level. DSM is distinguished by combining three elements in a single module: Gaussian reweighting $I(X;Z)$2 in contrastive space to downweight outlier or hard negatives that would otherwise dominate the denominator, the IB constraint $I(X;Z)$3 to shrink redundant variance and improve cluster compactness, and a single batch-difficulty proxy $I(X;Z)$4 to dynamically trade off classification and contrastive learning. The paper states that this combination is particularly well-suited to heterogeneous batches that intermittently contain hard samples.

The same source also delineates cases in which DSM may offer less benefit. With very small batch sizes, contrastive statistics and covariance estimates become noisy, and benefits may diminish. In near-perfectly balanced or clean datasets, there are fewer hard negatives or outliers, reducing the marginal gain of robust negative reweighting and IB. Under extreme and persistent class imbalance without any sampling or class-reweighting, DSM improves stability but may need to be combined with class-balanced sampling or class-weighted $I(X;Z)$5.

Practical guidance in the paper follows from these constraints. It recommends computing $I(X;Z)$6 with the provisional, unweighted sum $I(X;Z)$7 to avoid circularity, normalizing embeddings before similarity computation, and monitoring $I(X;Z)$8 to avoid over-compression or representation collapse. It also recommends starting with the default $I(X;Z)$9 and $I(Z;Y)$0 and tuning $I(Z;Y)$1 conservatively to the $I(Z;Y)$2 to $I(Z;Y)$3 range if needed, while noting that the paper does not specify a fixed value. If $I(Z;Y)$4 collapses on very hard batches with very large $I(Z;Y)$5, the text suggests mild floors or ceilings on $I(Z;Y)$6 and $I(Z;Y)$7, or smoothing $I(Z;Y)$8 over iterations such as EMA; these are described as standard stabilizers though not mandated by the paper.

Because DSM is a loss-level module, the paper further states that it transfers to audio-visual or multimodal DFER by applying the same robust contrastive and adaptive scaling on fused embeddings. It also identifies other tasks with heterogeneity and hard negatives, including action recognition and micro-expression recognition, as settings that can benefit from the same combination of robust contrastive reweighting, IB regularization, and gradient-based balancing.