Disentangled Gradient Learning (DGL)

• Disentangled Gradient Learning (DGL) is a multimodal framework that decouples gradient flow by using unimodal losses for encoders and multimodal losses for fusion and classifiers.
• The approach addresses an optimization conflict where fusion suppresses encoder gradients, ensuring modality-specific representations are effectively learned.
• Empirical results across audio-visual and RGB-D tasks demonstrate significant improvements over baselines, underscoring DGL’s practical advantage in multimodal learning.

Searching arXiv for the specific DGL papers to ground the article in the cited literature. Disentangled Gradient Learning (DGL) is a multimodal learning framework that decouples the optimization of modality encoders from the optimization of the modality fusion module. In the formulation introduced in "Boosting Multimodal Learning via Disentangled Gradient Learning" (Wei et al., 14 Jul 2025), DGL truncates the gradient back-propagated from the multimodal loss to each modality encoder and replaces it with gradients from unimodal losses, while simultaneously removing the gradient back-propagated from unimodal losses to the modality fusion module. The method is motivated by an optimization conflict in multimodal models: cross-modal fusion can decrease the gradient passed back to each modality encoder compared with unimodal training, which in turn contributes to under-optimization of modality-specific representations. The acronym DGL is also used in earlier CNN training literature for "Decoupled Greedy Learning" (Belilovsky et al., 2019, Belilovsky et al., 2021); in the multimodal context considered here, however, it denotes Disentangled Gradient Learning.

1. Conceptual scope and naming

Disentangled Gradient Learning was proposed for multimodal learning settings in which multiple modality-specific encoders feed a shared fusion module and classifier. The core objective is not to redesign the architecture, but to alter gradient routing during training so that different parameter subsets are optimized by different losses. In the two-modality setting emphasized in the original presentation, each modality has an encoder $f_m(\cdot;\theta_m)$, the modality representations are aggregated by a fusion module $\phi(\cdot;\theta_f)$, and the final logits are produced by a linear classifier with parameters $W,b$ (Wei et al., 14 Jul 2025).

The naming requires care because the abbreviation DGL already has an established meaning in another line of work. In "Decoupled Greedy Learning of CNNs" (Belilovsky et al., 2019) and its extended distributed-learning treatment (Belilovsky et al., 2021), DGL refers to a method for update-unlocked training of CNN modules via local supervised heads. Those papers address backpropagation locking in deep convolutional networks, whereas Disentangled Gradient Learning addresses encoder–fusion interference in multimodal optimization. The shared acronym therefore masks two distinct research programs.

Within the multimodal formulation, DGL is defined by two gradient-routing rules. First, encoder parameters are updated only by unimodal losses. Second, fusion and classifier parameters are updated only by the multimodal loss. This establishes a module-wise separation between representation learning and cross-modal integration, rather than balancing all objectives inside a single composite backward pass (Wei et al., 14 Jul 2025).

2. Optimization conflict in multimodal models

The central theoretical claim behind Disentangled Gradient Learning is that multimodal fusion can suppress the gradients received by modality encoders relative to unimodal training. The paper analyzes this effect under concatenation fusion with a shared linear classifier and cross-entropy loss. For sample $i$, modality encoders produce representations $h_i^{(m)} \in \mathbb{R}^{d_m}$, the fusion module produces $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$, and the classifier computes logits $s_i = W u_i + b$ (Wei et al., 14 Jul 2025).

For the two-modality case used throughout most derivations, let $z_i^{(m_1)}=h_i^{(m_1)}$, $z_i^{(m_2)}=h_i^{(m_2)}$, and $z_i^\tau=[z_i^{(m_1)}; z_i^{(m_2)}]$. Writing $\phi(\cdot;\theta_f)$0, the multimodal logits are

$\phi(\cdot;\theta_f)$1

while the unimodal logits for modality $\phi(\cdot;\theta_f)$2 are

$\phi(\cdot;\theta_f)$3

The corresponding feature-space gradients under cross-entropy are

$\phi(\cdot;\theta_f)$4

and

$\phi(\cdot;\theta_f)$5

By the chain rule, the encoder-parameter gradients are obtained by left-multiplication with the encoder Jacobian (Wei et al., 14 Jul 2025).

The suppression argument is expressed through the contribution of the second modality to the logits. Let $\phi(\cdot;\theta_f)$6. Then $\phi(\cdot;\theta_f)$7. Under the assumption that optimization pushes $\phi(\cdot;\theta_f)$8 toward the correct class center and away from others, the paper states

$\phi(\cdot;\theta_f)$9

and equality to $W,b$0 for $W,b$1. This implies that the correct-class margin is larger in the multimodal setting, so $W,b$2 and $W,b$3 for $W,b$4. The paper concludes that

$W,b$5

and correspondingly

$W,b$6

It summarizes this as $W,b$7 (Wei et al., 14 Jul 2025).

The significance of this result is that even a dominant modality may underperform its unimodal counterpart inside a multimodal model. The mechanism is not merely inter-modality imbalance in the usual sense; rather, the fusion process itself reduces the error signal delivered to encoders. This suggests that multimodal under-optimization is partly structural, arising from the interaction between encoder learning and fusion learning.

3. Gradient disentanglement mechanism

Disentangled Gradient Learning addresses the suppression effect by separating the sources of supervision for different parameter groups. The multimodal model still computes a multimodal loss and unimodal losses, but these losses are not allowed to update all components indiscriminately. Encoder updates are defined as

$W,b$8

while fusion and classifier updates are defined as

$W,b$9

Thus, multimodal gradients are truncated before they reach the encoders, and unimodal gradients are truncated before they reach the fusion and classifier (Wei et al., 14 Jul 2025).

The unimodal objectives are implemented by a modality-dropout mechanism that reuses the same fusion module and classifier while setting other modalities to zero. For two modalities, the paper defines

$i$0

and

$i$1

The multimodal loss remains

$i$2

DGL does not collapse these into a single conventional scalar objective with unrestricted backpropagation. Instead, it uses separate backward passes or equivalent gradient masking so that the gradients are routed to disjoint parameter subsets (Wei et al., 14 Jul 2025).

A key implementation device is detached multimodal forward propagation. The paper computes a detached multimodal representation

$i$3

and then defines

$i$4

Because the modality representations are detached, $i$5 cannot back-propagate into the encoders. Conversely, when propagating unimodal losses, gradients in the fusion module and classifier are zeroed or blocked so that unimodal supervision affects only the corresponding encoder (Wei et al., 14 Jul 2025).

The method introduces a scalar $i$6 to scale the unimodal losses for encoder optimization. The paper describes the training objective conceptually as

$i$7

and emphasizes that the operation is effectively a concatenation of separate optimization paths rather than a naive summation, because the two gradient flows are decoupled. A plausible implication is that DGL should be understood less as a new loss function than as a constrained gradient-routing scheme.

4. Training procedure and implementation

The training loop consists of three forward computations and two backward phases in the two-modality case. First, each modality encoder computes its representation: $i$8 and $i$9. Second, unimodal forward passes are formed by modality dropout through the shared fusion module: $h_i^{(m)} \in \mathbb{R}^{d_m}$0 and $h_i^{(m)} \in \mathbb{R}^{d_m}$1, from which $h_i^{(m)} \in \mathbb{R}^{d_m}$2 and $h_i^{(m)} \in \mathbb{R}^{d_m}$3 are computed. Third, a multimodal forward pass uses detached modality features: $h_i^{(m)} \in \mathbb{R}^{d_m}$4, yielding $h_i^{(m)} \in \mathbb{R}^{d_m}$5 (Wei et al., 14 Jul 2025).

Backpropagation then proceeds in two stages. In the encoder-update stage, the model backpropagates $h_i^{(m)} \in \mathbb{R}^{d_m}$6 while preventing gradients from reaching the fusion and classifier. In the fusion/classifier-update stage, the model backpropagates $h_i^{(m)} \in \mathbb{R}^{d_m}$7 while preventing gradients from reaching the encoders. The paper presents this in PyTorch-like code by toggling requires_grad_ flags for the relevant parameter groups, although it also notes that a single optimizer with parameter groups and manual masking is possible (Wei et al., 14 Jul 2025).

Several implementation details are stated explicitly. No extra classifiers are introduced: unimodal and multimodal paths share the same classifier $h_i^{(m)} \in \mathbb{R}^{d_m}$8 and the same fusion module $h_i^{(m)} \in \mathbb{R}^{d_m}$9. Detaching affects backward propagation only, not forward statistics accumulation, so BatchNorm or LayerNorm statistics update normally during forward execution. For cross-attention transformers or dense fusion blocks, the prescribed analogue is to apply stop-gradient to modality token streams before cross-modal fusion when computing the multimodal loss, and to feed only one modality while masking others to zeros for the unimodal losses (Wei et al., 14 Jul 2025).

The paper also states that the two backward passes increase compute modestly, but parameter count remains unchanged because the method does not add auxiliary heads. This distinguishes DGL from methods that enlarge the model to support separate unimodal predictors. The design choice suggests that the intervention is primarily algorithmic rather than architectural.

5. Empirical results across tasks and fusion regimes

Disentangled Gradient Learning was evaluated on audio-visual classification, multimodal sentiment classification, and RGB-D semantic segmentation. The audio-visual benchmarks were CREMA-D with 6 emotions, Kinetics-Sounds with 34 action classes, and VGGSound with 309 categories. MOSI was used for multimodal sentiment classification with Audio, Visual, and Text modalities, and NYUv2 for 40-class RGB-D segmentation. The experiments covered both simple concatenation fusion and dense cross-modal interaction mechanisms such as MMTM and MLP-Mixer (Wei et al., 14 Jul 2025).

For audio-visual classification with ResNet-18 encoders and concatenation fusion, the paper reports the following accuracies. Audio-only baselines were 62.18 on CREMA-D, 48.7 on Kinetics-Sounds, and 45.13 on VGGSound. Visual-only baselines were 68.23, 54.6, and 30.68, respectively. The multimodal baseline using concatenation achieved 65.10, 64.45, and 47.90. Under DGL, the unimodal branches inside the multimodal model reached Audio 63.12 and Visual 69.11 on CREMA-D, Audio 52.89 and Visual 60.11 on Kinetics-Sounds, and Audio 47.13 and Visual 33.45 on VGGSound. The DGL multimodal accuracies were 77.48 on CREMA-D, 74.78 on Kinetics-Sounds, and 52.53 on VGGSound. The paper states that these multimodal results exceed the second-best methods by +3.15%, +3.66%, and +1.34%, respectively (Wei et al., 14 Jul 2025).

For dense cross-modal interaction, the paper reports that on CREMA-D with MMTM fusion, DGL achieved Audio 60.30, Visual 69.50, and Multi 75.00, compared with a multimodal baseline of 60.12. On MOSI with MLP-fusion and three modalities, DGL achieved Audio 44.78, Visual 55.11, Text 77.39, and Multi 79.78, compared with a multimodal baseline of 76.83. On NYUv2 semantic segmentation using ESANet, DGL improved mIoU from 38.59 to 41.67 when training from scratch, and from 48.48 to 50.10 with ImageNet pretraining (Wei et al., 14 Jul 2025).

The ablations are central to the argument. On CREMA-D, the baseline accuracy was 65.10. Adding only multimodal truncation (MT), meaning truncating multimodal gradients to encoders, raised accuracy to 73.31. Adding only unimodal truncation (UT), meaning truncating unimodal gradients to fusion, raised accuracy to 74.22. Combining both as DGL yielded 77.48. The sensitivity study for $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$0 on CREMA-D reported 73.35 at $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$1, 75.46 at $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$2, 76.13 at $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$3, 77.48 at $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$4, and 77.08 at $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$5. The paper also reports that the attenuation factor

$u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$6

is $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$7 and approaches $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$8 after approximately 20 epochs in vanilla multimodal training, coinciding with a rapid drop in encoder gradient magnitude for the audio branch, whereas DGL maintains larger encoder gradients past epoch 20 (Wei et al., 14 Jul 2025).

A table is useful for consolidating the headline multimodal results.

Setting	Baseline	DGL
CREMA-D classification	65.10	77.48
Kinetics-Sounds classification	64.45	74.78
VGGSound classification	47.90	52.53
NYUv2 mIoU, from scratch	38.59	41.67
NYUv2 mIoU, ImageNet pretrain	48.48	50.10

These results were obtained under task-specific training setups explicitly reported in the paper. For audio-visual classification, the encoders were ResNet-18 with 512-dimensional outputs; audio used librosa spectrograms with window length 512 and overlap 353; CREMA-D used spectrogram size 257×299 and one frame per clip, whereas Kinetics-Sounds and VGGSound used 257×1004 and three frames per clip; optimization used SGD with momentum 0.9, weight decay $u_i = \phi(h_i^{(1)},\ldots,h_i^{(M)};\theta_f)$9, initial learning rate $s_i = W u_i + b$0 decayed by $s_i = W u_i + b$1 every 70 epochs, batch size 100, and 64 epochs. For NYUv2 with ESANet, the optimizer was Adam, training lasted 300 epochs, batch size was 8, and the learning rate was $s_i = W u_i + b$2 (Wei et al., 14 Jul 2025).

6. Relation to prior methods, assumptions, and limitations

The paper positions DGL against gradient modulation and unimodal-assistance approaches. Methods such as OGM, AGM, and PMR rebalance gradients within a unified objective, so encoders and fusion remain updated by the same mixed loss. Methods such as G-Blending, MLA, MMPareto, and Diagnosing and Re-learning add unimodal losses, but still allow those losses to update the fusion together with the encoders. DGL differs by disentangling module-wise optimization: encoders are exclusively optimized by unimodal losses, while fusion and classifier are exclusively optimized by the multimodal loss (Wei et al., 14 Jul 2025).

The contrast is also made at the level of general multi-objective optimization techniques. GradNorm balances task losses by scaling gradients to equalize training rates, MGDA computes a Pareto-stationary direction, and PCGrad projects conflicting gradients to reduce interference. In all of these cases, the gradients are still mixed over shared parameter subsets. DGL instead routes the gradients to disjoint parameter sets, eliminating encoder–fusion interference by construction (Wei et al., 14 Jul 2025).

The theoretical suppression claim is not presented as universal. It is stated under concatenation fusion, a shared linear classifier, cross-entropy training, and the assumption that non-target class contributions from the auxiliary modality are smaller than the target-class contribution. If another modality is adversarial and favors wrong classes, the suppression property may fail or become pathological. The paper also notes that if strong multimodal synergies require encoders to co-adapt in ways not captured by unimodal losses, then optimizing encoders exclusively by unimodal gradients may underfit the demands of fusion. In that case, tuning $s_i = W u_i + b$3 becomes important, and the text suggests that mixed training schedules with occasional joint gradients may be necessary; this is presented as a possible mitigation rather than a reported core component (Wei et al., 14 Jul 2025).

The method nevertheless scales naturally to $s_i = W u_i + b$4 modalities. For each modality $s_i = W u_i + b$5, one computes $s_i = W u_i + b$6 by feeding $s_i = W u_i + b$7 into fusion and routes $s_i = W u_i + b$8 only to $s_i = W u_i + b$9. The detached multimodal loss is computed using stop-gradient on all modality representations and routed only to fusion and task heads. The paper further states that the same routing principle can extend to complex fusion modules, attention mechanisms, transformers, and self-supervised multimodal objectives, provided unimodal variants are well-defined and parameter routing is enforced consistently (Wei et al., 14 Jul 2025).

A final source of confusion remains terminological. In the broader arXiv literature, DGL may denote either Disentangled Gradient Learning for multimodal optimization (Wei et al., 14 Jul 2025) or Decoupled Greedy Learning for update-unlocked CNN training (Belilovsky et al., 2019, Belilovsky et al., 2021). The two share a family resemblance at a high level—both alter gradient flow to relax dependencies in learning—but they operate in different settings, optimize different objectives, and use the acronym for unrelated methodological constructions.