Gradient-guided Adaptive Modulation (GAM)

• Gradient-guided Adaptive Modulation (GAM) is an adaptive technique that rebalances multi-modal training to mitigate modality competition.
• It isolates mono-modal responses via Shapley sampling and adjusts each modality's gradient using a discrepancy ratio from historical averages.
• Empirical results demonstrate that GAM boosts underperforming modalities while suppressing dominant ones, improving accuracy across multiple datasets.

Searching arXiv for the primary GAM paper and closely related multimodal modulation work. arXiv search query: (Li et al., 2023) Gradient-guided Adaptive Modulation (GAM), presented in the multi-modal learning literature as an adaptive gradient modulation method, is a training procedure for end-to-end joint optimization of multi-modal models under modality competition, the phenomenon in which one modality suppresses learning from other modalities during standard joint training. The method was introduced in "Boosting Multi-modal Model Performance with Adaptive Gradient Modulation" and is designed to boost the performance of multi-modal models with various fusion strategies by dynamically suppressing branches that are temporarily too strong and boosting those that are temporarily too weak, using mono-modal response estimates, running discrepancy statistics, and gradient reweighting (Li et al., 2023).

1. Problem setting and objective

GAM is motivated by the observation that, while the field of multi-modal learning keeps growing fast, the deficiency of the standard joint training paradigm has become clear through recent studies. The paper attributes the sub-optimal performance of the jointly trained model to the modality competition phenomenon. In the step-by-step exposition, this is described more directly: in end-to-end multi-modal joint training, one "strong" modality tends to dominate, suppressing learning from weaker but complementary modalities (Li et al., 2023).

The stated goal of GAM is to dynamically rebalance training so that each modality’s branch contributes proportional to its informativeness, thereby encouraging the model to exploit all modalities. The key idea is iterative and gradient-centric. At each iteration, the method isolates each modality’s "mono-modal response" via a Shapley-value-style attribution, computes a scalar mono-modal cross-entropy for each modality, compares these per-modality scores to a historical running-average reference, and then exponentially reweights each modality’s gradient signal according to the resulting discrepancy ratio.

A central claim of the method is architectural generality. Existing modulation methods are described as effective but limited to late fusion models. GAM is presented as applicable to any fusion strategy, including early, late, and hybrid fusion, because it never assumes separate output heads; instead, mono-modal responses are isolated by masking plus Shapley sampling. This suggests that the contribution of the method is not only a new weighting rule, but also a modality-isolation mechanism compatible with fused internal representations.

2. Mathematical construction

The formulation begins with a full multi-modal input $x=(x^{1},\dots,x^{K})$, a logit output $\phi(x)\in\mathbb R^{C}$, trainable parameters $\theta$, and a standard training loss $\mathcal L(\phi(x),y)$. Modalities are indexed by $m\in\{1,\dots,K\}$ (Li et al., 2023).

For each modality $m$, GAM defines a mono-modal response by a Shapley-style decomposition: $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$ where $x_S$ means zero-masking all modalities not in $S$. The construction satisfies

$\phi(x)=\sum_{m=1}^K\phi^m(x).$

The per-modality scalar used for modulation is the mono-modal cross-entropy

$\phi(x)\in\mathbb R^{C}$0

This quantity is interpreted as the average cross-entropy if only modality $\phi(x)\in\mathbb R^{C}$1 "fired." Historical reference values are maintained by a running average

$\phi(x)\in\mathbb R^{C}$2

The discrepancy stage compares current and historical inter-modality differences: $\phi(x)\in\mathbb R^{C}$3

$\phi(x)\in\mathbb R^{C}$4

The paper’s intuition is that if $\phi(x)\in\mathbb R^{C}$5 is much smaller than others, modality $\phi(x)\in\mathbb R^{C}$6 is "strong."

Given a positive hyper-parameter $\phi(x)\in\mathbb R^{C}$7, GAM defines an adaptive modulation coefficient

$\phi(x)\in\mathbb R^{C}$8

If $\phi(x)\in\mathbb R^{C}$9, the modality is stronger than its historical norm and $\theta$0, so its gradient is suppressed. If $\theta$1, the modality is weaker and $\theta$2, so its gradient is boosted.

The resulting modulated update is

$\theta$3

where $\theta$4 is the base learning rate. No extra smoothing or regularizer is introduced beyond the usual weight decay in the $\theta$5 update; the only "smoothing" is the running average $\theta$6.

3. Training procedure and operational interpretation

In algorithmic form, GAM augments a standard training loop with modality-wise attribution, discrepancy estimation, and gradient recombination (Li et al., 2023). For each iteration, a batch is sampled, full logits $\theta$7 are computed, and for each modality $\theta$8, the algorithm computes $\theta$9 via Shapley sampling and then the mono-modal loss

$\mathcal L(\phi(x),y)$0

The running means $\mathcal L(\phi(x),y)$1 are updated, then $\mathcal L(\phi(x),y)$2, $\mathcal L(\phi(x),y)$3, and $\mathcal L(\phi(x),y)$4 are computed for each modality. During back-propagation, the upstream gradient on the full logits is combined with the sum of modality-specific Jacobians weighted by $\mathcal L(\phi(x),y)$5, after which the parameters are updated with the base learning rate and any weight-decay step.

Operationally, the method uses present-versus-history mismatch as a control signal. A modality that is currently outperforming its historical relative strength is attenuated; a modality that is currently underperforming its historical relative strength is amplified. The paper summarizes this as encouraging the model to rely on the more informative modality. A plausible implication is that GAM is not enforcing equal contribution across modalities; rather, it is attempting to prevent transient domination from collapsing the optimization dynamics.

The paper also emphasizes that the exact computation of $\mathcal L(\phi(x),y)$6 can be expensive because it sums over $\mathcal L(\phi(x),y)$7 subsets. To reduce this cost, the method may approximate the Shapley sum by Monte-Carlo over a few random subsets per batch. This computational observation is important for practical deployment in settings with more than a small number of modalities.

4. Competition-strength metric and the mono-modal concept

A substantial part of the contribution is not only modulation but also quantification. To obtain a quantitative understanding of modality competition and the mechanism behind the effectiveness of the modulation method, the paper introduces a novel metric to measure competition strength, built on the mono-modal concept, a function designed to represent the competition-less state of a modality (Li et al., 2023).

For modality $\mathcal L(\phi(x),y)$8, the mono-modal concept $\mathcal L(\phi(x),y)$9 is defined as a surrogate "competition-free" predictor. In late fusion, this is obtained by retraining only the $m\in\{1,\dots,K\}$0-branch head on $m\in\{1,\dots,K\}$1. In early or hybrid fusion, the full network $m\in\{1,\dots,K\}$2 is trained while feeding $m\in\{1,\dots,K\}$3, with all other modalities zero-masked. In all cases, $m\in\{1,\dots,K\}$4 is the resulting logit.

The joint model’s penultimate feature $m\in\{1,\dots,K\}$5 is then linearly probed by ridge regression,

$m\in\{1,\dots,K\}$6

with

$m\in\{1,\dots,K\}$7

Competition strength is defined by

$m\in\{1,\dots,K\}$8

where $m\in\{1,\dots,K\}$9 is the sample mean of $m$0. By construction,

$m$1

The interpretation is explicit. If $m$2, the joint model’s features can perfectly linearly recover the competition-free concept, indicating weak competition. If $m$3, the joint features fail to match the mono-modal concept, indicating strong competition. This metric turns the qualitative idea of modality suppression into a probeable property of learned representations rather than only an outcome-level performance difference.

5. Empirical findings and observed behavior

Across five datasets—AV-MNIST, CREMA-D, UR-Funny, AVE, and CMU-MOSEI—and both early, late, and hybrid fusion, GAM is reported to improve total accuracy by 1–20 points over the joint-training baseline and to outperform prior per-modality schemes including MSLR, MSES, gradient-blending, and OGM-GE (Li et al., 2023).

The paper reports that GAM also changes the competition-strength profile $m$4. More informative modalities, defined in the exposition as those with higher mono-modal "concept" accuracy, see their $m$5 decrease, meaning that they are less compressed by competition. Less informative modalities often get a mild increase in $m$6 as they are boosted to find useful signal. This suggests that the modulation does not simply minimize every modality’s competition score; instead, it redistributes training pressure in a way that tracks informativeness.

A concrete example is given for AV-MNIST with late fusion. Under JointTrain, the reported values are $m$7, audio-$m$8, and vision-$m$9. Under GAM, the reported values are $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$0 with a $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$1 increase, $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$2, and $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$3, with both modalities showing decreased competition. The paper summarizes the broader pattern as confirmation of the intuition that the modulation encourages the model to rely on the more informative modality.

An additional empirical nuance is that the jointly trained model typically has a preferred modality on which the competition is weaker than on other modalities. However, this preferred modality need not dominate others. This observation is relevant because it distinguishes weak competition from dominance: a modality can be relatively protected from suppression without being the sole driver of predictive performance.

6. Hyper-parameters, scope, and terminology

The practical guidance accompanying GAM is narrowly targeted to reproducibility. The base learning rate $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$4 is to use the same schedule as the unimodal or joint baseline, with $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$5 with decay given as an example. The modulation strength $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$6 controls the sensitivity of $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$7 to discrepancy; typical values in the experiments range from $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$8 to $$\phi^m(x)\;=\;\sum_{S\subseteq\{1,\dots,K\}\setminus\{m\}} \frac{|S|!\,(K-|S|-1)!}{K!} \bigl[\phi(x_S\cup\{m\})-\phi(x_S)\bigr],$$9. If $x_S$0 is too small, then $x_S$1 and the method has no effect; if it is too large, unstable swings may occur. The recommended tuning heuristic is to ensure $x_S$2 roughly during early epochs. For the competition-strength probe, the ridge regression $x_S$3 should be large enough, with $x_S$4–$x_S$5 given as an example, so that $x_S$6 stabilizes on the validation set (Li et al., 2023).

A common misunderstanding is to treat GAM as a late-fusion-only technique because earlier modulation methods were described as limited in that way. The formulation in fact states the opposite: GAM applies to any fusion strategy because mono-modal responses are isolated by masking plus Shapley sampling rather than by assuming separate output heads. Another potential confusion is terminological. The acronym "GAM" is also used in "GAM-Depth: Self-Supervised Indoor Depth Estimation Leveraging a Gradient-Aware Mask and Semantic Constraints," but there it denotes a gradient-aware mask for photometric loss weighting, implemented as a Sobel filter plus a fixed sigmoid and not as modality-wise gradient modulation (Cheng et al., 2024).

Taken together, these details place Gradient-guided Adaptive Modulation within a specific line of work on correcting optimization imbalances in multi-modal joint training. Its defining features are Shapley-based mono-modal isolation, discrepancy-ratio tracking against running historical references, exponential gradient reweighting, and a separate competition-strength metric grounded in a competition-free mono-modal concept.