Weighted Modality Activation in Multimodal Models

• Weighted Modality Activation is a modality-attribution method that assigns explicit, sample-specific weights to inputs like satellite, weather, soil, and terrain for transparent prediction decomposition.
• The technique leverages a regression head to compute per-modality contributions, offering a distinct alternative to holistic approaches such as Shapley Value Sampling.
• Variations of WMA include static coefficients and adaptive activation methods that dynamically balance modality influence during both inference and training under data imbalance conditions.

Weighted Modality Activation (WMA) denotes an explicitly defined modality-attribution method in multimodal crop-yield prediction and, more broadly, a non-standardized family of mechanisms that assign unequal influence to modalities in multimodal models. In its named form, WMA decomposes a prediction into modality-wise contributions using modality-specific encoder outputs and the final regression head, yielding per-sample relevance scores for satellite, weather, soil, and terrain inputs (Najjar et al., 9 Aug 2025). Related literature uses closely allied ideas without a shared canonical terminology, ranging from fixed scalar fusion weights and sample-specific gates to routing-by-agreement, trust-weighted distillation, energy-based rebalancing, and modality-specific activation scaling for quantization (Wu et al., 2024, Wang et al., 2018, Tsai et al., 2020, Wang et al., 2024, Hu et al., 5 Mar 2026, Lu et al., 26 Feb 2026, He et al., 14 Apr 2026, Shen et al., 27 Aug 2025).

1. Term, scope, and conceptual lineage

In the strictest sense, WMA is the modality-attribution method proposed for multimodal crop-yield prediction, where separate modality encoders are fused by concatenation and followed by a linear regression head; WMA reads modality importance directly from that head and the learned modality embeddings (Najjar et al., 9 Aug 2025). The modalities in that setting are multispectral satellite time series, weather time series, soil properties, and terrain elevation maps, and the method is positioned as an intrinsic alternative to post-hoc Shapley-style attribution.

The surrounding literature suggests that WMA is better understood as a conceptual family rather than a universally standardized architecture. Early multimodal word-representation work already distinguished global, category-specific, and sample-specific modality gates, including vector-valued gates that reweight dimensions within each modality (Wang et al., 2018). In neuroimaging, a continuously annotated weighted stochastic block model coupled node-wise activation-like annotations with edge-wise connectivity through a tuning parameter $\alpha$, providing an explicit probabilistic trade-off between activation and network structure even though the term WMA was not used (Murphy et al., 2016). This suggests that “weighted modality activation” names a recurring design pattern: modality influence is controlled by explicit coefficients, but the locus of control may be the prediction head, the fusion module, the training objective, or a generative probabilistic model.

A recurring distinction across the literature is whether weighting is static or adaptive. Some methods use global coefficients shared across all samples, whereas others estimate sample-specific or concept-specific weights. This distinction is central to the later development of WMA-like systems.

2. Canonical formulation as modality attribution

In the crop-yield setting, WMA is defined after modality-specific encoding and concatenation fusion. For sample $i$, the fused representation is $\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$ and the prediction is decomposed as

$$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$

where $$\mathbf{w}=\operatorname{concat}(\mathbf{w}^{sa},\mathbf{w}^{w},\mathbf{w}^{so},\mathbf{w}^{dem})$$ is the final regression weight vector partitioned by modality, and $\hat{y}_i^m=\mathbf{w}^m\cdot \mathbf{z}_i^m$ is the contribution of modality $m$ (Najjar et al., 9 Aug 2025).

The WMA relevance score is then

$$\mathcal{R}_i^{m} = \left|\frac{\hat{y}_i^{m}}{\hat{y}_i-b}\right|.$$

Here, “activation” refers to the learned modality representation $\mathbf{z}_i^m$, “weighted” refers to multiplication by the corresponding regression-head slice $\mathbf{w}^m$, and the resulting score is a normalized relative contribution magnitude. The method is computed per pixel and, in the reported field-level analysis, aggregated by averaging modality scores over 32 randomly selected pixels per field (Najjar et al., 9 Aug 2025).

This formulation is architecture-grounded rather than perturbation-based. It does not traverse attention matrices, token alignments, or internal routing coefficients; instead, it attributes the final scalar prediction through the linear head. The same paper emphasizes that this is intended for multimodal regression with modality-specific encoders, concatenation fusion, and a linear or MLP-style regression head, and could be extended to other differentiable fusion/regression settings.

Empirically, the paper reports a strong divergence between WMA and Shapley Value Sampling (SVS). For corn, WMA averaged across fields assigns soil 41.3%, satellite 29.4%, DEM 15.1%, and weather 14.2%, whereas SVS assigns satellite 89.5%, weather 7.9%, soil below 2%, and DEM below 1% (Najjar et al., 9 Aug 2025). The same study also reports that Transformer-based models outperform convolutional and recurrent alternatives, achieving $i$0 gains of 0.10 and 0.04 at the subfield and field levels, respectively. The discrepancy between WMA and SVS is explicitly attributed to their computational basis: WMA relies only on the regression head, whereas SVS uses the entire model. This suggests that WMA measures head-level contribution rather than whole-model causal necessity.

3. Static coefficients versus adaptive activation

A large fraction of WMA-adjacent work concerns modality weights that are explicit but not truly adaptive. In zero-shot composed image retrieval, WeiMoCIR forms a query embedding by

$i$1

and final retrieval scores by

$i$2

Here $i$3 weights image versus text in query construction, and $i$4 weights image similarity versus caption similarity at retrieval time (Wu et al., 2024). The method uses validation-tuned global coefficients rather than per-query gates; the same paper explicitly states that it does not implement per-query adaptive gating, token-level activation, sample-dependent routing, or confidence-aware weighting. In that sense it is weighted modality fusion rather than modality activation in a strict adaptive sense.

MetaKD occupies an intermediate position. It learns a modality importance vector

$i$5

through bilevel meta-learning, normalizes it with softmax, and uses the resulting weights primarily to scale pairwise cross-modal distillation terms by $i$6 (Wang et al., 2024). The weights are dynamic during training but global at the task level rather than sample-specific at inference. This is closer to weighted modality supervision than to forward-pass activation gating.

By contrast, dynamic multimodal word representation learning offers a more direct activation-style formulation. Its sample-specific gate computes modality weights from the unimodal embeddings themselves,

$i$7

and fuses them by weighted concatenation,

$i$8

The same framework also includes global modality-specific gates and category-specific gates, making explicit a spectrum from corpus-level weighting to lexical sample-specific weighting (Wang et al., 2018). Taken together, these papers suggest a useful distinction: static scalar coefficients regulate modality balance globally, whereas stronger WMA formulations estimate activation weights from the current input or task state.

4. Concept-conditioned and cross-modal activation mechanisms

A more expressive line of work treats activation as both sample-specific and label-dependent. Multimodal Routing constructs unimodal, bimodal, and trimodal explanatory features and assigns each feature two multiplicative weights: an activation score $i$9 and a concept-routing coefficient

$\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$0

Concept vectors are then updated by

$\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$1

and the class logit becomes

$\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$2

This makes the effective contribution of feature $\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$3 to concept $\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$4 explicitly controlled by $\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$5 (Tsai et al., 2020). Relative to canonical WMA, this is concept-conditioned activation over unimodal and cross-modal interaction features rather than only raw modalities.

Attentive Modality Hopping in speech emotion recognition pushes activation one level deeper into within-modality temporal selection. It builds a context vector from two modalities and uses bilinear attention to summarize the third, for example

$\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$6

The process is repeated cyclically across video, audio, and text (Yoon et al., 2019). This is not an explicit scalar gate over whole modalities, but it is a dynamic, data-dependent reweighting of modality content conditioned on the other modalities.

These mechanisms clarify that “activation” can refer to different granularities. In crop-yield WMA, the attributed object is the final modality embedding $\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$7 (Najjar et al., 9 Aug 2025). In Multimodal Routing, it is a concept-specific explanatory feature score $\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$8 (Tsai et al., 2020). In Attentive Modality Hopping, it is a temporal attention distribution within a modality sequence (Yoon et al., 2019). The literature therefore supports a broad definition of WMA as a family of modulation schemes, but not a single canonical locus of weighting.

5. Balancing modality influence under missing, noisy, or imbalanced learning

A major development in WMA-like research is the shift from feature fusion to training-dynamics control. MWAM diagnoses modality dominance in the frequency domain through the Frequency Ratio Metric

$\mathbf{z}_i=\operatorname{concat}(\mathbf{z}_i^m)$9

then converts relative FRM into dynamic modality weights $$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$0 that scale branch gradients or auxiliary losses during training (Lu et al., 26 Feb 2026). The same paper states that MWAM is detached at inference, so the method functions as training-time modality activation control rather than inference-time gating.

AIM pushes this idea further by making weighting depth-aware and parameter-aware. At depth $$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$1, modality performance is transformed into soft weights

$$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$2

and the depth-wise modulation loss is

$$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$3

with depth imbalance itself weighted by

$$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$4

Dominant modalities are shifted toward degraded auxiliary or pseudo-weak blocks, while weaker modalities retain their full blocks (Shen et al., 27 Aug 2025). This is a hierarchical WMA variant: weights are assigned not only across modalities, but also across depths and branch variants.

EBMC combines implicit and explicit balancing. Its Energy-guided Modality Coordination defines

$$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$5

with equilibrium enforced by

$$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$6

Its Instance-aware Modality Trust Distillation then produces normalized sample-level trust weights

$$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$7

which weight distillation from unimodal teachers to the fused student (He et al., 14 Apr 2026). In effect, EMC acts as implicit training-time WMA, while IMTD is an explicit uncertainty-based WMA mechanism.

MetaKD belongs to the same robustness-oriented family, but with global task-level modality weights learned by meta-optimization rather than per-sample trust scores (Wang et al., 2024). Across these methods, the dominant interpretation of WMA shifts from “how much should this modality contribute to the fused feature?” to “how strongly should this modality be optimized, trusted, or distilled under imbalance and missingness?”

6. Extensions, ambiguities, and limitations

The term WMA is not semantically stable across the literature. Some papers use the phrase directly, some implement closely related mechanisms under different names, and one paper uses WMA to mean something entirely different.

Source	Mechanism	Relation to WMA
(Najjar et al., 9 Aug 2025)	Prediction-head modality attribution	Explicit use of “Weighted Modality Activation”
(Wu et al., 2024)	Global coefficients $$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$8 for fusion and scoring	Weighted modality fusion, not dynamic activation
(Asadollahzadeh et al., 28 May 2026)	Weighted Moving Average teacher over training trajectory	Acronym collision; not modality activation

Beyond naming, the literature also diverges in what is actually being weighted. MASQuant performs modality-specific activation scaling for quantization through diagonal smoothing matrices

$$\hat{y}_i = \mathbf{w}\cdot \mathbf{z}_i + b = \sum_m \mathbf{w}^{m}\cdot \mathbf{z}_i^{m} + b = \sum_m \hat{y}_i^m + b,$$9

optimized by modality-specific reconstruction loss (Hu et al., 5 Mar 2026). This is a static, calibration-based form of modality-aware activation weighting rather than attribution. In neuroimaging, the annotated weighted stochastic block model raises the annotation likelihood term to the power $$\mathbf{w}=\operatorname{concat}(\mathbf{w}^{sa},\mathbf{w}^{w},\mathbf{w}^{so},\mathbf{w}^{dem})$$0, explicitly controlling the contribution of activation-like node annotations relative to connectivity (Murphy et al., 2016). These examples show that WMA-like behavior can appear in quantization, generative modeling, and network analysis, not only in fusion modules.

Several limitations recur. First, many methods use global or task-level weights rather than sample-specific gates, as in WeiMoCIR and MetaKD (Wu et al., 2024, Wang et al., 2024). Second, head-based attribution can disagree sharply with whole-model perturbation methods, as shown by the divergence between WMA and SVS in crop-yield prediction (Najjar et al., 9 Aug 2025). Third, dynamic balancing methods may act only during training and not at inference, which changes the meaning of “activation” from forward-pass modulation to optimization control (Lu et al., 26 Feb 2026, Shen et al., 27 Aug 2025). Fourth, static weighting can fail on abstract concepts, subtle distinctions, or relative positioning, which WeiMoCIR identifies as limitations of fixed weighted averages (Wu et al., 2024). A plausible implication is that future WMA research will continue to move toward finer granularity—sample-specific, depth-specific, and uncertainty-aware modulation—while retaining the interpretability advantages of explicit weighting.

The present state of the field therefore supports two conclusions. Weighted Modality Activation is, in its explicit named form, a regression-head modality-attribution method for multimodal crop-yield prediction (Najjar et al., 9 Aug 2025). At the same time, the broader research landscape treats WMA as a family resemblance across multimodal systems: explicit coefficients or gates are used to regulate modality contribution, but the relevant object may be a fused embedding, a routing coefficient, a distillation term, a gradient path, or a quantization transform.