Beyond Two-Modality Weighting (BTW)

• The paper introduces BTW as a bi-level framework that uses instance-specific KL divergence and global MI to address noise domination in multimodal MoE systems.
• It dynamically computes modality weights without extra parameters, delivering improved performance on benchmarks like CMU-MOSI and MIMIC-IV.
• BTW scales to an arbitrary number of modalities and stabilizes output variance, making it applicable to tasks in sentiment analysis, emotion recognition, and clinical classification.

Searching arXiv for the BTW paper and closely related multimodal weighting work to ground the article. {"query":"arXiv (Hou et al., 25 Aug 2025) Beyond Two-modality Weighting multimodal weighting mutual information KL divergence mixture of experts", "max_results": 5} to=arxiv_search d天天json {"query":"(Hou et al., 25 Aug 2025) Beyond Two-modality Weighting", "max_results": 10} to=arxiv to=search_arxiv Beyond Two-modality Weighting (BTW) is a non-parametric, bi-level weighting framework for multimodal model integration that was introduced to address the limitations of multimodal Mixture-of-Experts (MoE) systems when the number of modalities exceeds two and additional modalities may contribute noise as well as complementary signal (Hou et al., 25 Aug 2025). It combines instance-level Kullback-Leibler (KL) divergence and modality-level mutual information (MI) to dynamically adjust modality importance during training, requires no additional parameters, and is designed to scale to an arbitrary number of modalities. In the formulation reported for sentiment regression and clinical classification, BTW uses unimodal and multimodal output distributions as reference points, derives per-example weights from KL divergence, derives global modality weights from MI, and then smooths the combined weights across epochs.

1. Conceptual scope and motivation

BTW was proposed against a background in which multimodal MoE architectures had become effective for fusing heterogeneous data, but also exhibited two recurrent problems: noise domination, in which extra modalities can inject more noise than complementary information, and lack of fine-grained control, in which existing information-theoretic or parametric weighting methods fail to capture both example-specific and modality-wide variation (Hou et al., 25 Aug 2025). On the CMU-MOSI benchmark, the motivating observation was that a text-only expert can outperform the full multimodal model. This observation was used to argue that multimodal aggregation does not automatically imply improved multimodal utilization.

The framework is explicitly positioned beyond pairwise or fixed weighting. MI-based methods are described as scoring modality alignment globally while ignoring per-example variation; Partial Information Decomposition is described as computationally intractable beyond two modalities and lacking instance-level granularity; and parametric gating or attention mechanisms are described as learning weights at the cost of additional parameters and training complexity (Hou et al., 25 Aug 2025). BTW addresses these points by combining local and global signals without adding trainable weights.

This motivation is consistent with a broader pattern across multimodal research. In audio-visual speech recognition, static fusion is described as providing no per-utterance adaptation, no account of decoding-time contribution changes, and no temporal correspondence analysis (Cappellazzo et al., 12 Mar 2026). In multimodal conversational emotion recognition, prior two-modality methods are described as rebalancing only pairs of modalities and ignoring true three-way interactions (Nguyen et al., 2024). In PET/CT segmentation, fixed weighting is contrasted with target-aware gating that adapts to tumor-related features (Lu et al., 2024). In multi-modal super-resolution, static scalar weights are contrasted with per-patch and time-dependent weighting (Luo et al., 11 May 2026). BTW belongs to this larger transition from static pairwise weighting toward adaptive weighting over more than two modalities, but its mechanism is specifically non-parametric and output-distribution-based.

2. Bi-level weighting formalism

BTW assumes $M$ modalities. For each training example indexed by $i$, modality $m$ produces a unimodal output distribution $p_{uni_m}(y \mid x^{(i)})$, and the unweighted multimodal MoE backbone produces a multimodal output distribution $p_{multi}(y \mid x^{(i)})$ (Hou et al., 25 Aug 2025). The method then constructs two levels of weighting.

At the instance level, BTW uses KL divergence to quantify how much a unimodal expert differs from the current multimodal prediction. In classification, the weight is

$$w^{(m)}_{i} = D_{KL}\bigl(p_{uni_m}(\,\cdot\,\vert\,x^{(i)}) \;\|\; p_{multi}(\,\cdot\,\vert\,x^{(i)})\bigr).$$

A larger divergence is interpreted as indicating that the modality carries unique information not yet captured by the joint model, so it should be emphasized more on that example. In regression, each prediction is interpreted as a Gaussian distribution, with $\sigma^{(m)}_i = (y_i-\mu^{(m)}_i)^2$, and the closed-form KL between two Gaussians is used (Hou et al., 25 Aug 2025). After computing $\{w^{(m)}_i\}_{m=1}^M$, the weights are L1-normalized across modalities:

$$\widetilde w^{(m)}_{i} = \frac{w^{(m)}_{i}}{\sum_{j=1}^M w^{(j)}_{i}}.$$

These normalized values are the local, or instance-level, weights.

At the modality level, BTW computes a global reliability score for each modality using mutual information between unimodal and multimodal outputs over the full dataset:

$$\alpha_m = I(U_m;M) = \sum_{u,m}\;p_{U_m,M}(u,m)\;\log\frac{p_{U_m,M}(u,m)}{p_{U_m}(u)\,p_{M}(m)}.$$

Here $i$0 and $i$1 (Hou et al., 25 Aug 2025). In classification, these are discrete class labels; in regression, MI is estimated via k-nearest-neighbor entropies using scikit-learn’s mutual_info_regression.

The two levels are then combined into a bi-level weight:

$i$2

This weight is smoothed across epochs by an exponential-moving average:

$i$3

where $i$4 is increased when the held-out metric, reported as MAE or F1, improves and decreased otherwise (Hou et al., 25 Aug 2025). The full method is therefore bi-level in two senses: it mixes local novelty and global reliability, and it stabilizes weights temporally across training.

3. Training workflow and integration into MoE

The operational workflow of BTW begins with unimodal initialization. First, $i$5 separate unimodal experts are trained so that $i$6 is available for all $i$7 and $i$8. Second, a standard MoE is trained on all $i$9 modalities jointly, yielding $m$0 (Hou et al., 25 Aug 2025). These predictions provide the stored references from which local KL weights and global MI weights are computed.

During multimodal training, each minibatch performs an updated forward pass through unimodal paths and the joint MoE, recomputes raw KL values $m$1, normalizes them into $m$2, combines them with $m$3 if full BTW is used, applies exponential-moving-average smoothing, and then multiplies each modality’s embedding in the MoE fusion layer by the resulting $m$4 before the backward pass updates the MoE parameters (Hou et al., 25 Aug 2025). The paper also defines two ablation variants: BTW-local (KL), which uses only $m$5, and BTW-global (MI), which uses only $m$6.

Because both KL and MI are computed externally, BTW adds no extra weights to the base MoE (Hou et al., 25 Aug 2025). The method is therefore described as plug-and-play and non-parametric, but it is not architecture-agnostic in the strongest possible sense: it requires architectures that expose unimodal outputs, such as late-fusion MoE, and pure early-fusion schemes cannot be directly weighted.

The framework’s stated rationale for variance stabilization is also two-part. Instance-level KL is reported to capture per-example novelty and to stabilize variance in dense regions of the output distribution, such as neutral sentiment. Modality-level MI is reported to correct ambiguity in high-variance or minority-class regions by rebalancing weights toward globally reliable modalities (Hou et al., 25 Aug 2025). This suggests that BTW is not merely a weighting heuristic; it is intended as a decomposition of multimodal uncertainty into local disagreement and global trustworthiness.

4. Experimental evidence

BTW was evaluated on CMU-MOSI and CMU-MOSEI for continuous sentiment regression in $m$7, using text, audio, and video modalities, and on MIMIC-IV for four-class length-of-stay classification, using time-series vitals, clinical notes, chest X-ray, and ECG modalities (Hou et al., 25 Aug 2025). The reported baselines were MulT, MMIM, and FuseMoE. Regression metrics were MAE, Pearson correlation, Acc-7, Acc-5, Acc-2, and Weighted-F1. Classification metrics were overall Accuracy, Macro-F1, and Weighted-F1.

Benchmark	Base MoE / FuseMoE	BTW outcome
CMU-MOSI	MAE 0.735, Corr 0.770	BTW-local: 0.714 / 0.786; full BTW: 0.716 / 0.781, Acc-7 47.52%
CMU-MOSEI	MAE 0.570, Corr 0.723	BTW-local: 0.566 / 0.727; full BTW Acc-7 52.62%
MIMIC-IV (LOS)	Accuracy 41.33%, Weighted-F1 40.33%	BTW-local: 43.67% / 43.00%; full BTW: 45.67% / 45.00%

On CMU-MOSI, FuseMoE alone achieved MAE $m$8 and Corr $m$9. BTW-local reduced MAE to $p_{uni_m}(y \mid x^{(i)})$0 and raised Corr to $p_{uni_m}(y \mid x^{(i)})$1, reported as an approximately $p_{uni_m}(y \mid x^{(i)})$2 MAE drop. Full BTW achieved MAE $p_{uni_m}(y \mid x^{(i)})$3, Corr $p_{uni_m}(y \mid x^{(i)})$4, and the best multi-class Acc-7 of $p_{uni_m}(y \mid x^{(i)})$5 (Hou et al., 25 Aug 2025). On CMU-MOSEI, MoE alone obtained MAE $p_{uni_m}(y \mid x^{(i)})$6 and Corr $p_{uni_m}(y \mid x^{(i)})$7; BTW-local obtained MAE $p_{uni_m}(y \mid x^{(i)})$8 and Corr $p_{uni_m}(y \mid x^{(i)})$9; and full BTW obtained the best multi-class Acc-7 of $p_{multi}(y \mid x^{(i)})$0 with only a slight drop in binary F1. On MIMIC-IV, FuseMoE obtained Accuracy $p_{multi}(y \mid x^{(i)})$1 and Weighted-F1 $p_{multi}(y \mid x^{(i)})$2; BTW-local reached $p_{multi}(y \mid x^{(i)})$3 and $p_{multi}(y \mid x^{(i)})$4; and full BTW reached Accuracy $p_{multi}(y \mid x^{(i)})$5 and Weighted-F1 $p_{multi}(y \mid x^{(i)})$6, reported as an absolute $p_{multi}(y \mid x^{(i)})$7 improvement (Hou et al., 25 Aug 2025).

The ablation results indicate that constant per-modality KL and MI-only variants underperform the full bi-level scheme, and that removing the MI term hurts performance on minority classes in length-of-stay classification (Hou et al., 25 Aug 2025). The discussion further reports that class 0 LOS corrections rose from $p_{multi}(y \mid x^{(i)})$8, which was used to support the claim that MI helps in rare-class regions. The paper also reports computational overhead: on CMU-MOSI, baseline per-epoch time is 6 min 45 s; BTW-local adds approximately 10 min, and full BTW adds approximately 11 min total. On CMU-MOSEI, baseline time is 38 min and BTW is approximately 120 min, including all precomputations (Hou et al., 25 Aug 2025). The per-epoch forward and backward cost is nevertheless described as comparable to the base MoE.

5. Position within the broader literature on adaptive modality weighting

Within multimodal research, BTW is one member of a broader class of methods that move beyond fixed pairwise coefficients, but it is technically distinct from other recent approaches. In AVSR, "Dr. SHAP-AV" uses Shapley values to analyze modality contributions rather than to set training weights, and introduces Global SHAP, Generative SHAP, and Temporal Alignment SHAP as three analytical views over modality balance, decoding-time dynamics, and frame-to-token correspondence (Cappellazzo et al., 12 Mar 2026). The AVSR results show that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation, exposing a persistent audio bias. BTW addresses a different problem—training-time MoE weighting rather than post hoc modality attribution—but both frameworks target fine-grained modality assessment beyond a single global fusion coefficient.

In multimodal conversational emotion recognition, Ada2I combines Adaptive Feature Weighting (AFW) and Adaptive Modality Weighting (AMW) and introduces a refined disparity ratio to modulate gradient updates across text, audio, and visual modalities (Nguyen et al., 2024). In PET/CT tumor segmentation, H2ASeg combines Modality-Cooperative Spatial Attention (MCSA) and Target-Aware Modality Weighting (TAMW) to perform hierarchical interaction and tumor-aware feature gating (Lu et al., 2024). In multi-modal super-resolution, M$p_{multi}(y \mid x^{(i)})$9ESR introduces a dynamic MoE design with a spatially dynamic modality weighting module and a temporally adaptive modality temperature scheduling mechanism, together with a generalization-risk analysis based on the alignment between modality weights and effective contributions (Luo et al., 11 May 2026). These methods all advance beyond static two-modality weighting, but they differ materially from BTW in mechanism: Ada2I is feature- and gradient-centric, H2ASeg is spatial and target-aware, M$$w^{(m)}_{i} = D_{KL}\bigl(p_{uni_m}(\,\cdot\,\vert\,x^{(i)}) \;\|\; p_{multi}(\,\cdot\,\vert\,x^{(i)})\bigr).$$0ESR is patch-wise and diffusion-time-aware, and Dr. SHAP-AV is attribution-centric.

A useful contrast also arises with Continuous Weight Balancing, which addresses sample reweighting for skewed continuous traits using density-ratio weighting $$w^{(m)}_{i} = D_{KL}\bigl(p_{uni_m}(\,\cdot\,\vert\,x^{(i)}) \;\|\; p_{multi}(\,\cdot\,\vert\,x^{(i)})\bigr).$$1 and shows how continuous weighting strictly generalizes piecewise-constant binned weighting (Wu et al., 2021). BTW likewise generalizes beyond coarse weighting, but it does so over modalities rather than over samples indexed by a continuous trait. The shared theme is replacement of coarse, fixed weighting with adaptive weighting derived from distributional structure.

6. Limitations, failure modes, and disambiguation

The limitations of BTW are explicit. When a modality’s unimodal predictions are poorly calibrated or completely missing, KL and MI can both become noisy (Hou et al., 25 Aug 2025). Under missing modalities, BTW-local remains robust, whereas the global MI term can mislead weights if a modality is absent too often. The framework also assumes supervised labels, and extending it to self-supervised or contrastive settings is identified as future work. Architecturally, BTW requires access to unimodal outputs and therefore does not directly apply to pure early-fusion schemes.

These caveats qualify a possible misconception that “non-parametric” implies universally inexpensive or universally reliable. BTW adds no trainable parameters, but it does require a one-time overhead to compute unimodal predictions and additional per-epoch KL and MI updates (Hou et al., 25 Aug 2025). Its reported scalability derives from being linear in the number of modalities and avoiding combinatorial explosion, not from eliminating the cost of multimodal calibration and repeated divergence estimation.

The acronym BTW also requires disambiguation. In this context it denotes Beyond Two-modality Weighting, the multimodal weighting framework described above (Hou et al., 25 Aug 2025). The same acronym appears in unrelated contexts, including the Bak–Tang–Wiesenfeld sandpile model and its perturbations in statistical physics, as in "Continuous Move From BTW to Manna Model" (Najafi et al., 2012). The similarity in acronym does not indicate conceptual overlap. A related but distinct source of confusion is terminological: Continuous Weight Balancing generalizes binned sample weighting over a continuous trait (Wu et al., 2021), whereas Beyond Two-modality Weighting generalizes modality weighting in multimodal MoE training (Hou et al., 25 Aug 2025).

In summary, BTW is defined by four properties that distinguish it within multimodal weighting research: it is bi-level, because it combines instance-level KL and modality-level MI; non-parametric, because it introduces no additional learnable parameters; multimodality-scalable, because it is formulated for arbitrary $$w^{(m)}_{i} = D_{KL}\bigl(p_{uni_m}(\,\cdot\,\vert\,x^{(i)}) \;\|\; p_{multi}(\,\cdot\,\vert\,x^{(i)})\bigr).$$2; and variance-stabilizing, because it is designed to balance local novelty against global reliability during multimodal training (Hou et al., 25 Aug 2025).