Before Fusion, Ask What to Keep: Contextual Calibration of Multimodal Signals

Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed. A modality that is useful for one input may become distracting for another, and local feature responses within the same modality can disagree with evidence from other sources. This work investigates how to adjust multimodal representations before they are merged by a downstream predictor. We develop a compact calibration module that compares each modality with the others at the summary level, extracts cues of cross-source support and conflict, and converts these cues into instance-wise and dimension-wise modulation signals. The calibration is applied to the original modality features rather than to already fused representations, enabling the model to suppress misleading components, preserve weak but useful evidence, and emphasize responses that are better supported by the current multimodal context. The module is designed as a plug-in component and can be attached to different fusion backbones without changing their prediction heads. Across five benchmarks covering sentiment understanding, action recognition, audio-visual event detection, and audio-visual emotion classification, the proposed pre-combination calibration strategy improves performance under both sequence-based and convolutional fusion settings. Additional analyses under modality removal, synthetic corruption, training dynamics, and feature-level visualization show that calibrating signals before fusion can reduce interference from unreliable modalities and produce more stable multimodal optimization.

• The paper introduces VGMR, a pre-fusion module that uses cross-modal agreement, discrepancy, and global or channel-level value signals to selectively refine modality features.
• VGMR reaches 0.8446 accuracy and 0.8029 Macro-F1 on MOSEI with a Transformer, outperforming parameter-matched fusion baselines and complementing methods such as OGM-GE and ARL.
• The paper shows that corrupted inputs can receive higher value estimates but lower gate responses, demonstrating that value reflects contextual impact rather than whether a feature should be preserved.

Motivation and problem statement

Multimodal learning does not reliably benefit from adding modalities. Prior work on modality imbalance has shown that dominant modalities can suppress weaker ones during joint optimization, and that weak or noisy modalities can inject harmful components into the fused representation. The dominant line of remedies—Grad-Blending, OGM-GE, PMR, AGM, MMPareto, D{content}R, MLA, and ARL—intervenes at the optimization level, adjusting losses, gradients, or learning dynamics after modality features have entered joint training. The paper argues this leaves an earlier question unanswered: before fusion, how should a model decide which components of each modality to enhance, retain, or suppress given the current cross-modal context?

A preliminary experiment on MOSEI supports the premise: a simple Concat+MLP pre-fusion transformation (0.8298 accuracy) already matches or exceeds MMPareto (0.8279), suggesting that lightweight pre-fusion refinement can partially mitigate interference before any optimization-level balancing is applied.

The paper's central conceptual move is to distinguish value from relevance, salience, confidence, or importance. Value is defined as context-conditioned impact evidence: a high-value component may strongly affect prediction under the current context, but that says nothing about whether it should be preserved—a confident but conflicting modality can be harmful. The final gate, not the value signal itself, learns the modulation direction through end-to-end task supervision. This design is loosely motivated by Partial Information Decomposition (redundant/unique/synergistic contributions), though the method does not estimate PID quantities explicitly.

Method: Value-Gated Modality Refiner

VGMR is a plug-in module applied to raw modality features $\mathbf{X}_m$ before they enter any fusion backbone. Its pipeline proceeds in five stages:

1. Projection and summary construction. Each modality is projected into a shared latent space; a summary vector combines average and max pooling via a modality-specific MLP. Because interaction occurs at summary level, no strict temporal alignment across modalities is required.
2. Cross-modal interaction modeling. For each target modality, VGMR aggregates non-target summaries into a context vector and computes pairwise agreement terms (element-wise products) and discrepancy terms (absolute differences) against every other modality.
3. Value estimation. A hidden value representation derived from the summary and interaction features yields two signals: a scalar global value $g_m$ (sample-level impact) and a channel-level value vector $\mathbf{p}_m$ (dimension-wise impact). Both are sigmoid-bounded and neither acts as a retention mask.
4. Fine-grained gate generation. The centered value signals are combined with raw-feature evidence and latent-space evidence through shape-aligned additive mappings, producing a gate with the same shape as the original feature—enabling time-step- and dimension-level modulation.
5. Amplification and residual retention. The sigmoid gate is rescaled around 1 by amplification coefficient $\alpha$ (0.5 by default) and blended toward identity by retention coefficient $\beta$ (0.2 by default), preventing over-suppression of weak but useful cues.

Training uses the task loss plus a value-alignment term: empirical modality contribution is estimated as the loss increase when a modality is removed ($\Delta_m$), and both $g_m$ and the mean channel value are aligned to it, preventing value scores from becoming arbitrary gate-internal activations. A regularization term guards against value collapse.

Main results

Across five benchmarks (MOSI, MOSEI, UCF101, AVE, CREMA-D) under both ResNet+Concat and Transformer backbones, VGMR achieves the best accuracy in most settings. Under the Transformer backbone it reaches 0.8446 accuracy / 0.8029 Macro-F1 on MOSEI, versus 0.8255/0.7810 for the plain baseline. On AVE and CREMA-D, ARL attains slightly higher Macro-F1, but VGMR retains the best accuracy—an honest concession rather than a sweep.

Several controlled comparisons strengthen the claim that gains are not artifacts:

Comparison on MOSEI (Transformer)	Acc	Macro-F1
Summation	0.8219	0.7785
Tensor Fusion	0.8289	0.7871
Concat+MLP (parameter-matched)	0.8298	0.7803
Cross-Attention	0.8317	0.7941
Sigmoid+Tanh Gate (parameter-matched)	0.8326	0.7924
VGMR	0.8446	0.8029

Since Concat+MLP and Sigmoid+Tanh Gate were tuned to match VGMR's parameter count (~3.04M extra parameters), the improvement cannot be attributed to capacity alone. VGMR also composes with optimization-level balancing: plugging it into OGM-GE adds +1.01 accuracy points and +1.90 F1 points; into ARL, +0.75 and +0.14.

Diagnostic analyses

Component ablation on MOSEI shows all parts contribute: removing both value signals drops accuracy to 0.8287; removing both agreement and discrepancy terms drops it to 0.8257. Notably, removing the discrepancy term hurts more than removing agreement, indicating conflict evidence is especially important for identifying unreliable responses. Channel-level value alone outperforms global value alone, showing sample-level scalars are too coarse.

Value-gate behavior under corruption is the paper's most instructive result. When 30% of text timesteps on MOSI are replaced with Gaussian noise, global value rises from 0.598 to 0.788 and channel value from 0.554 to 0.707, while the average gate response falls from 0.481 to 0.417. This directly demonstrates the paper's core claim: higher value means stronger contextual impact, not stronger preservation—the gate uses value evidence to suppress corrupted high-impact responses.

Modality removal reveals a contrast with balancing methods. Removing text causes large drops for all methods (text dominates). But several baselines actually improve when audio or vision is removed (e.g., Grad-Blending +0.0006 without vision), confirming those weak modalities inject noise under direct fusion. VGMR instead degrades when either is removed (-0.0036 without audio, -0.0075 without vision), consistent with selective retention of task-relevant weak-modality content rather than blanket reweighting.

Robustness and stability. Under varying Gaussian noise ratios and intensities on MOSI, VGMR maintains higher accuracy than plain and Grad-Blending Transformers, with the gap widening under heavier corruption. Gradient-direction diagnostics (cosine between audio-only and audio-video gradients on shared parameters) show the plain Transformer oscillating between positive and negative alignment, whereas VGMR yields a smoother curve near zero—suggesting reduced cross-modal dominance and directional conflict. Qualitatively, VGMR applies stronger suppression to audio and visual responses than to text, both per-sample and globally on MOSEI.

Limitations and open questions

The paper is candid about costs and scope. VGMR raises parameters from 0.72M to 3.04M and inference latency from 1.17ms to 3.87ms per batch over the vanilla Transformer—acceptable but non-negligible, and training time (10.01s/epoch) exceeds lightweight alternatives even if it stays below Grad-Blending and AGM. Scalability to larger backbones, longer sequences, and higher-dimensional features remains untested, as do mitigations such as low-rank projections or sparse gating.

Methodologically, the pooled-summary design avoids alignment requirements but cannot detect localized conflicts such as occlusions, frame drops, or short audio-text mismatches; window-level or time-resolved value estimation is left open. Although PID motivates the design, the connection between learned value signals and redundancy/unique/synergistic information is not quantified. Robustness evaluation covers only Gaussian feature corruption and controlled text corruption—missing modalities, temporal masking, sensor dropout, asynchrony, and distribution shift remain unexamined. Finally, CREMA-D results use fixed lightweight features and a shallow head, so absolute numbers reflect the controlled protocol rather than state-of-the-art performance on that dataset.

Conclusion

This paper reframes modality imbalance as a pre-fusion feature-refinement problem and proposes VGMR, which estimates global- and channel-level value evidence from cross-modal agreement and discrepancy, then conditions fine-grained gates that refine original modality features before fusion. The key empirical finding—that corrupted inputs receive higher value estimates but lower gate responses—validates the separation between impact evidence and modulation direction. With consistent gains across five benchmarks, complementarity with optimization-level balancers, and improved noise robustness at moderate overhead, VGMR offers a well-supported addition to the multimodal toolkit, contingent on future validation at larger scale and under more diverse corruption conditions.