Integrity-weighted Cross-modal Completion

• Integrity-weighted Cross-modal Completion is a dynamic fusion mechanism that prioritizes modalities based on input integrity and reconstruction quality.
• It employs adaptive cross-modal attention to integrate language, acoustic, and visual cues, avoiding static concatenation methods.
• Empirical evaluations show that the method significantly improves robustness and sentiment prediction accuracy even under severe modality missingness.

Integrity-guided Adaptive Fusion (IF) is a dynamic multimodal fusion mechanism that directs attention and modality selection during prediction based on learned measures of input completeness ("integrity") and reconstruction quality. Developed as a core component within the Senti-iFusion framework for robust Multimodal Sentiment Analysis (MSA) under conditions of inter- and intra-modality missingness, IF addresses real-world scenarios where language, acoustic, or visual inputs may be partially observed or corrupted. IF eschews simple concatenation or fixed-weight averaging, instead leveraging modality integrity and adaptive cross-modal attention to maximize both the reliability and informativeness of fused predictions (Li et al., 21 Nov 2025).

1. Theoretical Foundation and Design Objectives

A primary objective of Integrity-guided Adaptive Fusion is to improve MSA robustness by dynamically prioritizing modalities that are empirically most "complete" and semantically well-recovered at fusion time. The mechanism fulfills three guiding principles:

1. Dynamic Modality Prioritization: At each prediction step, select the dominant modality whose observed tokens are least masked or corrupted and whose reconstructed embedding most closely aligns with ground-truth features.
2. Fallback to Auxiliary Modalities: When the dominant modality's integrity or feature quality is suboptimal, IF enables the model to rely more heavily on semantic cues available in auxiliary modalities.
3. Cross-modal Attention Mechanism: Rather than relying on concatenation or static weighting, IF applies cross-modal attention, letting data-driven resemblance between query and key representations determine fusion weights.

Completeness is operationalized via a learned integrity score $\hat{I}_m \in [0,1]$ for each modality $m$, predicting token unmasking at inference. Quality is enforced via dual reconstruction losses at the semantic and feature levels.

2. Mathematical Formulation

2.1 Integrity Scoring and Loss

For each modality $m$, the integrity estimation head (2-layer Transformer + linear projection) computes

$$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$

where $\hat{u}_m$ are incomplete embeddings and $X_{ie}$ is a learned [IE] token.

The module is supervised by

$$\mathcal{L}_{ie} = \frac{1}{N}\sum_{k=1}^{N} \|\hat{I}_m^k - I_m^k\|_2^2,$$

with $I_m^k = 1 - \text{(fraction of masked tokens)}$.

2.2 Reconstruction Quality Regularization

No explicit scalar quality score $q_m$ is defined; instead, dual-depth completion is enforced via:

• Feature-level MSE: $$\mathcal{L}_{mse}^g = \frac{1}{N}\sum_m \|\tilde{u}_m - u_m\|^2$$
• Semantic-level MSE: $m$0
• Feature/semantic-level MI losses:

$m$1

$m$2

These drive shared and completed features toward maximally reconstructing both fine-grained and semantically structured cues required for sentiment prediction.

2.3 Dominant Modality Selection

Batchwise mean integrity guides selection:

$m$3

The dominant modality is $m$4, with remaining modalities as auxiliaries.

2.4 Adaptive Cross-Modal Attention for Fusion

Dominant features $m$5 are processed by two Transformer layers, yielding $m$6. For three fusion steps:

• Queries $m$7 from transformed dominant features
• Keys/values $m$8 from auxiliary modality surrogates $m$9
• Attention weights:

$m$0

• Fused representations updated as:

$m$1

Following fusion, special [CLS]-style tokens are prepended and the sequence is passed to a cross-modal Transformer and linear prediction head.

2.5 Two-Stage Progressive Training

• Stage 1: Minimize $m$2 (with frozen predictor)
• Stage 2: Optimize

$m$3

Only after the initial stage are all model parameters, including fusion and predictor, trained jointly.

3. Architectural Composition and Data Flow

The mechanistic workflow of IF within Senti-iFusion is as follows:

1. Multimodal inputs $m$4 undergo random inter- and intra-modality masking to produce $m$5.
2. Embedding encoders $m$6 generate modality-specific embeddings $m$7.
3. The integrity estimation module $m$8 outputs integrity scores $m$9.
4. The completion module:
   • Disentangles each $$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$0 into shared $$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$1 and private $$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$2 features.
   • Constructs surrogate $$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$3.
   • Decodes surrogates and re-encodes to apply dual-depth losses.
5. Integrity-guided Adaptive Fusion:
   • Aggregates $$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$4 scores over the batch and selects the dominant modality.
   • Processes this dominant modality through Transformer layers and applies attention with auxiliary modalities using learned keys/values.
   • Updates the intermediate fused representation for three fusion rounds.
6. The concatenated [CLS]-like tokens and representations are input to a cross-modal Transformer, with the resulting pooled token mapped to the final sentiment prediction.

A summary of module sequence and functions is given below:

Step	Module	Key Output / Action
1	Embedding Encoder ($$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$5)	Incomplete modality embeddings ($$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$6)
2	Integrity Estimation ($$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$7)	Integrity scores ($$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$8)
3	Completion/Disentanglement	Shared and private features; surrogates
4	Dual-depth Loss Application	Semantic and feature-level consistency
5	IF Module	Fusion via dominant selection + cross-attention
6	Prediction Head	Sentiment prediction ($$\hat{I}_m = \mathcal{E}^{ie}_m(\mathrm{Concat}(X_{ie}, \hat{u}_m))$$9)

4. Algorithmic Steps and Implementation

The pseudocode for IF is as follows:

$X_{ie}$8

Exact training configurations include input length $\hat{u}_m$0, hidden dimension $\hat{u}_m$1, batch size 64, AdamW optimizer with learning rate $\hat{u}_m$2 and weight decay $\hat{u}_m$3, cosine annealing, warm-up, and early stopping. Loss weights: $\hat{u}_m$4, $\hat{u}_m$5, $\hat{u}_m$6; decoder weights $\hat{u}_m$7, $\hat{u}_m$8, $\hat{u}_m$9, $X_{ie}$0 (Li et al., 21 Nov 2025).

5. Empirical Results and Ablation Findings

Empirical evaluations demonstrate that IF consistently enhances robustness under modality-missingness, particularly when compared to static fusion approaches. Under a $X_{ie}$1 drop rate on MOSI and MOSEI datasets:

• Omitting integrity-weighted surrogates ("w/o IIR") increases MAE from $X_{ie}$2 and lowers F1.
• Eliminating the integrity loss ("w/o $X_{ie}$3") reduces Acc-7 by $X_{ie}$43\% (MOSI).
• Removing dual-depth completion loss ("w/o $X_{ie}$5") decreases F1 by $X_{ie}$6.
• Bypassing the two-stage strategy degrades both MAE and accuracy.

Robustness analysis under extreme missingness (drop rate $X_{ie}$7) reveals that IF prevents the model from collapsing to majority-class prediction, preserving fine-grained prediction performance (lower MAE, higher ACC-5 and F1). The essentiality of integrity-guided fusion is established by these ablations; it is the component that enables reliable adaptation to modality reliability at test time, outperforming competitive baselines in all missing-modality settings (Li et al., 21 Nov 2025).

6. Implications and Significance

Integrity-guided Adaptive Fusion redefines modality fusion as an integrity- and quality-aware, sequence-level adaptive process. The mechanism’s dynamic selection and attention-driven fusion repeatedly prove essential for fine-grained sentiment prediction in the presence of substantial multimodal noise or missingness. A plausible implication is that related multimodal inference tasks (e.g., action recognition, emotion detection) may similarly benefit from incorporating both explicit integrity estimation and cross-modal completion with attention, rather than relying on static or imputed representations.

The modular two-stage training—first focusing on stable integrity/quality modeling, then end-to-end fusion—contributes to training stability and overall accuracy. As demonstrated, IF maintains granular predictive performance even as competing techniques degrade under increasing input uncertainty (Li et al., 21 Nov 2025).