Primary-Modality Cross-Attention (PCCA)

• Primary-Modality-Centric Cross-Attention (PCCA) is a mechanism that dynamically selects the dominant modality to drive effective multimodal fusion in sentiment analysis.
• It employs a graph-based dynamic sequence compressor with capsule networks to align and reduce noise in acoustic and visual streams.
• PCCA improves model accuracy by centering attention on key primary modality cues while integrating complementary information from secondary modalities.

Primary-modality-Centric Cross-Attention (PCCA) is a mechanism developed for effective multimodal fusion in sentiment analysis systems. It is designed to address the challenges in balancing contributions from language, acoustic, and visual modalities by centering cross-attention on the dynamically selected dominant modality, while controlling noise and redundancy especially prevalent in non-language streams. PCCA is an integral module within the modality optimization and dynamic primary modality selection framework (MODS), improving representational fidelity and model accuracy by enhancing primary modality features and facilitating structured cross-modal context exchange (Yang et al., 9 Nov 2025).

1. Motivation and Problem Formulation

Multimodal Sentiment Analysis (MSA) seeks to predict sentiment labels based on joint language, acoustic, and visual inputs, e.g., transcribed utterance text $L$, acoustic feature sequence $A$, and visual feature sequence $V$. Empirical evidence in large-scale benchmarks demonstrates that unimodal scores are often imbalanced, leading to suboptimal fused feature representations. Conventional fusion mechanisms typically designate a fixed primary modality, leveraging dominant language cues but neglecting context when acoustic or visual information becomes transiently dominant. These approaches fail to exploit dynamic modality importance and suffer performance degradation under variable noise and sequential redundancy, especially when non-language modalities serve as primary streams.

PCCA is constructed to resolve these deficits through sample-adaptive primary modality selection, followed by an attention mechanism that enhances and propagates primary modality features while enabling rich cross-modal interaction.

2. Architectural Principles and Methodology

PCCA operates within the broader MODS paradigm, following three critical stages:

1. Modality Sequence Compression: A Graph-based Dynamic Sequence Compressor (GDC) processes acoustic and visual sequences $H_m \in \mathbb{R}^{T_m \times d_m}$, compressing them to $T_l$ steps (the length of language modality) and suppressing redundant time-steps through capsule networks and GCN layers.
2. Dynamic Primary Modality Selection: The sample-adaptive Primary Modality Selector (MSelector) identifies the primary modality for each input, based on dynamic dominance estimation, typically yielding one modality $M^*$ as the primary.
3. Primary-modality-Centric Cross-Attention: PCCA constructs cross-attention maps in which the selected dominant modality $M^*$ serves as the query for all other modalities, ensuring attention weights focus on $M^*$, but facilitating information flow from secondary modalities.

Formally, let $H_{M^*} \in \mathbb{R}^{T_l \times d}$ be the compressed primary modality sequence after GDC, and $\{H_m\}_{m\neq M^*}$ be secondary modalities (all compressed and aligned to $T_l$). PCCA computes:

$$\mathrm{Attn}_{PCCA} = \mathrm{softmax}\left(\frac{Q_{M^*} K_{m}^T}{\sqrt{d}}\right) V_{m}$$

where $Q_{M^*}, K_m, V_m$ denote query, key, and value projections from $H_{M^*}$ and $H_m$ respectively.

This centric paradigm produces fused representations $F_{M^*}$ that maximize retention of primary cues while adaptively supplementing with context from secondary modalities.

3. Technical Workflow and Hyper-parameters

The implementation proceeds as:

1. Compress all non-language modalities using GDC:
   • Capsule dimension $d \in \{64,128\}$.
   • Routing iterations $R=3$.
   • GCN layers $L=2$.
   • All outputs are aligned to $T_l$ (language sequence length).
2. Select the dominant modality using MSelector: Based on sample-level activation and alignment, select $M^*$.
3. Cross-attention centric to $M^*$: For each secondary modality $m \neq M^*$, compute cross-attention from $H_{M^*}$ to $H_m$; concatenate resulting fused vectors.
4. Final fusion: Concatenate $F_{M^*}$ (post-attention enhanced) with unimodal outputs for downstream sentiment regression/classification.

Ablation results show that removing GDC or capsule-based compression consistently yields a 2–4% drop in sentiment analysis accuracy on benchmark datasets, emphasizing the necessity of redundancy compression before centric fusion.

4. Compression and Alignment Rationale

Non-language modalities (acoustic, visual) exhibit variable lengths $T_m \gg T_l$ and substantial redundancy. GDC compresses $T_m$ to $T_l$ via dynamic routing, producing capsules $N_m^j$ where noisy or redundant steps contribute negligible routing coefficients. The resulting $E_m$ (edge weights) and $D_m$ (degree matrix) encode intra-modality affinities, and GCN neighborhood aggregation smooths fluctuations prior to attention-based fusion.

By aligning all modalities temporally to the language modality, PCCA ensures cross-attention operates on semantically consistent sequences and prevents dominant modalities from being overwhelmed by irrelevant secondary information. All modules are trainable end-to-end via global regression loss $\mathcal{L}_{\mathrm{reg}}$ and InfoNCE contrastive loss $\mathcal{L}_{\mathrm{NCE}}$.

5. Impact and Empirical Performance

PCCA within MODS achieves state-of-the-art accuracy across four multimodal sentiment analysis benchmarks, outperforming conventional fixed-primary fusion approaches. Information-dense compression by GDC, combined with dynamic modality centricity, yields improved sentiment label correlation and reduced error, as measured by standard accuracy and regression metrics. Empirically, removal of PCCA or its dynamic selection in the MODS pipeline produces measurable drops in accuracy, confirming its effectiveness in balancing modality contributions and mitigating sequential noise (Yang et al., 9 Nov 2025).

6. Related Work and Comparative Analysis

PCCA contrasts with prior fixed primary modality approaches, which statically maximize language modality advantages regardless of variable modality dominance. Earlier methods fail under sample-wise modality shifts and are vulnerable to redundant sequence length in acoustic/visual inputs. By integrating GDC's redundancy suppression and primary-centric fusion, MODS with PCCA achieves enhanced representational quality and more robust performance. Ablation with GDC removed yields 3–4% accuracy deterioration, underscoring the benefits of sequential compression and dynamic attention mechanisms.

7. Future Directions and Limitations

The adaptive nature of PCCA opens avenues for sample-level diagnosis of cross-modal dominance, facilitating architectures that react to intra-video shifts or external perturbations. A plausible implication is that further expansion to more modalities (e.g., physiological signals) could inherit benefits from primary modality centric fusion, pending sequence alignment and redundancy control. However, the performance gain relies on precise compression and temporally aligned cross-modal sequences; suboptimal GDC configuration or insufficient routing iterations may impede centric attention efficacy.

Modality imbalance and sequential noise remain central challenges, and future research may explore more sophisticated centroid selection strategies or hierarchical attention profiles within the PCCA paradigm.