Mitigating Multimodal Inconsistency via Cognitive Dual-Pathway Reasoning for Intent Recognition

Abstract: Multimodal Intent Recognition (MIR) aims to understand complex user intentions by leveraging text, video, and audio signals. However, existing approaches face two key challenges: (1) overlooking intricate cross-modal interactions for distinguishing consistent and inconsistent cues, and (2) ineffectively modeling multimodal conflicts, leading to semantic cancellation. To address these, we propose a novel Cognitive Dual-Pathway Reasoning (CDPR) framework, which constructs a stable semantic foundation via the intuition pathway and mitigates high-level semantic conflicts through the reasoning pathway, cooperatively establishing deep semantic relations. Specifically, we first employ a representation disentanglement strategy to extract modality-invariant and specific features. Subsequently, the intuition pathway aggregates cross-modal consensus using shared features for solid global representations. The reasoning pathway introduces an inconsistency perception mechanism, combining semantic prototype matching with statistical probability calibration to precisely quantify conflict severity, and dynamically adjusting the weights between both pathways. Furthermore, a multi-view loss function is adopted to alleviate modality laziness and learn structured features at different stages. Extensive experiments on two benchmarks show that CDPR achieves SOTA performance and superior robustness in mitigating multimodal inconsistency. The code is available at https://github.com/Hebust-NLP/CDPR.

• The paper proposes the CDPR framework, integrating shared and private modal pathways to rapidly aggregate consensus and resolve semantic conflicts.
• It employs adaptive gating based on semantic conflict energy and statistical divergence to selectively suppress unreliable modalities under noisy conditions.
• Experimental evaluations demonstrate that CDPR outperforms existing baselines in accuracy, precision, and computational efficiency across complex scenarios.

Cognitive Dual-Pathway Reasoning for Multimodal Intent Recognition

Background and Motivation

Multimodal Intent Recognition (MIR) is central for HCI systems to capture complex user intent by leveraging heterogeneous signals from text, audio, and video. MIR facilitates refined inference for tasks in multimedia retrieval, intelligent agents, autonomous driving, and medical diagnosis. The field has progressed from early adaptation of multimodal sentiment analysis baselines to increasingly sophisticated modeling of intent through specialized datasets such as MIntRec and MIntRec2.0, which expose fine-grained intent categories and highlight the challenge of multimodal semantic inconsistency.

Two key limitations persist across prior approaches:

1. Lack of adaptive reasoning mechanisms which distinguish consistent from inconsistent cues and dynamically regulate inference.
2. Ineffective modeling of semantic conflicts (modal discrepancies), leading to incorrect fusion and semantic cancellation.

CDPR Framework Architecture

The proposed Cognitive Dual-Pathway Reasoning (CDPR) framework addresses these deficiencies by integrating a dual-pathway architecture inspired by cognitive science:

• Feature Disentanglement: Modal features are partitioned via MLPs into shared modality-invariant subspaces and private modality-specific subspaces.
• Intuition Pathway: Operates on shared features to rapidly aggregate cross-modal consensus. Non-linear fusion with learnable scaling preserves contextual stability while facilitating synergy enhancement.
• Reasoning Pathway: Models semantic inconsistency using private features, quantifies conflict severity via a learnable semantic conflict prototype vector, and calibrates reliability with statistical measures (Jensen-Shannon Divergence and entropy).
• Adaptive Pathway Gating: A global gating factor, derived from semantic conflict energy and statistical divergence, modulates the weighting between intuition and reasoning pathways per sample.
• Multi-View Loss Supervision: Enforces discriminability and prevents modality laziness using cross-entropy on final output, reasoning path output, and each unimodal prediction. Orthogonality and moment discrepancy losses regularize private and shared feature spaces.

Inconsistency Perception and Reliability Calibration

CDPR operationalizes explicit inconsistency perception:

• Semantic conflict is detected by projecting difference vectors (modal deviations from centroid) onto learnable prototypes, filtering out noise and static bias.
• Reliability of each modality is evaluated at the decision level; modalities with high uncertainty or high conflict are adaptively suppressed.
• The joint effect of semantic conflict energy and statistical modulation governs the gating factor, ensuring robust handling of samples exhibiting high multimodal divergence.

Experimental Evaluation and Results

CDPR is systematically evaluated on MIntRec and MIntRec2.0:

• On MIntRec, CDPR achieves top accuracy (75.15%) and weighted precision (75.37%), surpassing the strongest baseline by 1.44% and 1.80%. Gains in F1, precision, recall, and weighted F1 range from 0.71% to 1.53%.
• On the more complex MIntRec2.0, CDPR attains 60.82% accuracy and 53.86% F1, outperforming baselines by 2.17% and 1.59%. Robust gains extend to weighted scores, supporting the assertion that CDPR generalizes effectively to datasets with intricate conflict structures.
• Feature disentanglement and loss function ablations affirm the necessity of shared/private subspaces, pathway separation, orthogonality, and multi-view supervision.

CDPR demonstrates notable prowess on 'hard' intent categories distinguished by multimodal conflict (e.g., Taunt, Oppose). F1 boosts of 6.75% and 7.79% over baselines manifest the effectiveness of deep semantic conflict modeling. However, performance remains below human-level classifiers—indicating ongoing challenges in nuanced multimodal alignment.

Robustness and Efficiency Analyses

CDPR's robustness is confirmed against synthetic Gaussian text noise:

• At $\sigma = 0.3$, the F1 (55.65%) persists well above baselines, and at extreme levels ($\sigma = 0.7$), CDPR maintains 22.68% F1 when competitors drop below 12%.
• This implies that existing models are vulnerable to text degradation, while CDPR dynamically shifts reliance to more reliable modalities, ensuring semantic stability under noisy or adversarial conditions.

In computational evaluations:

• CDPR requires approximately 48% fewer parameters and less than half the GPU memory compared to MVCL-DAF and MIntOOD, yet achieves 123% higher inference throughput (75.18 samples/s).
• This efficiency validates the practical deployability of dual-pathway mechanisms without unnecessary parameter inflation.

Feature Distribution Visualization

t-SNE visualizations on MIntRec reveal that CDPR's shared subspaces are tightly aligned across modalities, while private subspaces exhibit distinct clusters. This substantiates the orthogonal regularization's efficacy and confirms the dual-pathway approach's ability to decouple consensus and conflict signals.

Implications and Future Directions

CDPR delivers coherent advances for MIR:

• The architecture's adaptive reasoning mirrors human cognitive processes, demonstrating that integrating modality-invariant rapid consensus with modality-specific deep conflict resolution is essential for robust MIR.
• The explicit modeling of semantic conflict and reliability, tied to learnable prototypes and statistical divergence, marks a shift from indiscriminate fusion toward sample-specific decision-making.

Practical deployments will benefit from the framework's hardware efficiency and noise robustness, especially in interactive intent recognition, autonomous agents, and multi-modal dialogue systems. The gap to human-level intent inference—particularly in subtle conflict-laden categories—suggests ongoing need for richer semantic representations and more sophisticated cross-modal alignment. Further work could explore hierarchical pathway architectures, leveraging LLMs for semantic relational reasoning or incorporating causal intervention for deeper conflict analysis.

Conclusion

The Cognitive Dual-Pathway Reasoning (CDPR) framework introduces a fundamentally adaptive MIR paradigm, combining consensus-driven intuition and conflict-aware reasoning. Extensive quantitative and qualitative analyses demonstrate SOTA accuracy, robustness to input degradation, and computational efficiency. The framework's mechanisms for semantic conflict quantification, reliability gating, and representation disentanglement offer meaningful progress toward reliable, scalable, and nuanced multimodal intent understanding (2605.09468).