MVCL-DAF++: Enhanced Multimodal Intent Recognition

• MVCL-DAF++ is a multimodal intent recognition framework that enhances semantic grounding and robustness using class-level prototypes.
• It employs hierarchical cross-modal fusion, combining coarse global summaries with fine token-level dynamics via dynamic attention fusion.
• Empirical results on MIntRec benchmarks demonstrate state-of-the-art improvements, including a +4.18% WF1 gain on rare and open-intent conditions.

Searching arXiv for the specified paper and the prior MVCL-DAF framework. MVCL-DAF++ is a multimodal intent recognition (MMIR) framework that extends MVCL-DAF with two additional modules: Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion. It is proposed to address weak semantic grounding and poor robustness under noisy or rare-class conditions in MMIR. On MIntRec and MIntRec2.0, it reports new state-of-the-art results, including improvements in rare-class recognition of $+1.05\%$ and $+4.18\%$ WF1, respectively (Huang et al., 22 Sep 2025).

1. Problem setting and design objectives

The motivating problem is MMIR under conditions where semantic representations are weakly grounded across modalities and become unreliable when one modality is missing or corrupted. The stated failure mode is that instance-level encoders often fail to capture shared class semantics across modalities, leading to noisy or ambiguous representations. The framework is also designed for settings in which imbalanced classes and noisy inputs, such as blurred video or poor audio, degrade performance, especially on long-tailed or open-intent scenarios.

MVCL-DAF++ is defined by three explicit objectives. The first is to introduce explicit semantic grounding through class-level prototypes to regularize the embedding space; this is termed Prototype-Aware Contrastive Alignment. The second is to improve hierarchical cross-modal fusion by combining global modality summaries with token-level features; this is termed Coarse-to-Fine Dynamic Attention Fusion. The third is to achieve robustness to noise and rare-class samples on the benchmarks MIntRec and MIntRec2.0.

These objectives position the model as a semantically regularized extension of multimodal contrastive learning. A plausible implication is that the framework is intended not merely to aggregate modalities, but to constrain the geometry of the learned space at both the class level and the token level.

2. System composition and processing pipeline

MVCL-DAF++ extends the prior MVCL-DAF framework with two principal modules: Prototype-Aware Contrastive Alignment (PCA), which anchors instance embeddings to learned class prototypes, and Coarse-to-Fine Dynamic Attention Fusion (DAF), which integrates coarse global features and fine token features via dynamic cross-modal attention.

Its workflow is specified as a sequence of six stages. First, text, visual, and acoustic inputs are separately encoded into token sequences. The text branches are masked text $E_{tm}$ and labeled text $E_{tl}$ via BERT. The visual branch $E_v$ is produced via a CNN or vision transformer. The acoustic branch $E_a$ is produced via BiPeephole LSTM + CTC Align. Second, the model performs multi-view representation learning with token-level contrastive losses. Third, it applies coarse feature extraction with a modality-aware Transformer to produce global summaries $M_c$. Fourth, Coarse-to-Fine Dynamic Attention Fusion yields fine features $M_f$ and fused classification features $M_{cf}$. Fifth, Prototype-Aware Contrastive Alignment enforces instance–prototype correspondence. Sixth, a classifier is applied on $M_{cf}$ with cross-entropy.

This decomposition separates encoding, fusion, alignment, and classification into distinct functional stages. The architecture therefore combines instance-level representation learning with explicit class-level regularization and hierarchical multimodal interaction.

3. Prototype-Aware Contrastive Alignment

Prototype-Aware Contrastive Alignment introduces class-level prototypes into the contrastive learning objective. For each class $+4.18\%$0, where $+4.18\%$1, the model maintains a prototype vector $+4.18\%$2, denoted $+4.18\%$3 in the paper. At each mini-batch, the prototype is computed as the mean of the L2-normalized instance embeddings in that class:

$+4.18\%$4

where $+4.18\%$5 and $+4.18\%$6 is the L2-normalized instance embedding. The prototype is then normalized so that $+4.18\%$7 to stabilize training.

For each instance $+4.18\%$8 with true label $+4.18\%$9, the prototype-aware InfoNCE loss is

$E_{tm}$0

with $E_{tm}$1 and temperature $E_{tm}$2.

The prototype update policy has two stated variants. In the basic setting, prototypes are recomputed per mini-batch. Optionally, an exponential moving average can be applied,

$E_{tm}$3

with $E_{tm}$4–$E_{tm}$5, followed by re-normalization. The paper uses per-batch aggregation and L2-normalization.

The intended role of this module is explicit semantic grounding through class-level prototypes. Within the stated design, prototypes regularize the embedding space by encouraging instance embeddings to align with their class-level anchors rather than relying solely on pairwise instance relations.

4. Coarse-to-Fine Dynamic Attention Fusion

The DAF component is organized around global modality summaries and token-level features. For each modality $E_{tm}$6, corresponding to text, vision, and audio, a global summary vector $E_{tm}$7 is extracted via a shallow modality-aware Transformer encoder:

$E_{tm}$8

These global summaries may be obtained, for example, by mean-pool or a $E_{tm}$9 token. In parallel, the modality encoders produce token embeddings $E_{tl}$0, $E_{tl}$1, and $E_{tl}$2.

Cross-modal interaction is defined through attention scores that use global summaries to query token-level features. An example is visual-to-text attention at token $E_{tl}$3:

$E_{tl}$4

where $E_{tl}$5 are projection matrices.

The fused representation is then constructed in stages. A coarse attention operation produces weighted global features such as

$E_{tl}$6

Fine attention similarly computes token-level cross-modal signals. The combination stage is defined as

$E_{tl}$7

The model uses $E_{tl}$8 in the token-level contrastive loss, while $E_{tl}$9 is concatenated or summed and fed to the classifier.

This construction makes the fusion explicitly hierarchical. Global modality summaries inform attention over token-level representations, so the fusion mechanism is not limited to local alignment alone. The stated purpose is hierarchical cross-modal interaction.

5. Objective function and optimization regime

Training combines classification, instance-level contrastive learning, and prototype-aware contrastive learning. The classification loss is

$E_v$0

The prototype-aware contrastive loss is

$E_v$1

The multi-view instance-level contrastive objective uses InfoNCE between the labeled text anchor $E_v$2 and the positive views $E_v$3:

$E_v$4

where

$E_v$5

The overall objective is

$E_v$6

with $E_v$7.

The optimization regime is specified as follows: AdamW optimizer, learning rate $E_v$8, weight decay $E_v$9, batch size $E_a$0, temperature $E_a$1, and up to $E_a$2 epochs with early stopping at patience $E_a$3. Experiments are conducted on a single NVIDIA A100-40GB GPU, and results are averaged over $E_a$4 random seeds for reporting. The source code is available at https://github.com/chr1s623/MVCL-DAF-PlusPlus.

A common simplification would be to view the prototype term as a replacement for instance-level contrastive learning. The stated objective does not do so: it retains both $E_a$5 and $E_a$6, indicating that prototype-level alignment and instance-level alignment are treated as complementary rather than interchangeable.

6. Benchmarks, results, and ablation evidence

The empirical evaluation uses two datasets. MIntRec 1.0 contains $E_a$7 samples, $E_a$8 intent classes, and aligned text+video+audio. MIntRec 2.0 contains $E_a$9 samples and $M_c$0 classes, with a long-tailed distribution and open intents. The reported metrics are ACC, WF1, WP, and R (Huang et al., 22 Sep 2025).

The main WF1 results averaged over $M_c$1 seeds are:

Method	MIntRec WF1	MIntRec2.0 WF1
MVCL-DAF	74.61%	55.05%
MVCL-DAF++	75.66%	59.23%

These correspond to gains of $M_c$2 on MIntRec and $M_c$3 on MIntRec2.0. The same improvements are also reported as rare-class and noisy-condition gains: $M_c$4 WF1 on MIntRec rare classes and $M_c$5 WF1 on MIntRec2.0 rare/open intents.

The ablation studies isolate the contribution of the two principal modules and of the loss composition. Removing Prototype-Aware Alignment produces a WF1 drop of approximately $M_c$6 on MIntRec. Removing DAF fusion produces a WF1 drop of approximately $M_c$7. In the loss-only ablation, the classifier-only setting yields WF1 values of $M_c$8 on MIntRec / MIntRec2.0; adding contrastive learning yields $M_c$9; adding prototype learning yields $M_f$0; and combining all components yields $M_f$1, which is the best result.

These findings support the paper’s stated interpretation that prototype-guided learning improves semantic consistency and robustness under noise and imbalance, while hierarchical fusion yields richer cross-modal representations. The empirical pattern also suggests that the largest reported benchmark gain is on the long-tailed and open-intent setting of MIntRec2.0, which is consistent with the model’s stated emphasis on robustness under rare-class conditions.