MVCL-DAF++: Advanced MMIR Framework

• The framework's key contribution is integrating prototype-aware contrastive alignment to improve intra-class compactness and stability in multimodal intent recognition tasks under noisy conditions.
• Its coarse-to-fine dynamic attention fusion effectively captures both global and token-level interactions across text, visual, and acoustic modalities.
• Empirical results on MIntRec benchmarks demonstrate superior performance in rare-class recognition compared to previous state-of-the-art models.

MVCL-DAF++ is an advanced multimodal intent recognition (MMIR) framework that addresses limitations in semantic grounding and robustness, particularly under noisy and rare-class conditions. By integrating @@@@1@@@@ with a coarse-to-fine dynamic attention fusion mechanism, MVCL-DAF++ achieves superior alignment of modality representations and enhances rare-class recognition capabilities. The architecture delivers state-of-the-art results on benchmarks such as MIntRec and MIntRec2.0 and establishes robust protocols for both training and inference in MMIR pipelines (Huang et al., 22 Sep 2025).

1. Framework Architecture

MVCL-DAF++ advances over the baseline MVCL-DAF with two principal components: Prototype-Aware Contrastive Alignment (PAC) and Coarse-to-Fine Dynamic Attention Fusion (DAF).

1.1 Prototype-Aware Contrastive Alignment

For a mini-batch of $N$ instances across $C$ classes:

• Each instance $i$ is encoded to an embedding $h_i \in \mathbb{R}^d$ post-shared multimodal encoder and initial DAF.
• For each class $c$, the indices $I_c = \{ i \mid y_i = c \}$ are identified.
• The class-prototype $r_c$ is batch-computed and L2-normalized:

$$r_c = \mathrm{normalize}\left( \frac{1}{|I_c|} \sum_{i \in I_c} h_i \right)$$

• Prototype updates are skipped for classes absent in the batch.

The prototype InfoNCE loss for instance $i$ ($y_i$ its label):

$$\mathcal{L}_{\text{proto}}^{(i)} = -\log \frac{ \exp \left( \mathrm{sim}(h_i, r_{y_i})/\tau \right) }{ \sum_{c=1}^{C} \exp \left( \mathrm{sim}(h_i, r_c) / \tau \right) }$$

with cosine similarity $\mathrm{sim}(u, v) = u^T v / \|u\| \|v\|$ and temperature $\tau$. Averaging over the batch gives the full loss:

$$\mathcal{L}_{\text{proto}} = \frac{1}{N} \sum_{i=1}^N \mathcal{L}_{\text{proto}}^{(i)}$$

PAC improves intra-class compactness, inter-class separation, mitigates drift from noisy/out-of-distribution examples, and stabilizes rare-class prototypes even with few batch samples.

1.2 Coarse-to-Fine Dynamic Attention Fusion

DAF captures both global ("coarse") and token-level ("fine") cross-modal interactions.

• Tokens for each modality $$m \in \{\text{text}, \text{visual}, \text{acoustic}\}$$: $X^m \in \mathbb{R}^{L_m \times d_m}$
• Each $X^m$ is projected and encoded via a modality-specific Transformer, producing:
  • Global summary $$g_m = \mathrm{Pool}(\mathrm{TransEnc}_m(X^m)) \in \mathbb{R}^d$$
  • Token-wise features $$T^m = \mathrm{TransEnc}_m(X^m) \in \mathbb{R}^{L_m \times d}$$

Coarse Fusion

Global context $G = [g_t; g_v; g_a]$:

$$A_c = \mathrm{softmax} \left( \frac{G G^T}{\sqrt{d}} \right), \quad M_c = A_c G$$

Pooling $M_c$ produces $m_c \in \mathbb{R}^d$.

Fine Fusion with Dynamic Attention

For modality $m$ at position $\ell$:

$$\alpha^m_{\ell,\text{coarse}} = \frac{ \exp\left( (W_q t^m_\ell)^T (W_k m_c)/\sqrt{d} \right) }{ \exp\left( (W_q t^m_\ell)^T (W_k m_c)/\sqrt{d} \right) + \exp\left( (W_q t^m_\ell)^T (W_k t^m_\ell)/\sqrt{d} \right) }$$

$$\alpha^m_{\ell,\text{fine}} = 1 - \alpha^m_{\ell,\text{coarse}}$$

$$f^m_\ell = \alpha^m_{\ell,\text{coarse}} \cdot m_c + \alpha^m_{\ell,\text{fine}} \cdot t^m_\ell$$

Aggregated fused token representations $\{f^t, f^v, f^a\}$ are concatenated or summed and further fused via self-attention, yielding $M_f$ (fine) and $M_{cf}$ (coarse-enhanced), the latter used for classification.

2. Training Protocol and Loss Design

2.1 Batch and Optimization Strategy

• Batch size: 32 multimodal instances per GPU (text, visual, acoustic).
• Batch composition ensures most classes are represented—optionally using class-aware sampling in long-tailed regimes.
• Optimizer: AdamW with initial learning rate $2 \times 10^{-5}$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, weight decay 0.2.
• Linear warmup for first 10% of steps, then linear decay; maximum 100 epochs with early stopping (patience 10 epochs on validation WF1).

2.2 Loss Functions

The multi-term objective:

$$\mathcal{L} = \mathcal{L}_{\text{cls}} + \lambda_{\text{contra}} \mathcal{L}_{\text{contrastive}} + \lambda_{\text{proto}} \mathcal{L}_{\text{proto}}$$

• $\mathcal{L}_{\text{cls}}$: CrossEntropy over $M_{cf}$
• $\mathcal{L}_{\text{contrastive}}$: Multi-view instance InfoNCE loss, aligning textual, visual, acoustic, masked, and fine-grained embeddings:

$$\mathcal{L}_{\text{contrastive}} = \sum_{(a, p)} \left[ -\log \frac{ \exp(\mathrm{sim}(h_a, h_p)/\tau) }{ \sum_j \exp(\mathrm{sim}(h_a, h_j)/\tau) } \right]$$

Typical anchor-positive pairs: (text, fine), (text, visual), (text, acoustic), (text, masked-text).

• $\mathcal{L}_{\text{proto}}$: Prototype-aware contrastive loss as above.
• Hyperparameters: $\tau = 0.1$, $\lambda_{\text{contra}} = 1.0$, $\lambda_{\text{proto}} = 1.0$.

3. Experimental Protocol and Empirical Results

3.1 Datasets

• MIntRec 1.0: 2,224 samples, 20 intent classes, class-balanced.
• MIntRec 2.0: 15,040 samples, 30 intent classes, long-tailed and open-intent distribution.

3.2 Evaluation Metrics

• Accuracy (ACC)
• Weighted F1 (WF1)
• Weighted Precision (WP)
• Recall (R)

3.3 Results and Ablation

Dataset	Model	ACC	WF1	WP	R	Rare-Class ΔWF1
MIntRec1.0	MVCL-DAF	74.72%	74.61%	—	—	—
MIntRec1.0	MVCL-DAF++	76.18%	75.66%	76.17%	74.39%	+1.05%
MIntRec2.0	Previous best	—	55.05%	—	—	—
MIntRec2.0	MVCL-DAF++	60.40%	59.23%	60.51%	53.96%	+4.18%

Ablation analysis shows:

• Removing prototype alignment: WF1 drop ≈1.04% (MIntRec), ≈1.27% (MIntRec2.0).
• Removing coarse-to-fine fusion causes similar drops.
• Loss ablation (MIntRec, Table 2):

| Loss Components | WF1 (%) | |----------------------------------|---------| | Classification only | 73.88 | | + Instance contrastive | 74.42 | | + Prototype only | 74.60 | | All three | 75.66 |

4. Algorithmic Flow

For each training step:

1. Sample a batch of $B$ triplets $\{ X^t, X^v, X^a, y \}$.
2. Encode each modality: $T^m = \mathrm{TransEnc}_m(X^m)$, $m \in \{\text{t}, \text{v}, \text{a}\}$.
3. Compute global summaries: $g_m = \mathrm{Pool}(T^m)$.
4. Conduct coarse fusion of $G = [g_t; g_v; g_a]$ to obtain $m_c$.
5. For each token $t^m_\ell$, compute $\alpha^m_{\ell,\text{coarse}}$, $\alpha^m_{\ell,\text{fine}}$, and fuse to $f^m_\ell$.
6. Fuse across modalities to construct $M_f$ (tokens), $M_{cf}$ (pooled).
7. Predict logits from $M_{cf}$; apply CrossEntropy to yield $\mathcal{L}_{\text{cls}}$.
8. Build positive/negative pairs and prototypes; compute $\mathcal{L}_{\text{contrastive}}$ and $\mathcal{L}_{\text{proto}}$.
9. Construct total loss; perform backward pass and update parameters (including prototype states).

5. Practical Considerations

• Model built upon BERT-base for textual inputs and small Transformers for video/audio, totaling ≈110M parameters.
• On NVIDIA A100, inference latency is ≈15–20 ms per multimodal instance.
• Prototype stability requires ensuring at least one batch sample per class or carrying forward the latest valid prototype.
• Temperature $\tau$ and loss weights $\lambda_{\text{contra}},\lambda_{\text{proto}}$ are sensitive; default $\tau = 0.1$, $\lambda = 1.0$ found robust.
• Early stopping (patience = 10) helpful against overfitting rare, noisy classes.
• In extreme class imbalance, class-aware sampling is recommended to prevent prototype collapse.
• L2-normalization of both $h_i$ and $r_c$ is essential before cosine-based loss calculations.

MVCL-DAF++ establishes state-of-the-art performance on MMIR benchmarks, with notable gains in robustness for rare-class recognition and improved reliability under noisy data settings (Huang et al., 22 Sep 2025).