MIntRec2.0: Multimodal Intent Benchmark

• MIntRec2.0 is a large-scale multimodal benchmark for intent recognition in conversations, featuring text, video, and audio modalities.
• The dataset supports both in-scope classification with 30 intent classes and out-of-scope detection in realistic multi-turn, multi-party settings.
• It employs a rigorous annotation process and advanced fusion techniques to evaluate multimodal inconsistency, context use, and dataset biases.

MIntRec2.0 is a benchmark dataset and evaluation framework for multimodal intent recognition in conversations, designed to support both fine-grained in-scope intent classification and out-of-scope (OOS) detection in realistic multi-turn, multi-party settings. It was introduced as a large-scale extension of the earlier MIntRec benchmark, expanding the problem from small-scale single-turn intent recognition to open-world conversational inference with text, video, and audio, explicit speaker identity, and a 30-class intent taxonomy (Zhang et al., 2024). In later multimodal intent recognition literature, MIntRec2.0 has become a principal evaluation target for inconsistency-aware fusion, prototype-guided alignment, open-set robustness, and dataset-bias analysis (Wang et al., 10 May 2026, Huang et al., 22 Sep 2025, Wang et al., 12 Sep 2025, Mullick et al., 22 Aug 2025).

1. Definition and corpus scope

MIntRec2.0 is positioned as the first large-scale multimodal multi-party conversational intent dataset, created to close the gap between earlier multimodal intent benchmarks and more realistic conversational settings (Zhang et al., 2024). Relative to the original MIntRec, which contains 2,224 utterances and 20 intent classes, MIntRec2.0 contains 15,040 utterances and 30 in-scope intent classes, and additionally incorporates OOS utterances arising in multi-turn conversations.

Aspect	Value
Dialogues	1,245
Utterances / samples	15,040
In-scope utterances	9,304
OOS utterances	5,736
In-scope intent classes	30
Modalities	Text, video, audio
Setting	Multi-turn, multi-party conversation
Speaker annotation	Explicit speaker identity

The corpus comprises 12.3 hours of video, 118,477 words, and 9,524 unique words. The average utterance length is 7.9 words, with a maximum utterance length of 46 words; the average video clip duration is 3.0 seconds, with a maximum of 19.9 seconds. The source distribution is 53% Superstore, 22% The Big Bang Theory, and 25% Friends. The dialogue structure is explicitly multi-party: 51.5% of dialogues contain more than two speakers (Zhang et al., 2024).

This scope is central to the dataset’s technical role. MIntRec2.0 is not merely a larger collection of utterances; it couples multimodal intent recognition with open-world rejection and contextual speaker-aware reasoning. This combination explains why later work treats it as a harder benchmark than MIntRec for long-tailed, ambiguous, and conflict-heavy multimodal understanding (Huang et al., 22 Sep 2025, Wang et al., 10 May 2026).

2. Corpus construction and annotation procedure

The raw material was collected from three TV series, with videos obtained from YouTube and subtitles from OpenSubtitles. The creators split source videos into continuous utterance-level segments according to subtitle timestamps and extracted corresponding audio segments. These utterances were then manually grouped into dialogues based on whether they belonged to the same scene and episode, following the dialogue segmentation principle used in MELD (Zhang et al., 2024).

Annotation was performed by six college students proficient in English. A dedicated multimodal annotation platform allowed joint inspection of transcript, video, and audio. Each utterance was assigned exactly one label from 30 in-scope intent labels plus one OOS label. Every utterance was labeled by three annotators, and final labels were determined by majority voting; utterances receiving three different votes were discarded. Inter-annotator agreement, measured by Fleiss’ kappa, ranged from 0.66 to 0.70, with an average of 0.69, which the dataset paper interprets as substantial agreement (Zhang et al., 2024).

The benchmark explicitly adopts a single-intent assumption per utterance. The justification given is empirical: in an auxiliary multi-intent annotation on the test set, only 136 out of 3,230 utterances, or 4.2%, had a second probable intent, and none had a third. This choice simplifies evaluation but also defines the task narrowly, emphasizing utterance-level dominant intent over richer pragmatic mixtures.

Speaker identity is a first-class annotation variable. The dataset manually annotates 21 main characters in Superstore, 7 in The Big Bang Theory, and 6 in Friends, covering 90.4% of the data; remaining less frequent characters are also differentiated within dialogues. As a result, MIntRec2.0 supports speaker-aware contextual retrieval rather than generic chronological context aggregation (Zhang et al., 2024).

3. Intent taxonomy, task formulation, and open-world protocol

MIntRec2.0 uses a hierarchical intent taxonomy derived from the original MIntRec taxonomy. The original benchmark defined 20 fine-grained intents under two coarse groups, “Express emotions or attitudes” and “Achieve goals.” MIntRec2.0 adds 10 high-frequency intent classes required by continuous multi-turn conversation, yielding 30 in-scope fine-grained classes plus one OOS category (Zhang et al., 2024).

The five added classes under “Express emotions or attitudes” are Doubt, Acknowledge, Refuse, Warn, and Emphasize. The five added classes under “Achieve goals” are Ask for opinions, Confirm, Explain, Invite, and Plan. These new classes account for 37.3% of the utterances. The OOS category covers utterances that do not belong to any of the 30 predefined intent classes, including utterances mainly conveying factual information without useful task-specific intent and utterances reflecting open-ended intentions not covered by the taxonomy.

The dataset supports both single-turn and multi-turn settings. In the single-turn setting, each utterance is processed independently with text, video, and audio. In the multi-turn setting, the input is a dialogue, but the benchmark does not use unrestricted full-dialogue encoding. Instead, it retrieves same-speaker historical context for the current utterance and concatenates that history before multimodal fusion. For text, segment embeddings distinguish current utterance from context; for nonverbal modalities, a one-dimensional zero vector is inserted as a separator (Zhang et al., 2024).

The multimodal fusion problem is formulated as

$$\mathbf{z}=\mathcal{F}(\mathbf{E}^{T}, \mathbf{E}^{V}, \mathbf{E}^{A}),$$

where $\mathbf{E}^{T}$, $\mathbf{E}^{V}$, and $\mathbf{E}^{A}$ denote text, visual, and audio embedding sequences, and $\mathcal{F}$ is the multimodal fusion network. The benchmark then separates training treatment for in-scope and OOS samples. For in-scope samples it uses cross-entropy, while for OOS samples it uses Outlier Exposure (OE), encouraging a uniform soft target over the known intent classes. At inference, OOS rejection is performed with DOC (Deep Open Classification), using classwise sigmoid confidence and per-class thresholds of the form

$$\delta_k = \operatorname{max}(0.5, 1 - \alpha\sigma_k),$$

with $\alpha = 1$ (Zhang et al., 2024).

The split statistics clarify an important distinction that later papers do not always preserve. For the full benchmark, the dataset is split into 871 training dialogues with 9,989 utterances, 125 validation dialogues with 1,821 utterances, and 249 testing dialogues with 3,230 utterances. Within these totals, the in-scope portions are 6,165 training, 1,106 validation, and 2,033 testing utterances. This is why later closed-set MIR papers often describe MIntRec2.0 using only the 9,304 in-scope samples and the 6,165/1,106/2,033 split, even though the original benchmark definition includes 5,736 OOS utterances as well (Zhang et al., 2024, Wang et al., 10 May 2026).

4. Representation pipeline, official baselines, and reference results

The benchmark provides an explicit multimodal feature-extraction pipeline. Text uses BERT, specifically BERT$_{\mathrm{LARGE}}$ with 24 transformer layers; visual processing localizes the speaking person through scene detection, Faster R-CNN, S3FD face detection, face tracking, and audio-visual active speaker detection using “Is Someone Speaking?”, then applies Mask R-CNN, Swin Transformer pretrained on ImageNet-1K, RoIAlign, and average pooling; audio is loaded with librosa at 16,000 Hz and encoded with WavLM (Zhang et al., 2024). The resulting feature dimensions are $D^T = 1024$, $D^V = 256$, and $\mathbf{E}^{T}$0. In single-turn experiments, maximum padded lengths are 50 for text, 180 for video, and 400 for audio.

The benchmark supports both aligned and unaligned fusion styles. MAG-BERT is evaluated as an aligned model, using CTC for single-turn alignment and conv1d for multi-turn alignment. MulT is evaluated as an unaligned multimodal transformer. General training uses AdamW, 40 epochs, PyTorch 1.13.1, Tesla V100-SXM2-32GB hardware, and results averaged over five runs with random seeds 0–4 (Zhang et al., 2024).

The reference results establish several baseline properties. In single-turn training without OOS data, the text baseline obtains F1 51.60 and ACC 59.30, MAG-BERT obtains F1 55.17 and ACC 60.58, and MulT obtains F1 54.12 and ACC 60.66. In the same regime, OOS-related performance is much lower: F1-OOS is 30.40 for text, 34.03 for MAG-BERT, and 38.57 for MulT. When OOS training data are added, F1-OOS rises sharply to 61.54 for text, 62.47 for MAG-BERT, and 61.66 for MulT, while in-scope gains remain modest (Zhang et al., 2024).

The multi-turn setting with same-speaker context yields only limited improvement. With OOS training, Context TEXT reaches F1 53.61 and ACC 59.04, Context MAG-BERT reaches F1 53.89 and ACC 59.84, and Context MulT reaches F1 53.96 and ACC 59.48. This limited gain is one of the dataset paper’s central findings: context is available, but exploiting it effectively remains difficult (Zhang et al., 2024).

The benchmark also includes comparisons with ChatGPT and human evaluators. ChatGPT-0 reaches in-scope ACC 35.27 and overall F1 28.34, while Humans-100 reach in-scope ACC 71.03, WF1 75.63, joint ACC 71.86, F1-OOS 75.41, and overall F1 69.49. The reported gap between LLMs and humans is therefore substantial. This suggests that MIntRec2.0 measures a cognitively demanding form of intent understanding that is not reduced to conventional text classification (Zhang et al., 2024).

5. Subsequent use as a benchmark in MIR research

Later MIR papers use MIntRec2.0 in two distinct ways. Some treat it as a closed-set in-scope benchmark using the 9,304 in-scope utterances and the 6,165/1,106/2,033 split; others retain the original joint in-scope/OOS evaluation protocol. This bifurcation is technically important because results reported under these two regimes are not directly comparable.

In “Mitigating Multimodal Inconsistency via Cognitive Dual-Pathway Reasoning for Intent Recognition,” MIntRec2.0 is one of the two core experimental benchmarks and is explicitly described as “an extended version of MIntRec” that “further expands the data scale and categorical diversity” (Wang et al., 10 May 2026). That work uses the in-scope portion as a main benchmark for overall performance comparison, ablation analysis, efficiency analysis, and qualitative case studies. Under its protocol, CDPR reports ACC 60.82, F1 53.86, P 57.88, R 53.40, WF1 59.54, and WP 60.23 on MIntRec2.0, with reported gains over the best baseline of +2.17 ACC, +1.59 F1, +0.97 P, +1.05 R, +1.38 WF1, and +1.27 WP. Its methodological explanation centers on representation disentanglement into shared and private subspaces, an intuition pathway for consensus, a reasoning pathway for conflict resolution, and an inconsistency perception mechanism that combines prototype matching with calibration statistics.

In “MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion,” MIntRec2.0 is again treated as a primary benchmark and characterized as extending MIntRec “to a more challenging setting with 30 intent classes, incorporating long-tailed distributions and open-intent utterances” (Huang et al., 22 Sep 2025). The paper reports that MIntRec2.0 contains 1,245 dialogues and 15,040 samples spanning text, video, and audio. On that benchmark, MVCL-DAF++ reports ACC 60.40, WF1 59.23, WP 60.51, and R 53.96, improving over MVCL-DAF by +2.60 ACC, +4.18 WF1, +4.69 WP, and +11.93 R. The paper interprets these gains as evidence for prototype-aware semantic grounding and coarse-to-fine fusion, particularly under long-tailed and ambiguous conditions, although it does not provide a standalone rare-class metric table.

In “DyKen-Hyena: Dynamic Kernel Generation via Cross-Modal Attention for Multimodal Intent Recognition,” MIntRec2.0 is used in its open-set form and explicitly framed as a benchmark for joint in-scope intent recognition and OOS detection (Wang et al., 12 Sep 2025). The paper follows the official protocol and reports not only Acc, WF, WP, F1, Prec, and Rec, but also oid_acc, F1-IS, F1-OOS, and oid_f1. DyKen-Hyena achieves Acc 60.54, WF 59.79, WP 60.21, F1 55.02, Prec 58.35, Rec 54.53, oid_acc 45.63, F1-IS 46.21, F1-OOS 38.69, and oid_f1 45.97. The most emphasized result is the +10.46 absolute improvement in F1-OOS over the best baseline, from 28.23 to 38.69, which the paper attributes to audio-visual conditioned dynamic kernels that modulate token-level textual processing rather than merely fusing modality features.

Taken together, these later studies show that MIntRec2.0 has become a standard stress test for multiple MIR hypotheses: multimodal inconsistency handling, long-tailed semantic alignment, hierarchical fusion, and open-set robustness. They also show that benchmark usage is no longer uniform. A plausible implication is that any interpretation of MIntRec2.0 results now depends on whether a paper is evaluating the full open-world benchmark or only its in-scope subset.

6. Modality bias, limitations, and benchmarking debates

A major later reassessment argues that MIntRec2.0, despite its multimodal design, is strongly text-dominated. “Text Takes Over: A Study of Modality Bias in Multimodal Intent Detection” treats MIntRec2.0 as one of its two core benchmarks and describes it as “the largest available dataset for multimodal intent detection,” with 15,040 samples, a 30-class intent taxonomy, and text, audio, and video modalities (Mullick et al., 22 Aug 2025). On the original MIntRec2.0 benchmark, the paper reports that a text-only Mistral-7B model reaches 65.2 Acc and 64.4 F1, outperforming the listed multimodal non-LLM baselines such as MulT at 58.4 Acc and 51.5 F1, MAG at 58.2 Acc and 49.4 F1, MISA at 57.8 Acc and 51.9 F1, and SDIF at 58.6 Acc and 52.5 F1.

The paper supports the text-bias claim through automatic modality annotation, modality ablations, and human verification. Using MulT as the automatic annotator, it reports that 57.85% of MIntRec2.0 samples can be handled by text alone, 28.43% require T+V+A, and the aggregated share of samples requiring text in some form, denoted $\mathbf{E}^{T}$1, is 94.00%. Human annotation of 1,489 MIntRec2.0 samples yields a similar pattern: 80.63% are labeled as text-only, 7.60% as T+V+A, 6.50% as T+V, 4.31% as T+A, and 0.96% as V. The debiasing procedure then removes textually biased samples by majority vote over three text-dominant classifiers, producing an exact total reduction of 54.04% and leaving a debiased MIntRec2.0 with 2,826 train, 475 dev, and 891 test instances across 27 intent categories after removing Apologize, Thank, and Greet (Mullick et al., 22 Aug 2025).

Performance on this debiased version drops sharply across model families. BERT falls from 57.1 Acc / 49.3 F1 to 18.5 Acc / 22.6 F1; Mistral-7B falls from 65.2 / 64.4 to 33.6 / 34.7; MulT falls from 58.4 / 51.5 to 22.7 / 25.4; SDIF falls from 58.6 / 52.5 to 19.4 / 23.1; GPT-4V falls from 41.8 / 41.4 to 30.5 / 31.0. The authors argue that this degradation is not merely a size effect, since size-matched random subsets do not cause comparable collapse. This suggests that raw leaderboard gains on MIntRec2.0 can overstate genuine multimodal understanding when textual cues dominate.

The original dataset paper also records limitations that remain relevant to any interpretation of the benchmark. It notes that MIntRec2.0 is built from TV-show dialogues only, that privacy concerns would arise in real deployments using text, face, and voice data, that cultural diversity may be limited, and that context use, OOS handling, and nonverbal modeling still leave substantial room for improvement (Zhang et al., 2024). These caveats, together with the later bias analysis, have shifted MIntRec2.0’s meaning within the literature. It remains a central benchmark, but it is increasingly treated not only as a test of multimodal intent recognition performance, but also as a test of how faithfully that performance reflects multimodal rather than primarily textual competence.