MIntRec-1: Multimodal Intent Benchmark

• MIntRec-1 is a multimodal intent recognition benchmark derived from Superstore, combining text, video, and audio to capture real-world conversational intent.
• It comprises 2,224 samples across 43 videos with 20 fine-grained intent classes organized into 2 coarse categories, enabling evaluation of missing modalities and semantic inconsistencies.
• The benchmark reveals modality bias, with over 90% of samples relying on text cues, challenging multimodal fusion models to overcome text-dominance.

MIntRec-1 is the closed-set multimodal intent recognition benchmark introduced as MIntRec and later referred to as MIntRec-1 / MIntRec 1.0 in subsequent work. It contains 2,224 high-quality samples from 43 videos, with 20 fine-grained intent classes grouped into 2 coarse-grained intent classes, and each sample includes text, video, and audio. The dataset was built from the TV series Superstore to study intent recognition in real-world conversational scenes rather than in text-only query settings, and it has become a standard benchmark for multimodal intent recognition, robustness under missing modalities, out-of-scope detection, and multimodal inconsistency analysis (Zhang et al., 2022, Huang et al., 22 Sep 2025).

1. Origins and benchmark identity

MIntRec was introduced to address a limitation of prior intent datasets: most existing resources were described as fundamentally limited because they were built almost entirely from text-only utterances in constrained task-oriented settings such as voice assistants or customer-service queries. The motivation was that many real-world intents are ambiguous from text alone and can only be disambiguated by combining text, video, and audio. In this formulation, multimodal intent recognition is the task of inferring a speaker’s intention from verbal and nonverbal evidence jointly (Zhang et al., 2022).

The original paper positioned MIntRec against several earlier resources. ATIS, SNIPS, CLINC150, HWU64, and BANKING77 were characterized as mostly single-modality text corpora. Intentonomy was described as visual intent analysis based on images from social media rather than conversational spoken language, and MDID was described as combining image and text from Instagram posts but remaining far from natural spoken dialogue. MIntRec was therefore designed as a benchmark for real-world conversational scenes in which intention is expressed through facial expression, body movement, gesture, gaze, and tone of voice, in addition to lexical content (Zhang et al., 2022).

In later work, the benchmark called MIntRec in the original release is explicitly treated as MIntRec-1 / MIntRec 1.0. This later nomenclature mainly serves to distinguish the original closed-set classification benchmark from MIntRec2.0, which adds out-of-scope (OOS) intent detection (Huang et al., 22 Sep 2025).

2. Corpus construction and intent taxonomy

The dataset is built from the TV series Superstore, chosen because it provides a large cast of recurring characters with distinctive behaviors and speaking styles, many everyday scenes with diverse social interactions, and rich multimodal signals including expressions, gestures, and tone. The raw videos and subtitles were collected from YouTube and OpenSubtitles. The construction pipeline extracted utterance start/end timestamps, split the raw videos into utterance-level video segments, extracted aligned audio segments using moviepy, and paired each text utterance with its aligned video and audio segment (Zhang et al., 2022).

The final corpus contains 2,224 samples, 43 videos, 20 fine-grained intent classes, 2 coarse-grained intent classes, and 3 modalities. Reported corpus statistics include 15,658 total words, 2,562 unique words, average utterance length 7.04 words, maximum utterance length 26, average video segment length 2.38 s, and maximum video segment length 9.59 s (Zhang et al., 2022).

A major contribution of MIntRec is its intent taxonomy. The two coarse-grained categories are:

• Express emotions or attitudes
• Achieve goals

The fine-grained classes under Express emotions or attitudes are Complain, Praise, Apologise, Thank, Criticize, Care, Agree, Taunt, Flaunt, Oppose, and Joke. The fine-grained classes under Achieve goals are Inform, Advise, Arrange, Introduce, Comfort, Leave, Prevent, Greet, and Ask for help (Zhang et al., 2022).

The original dataset paper also distinguished classes by their dominant signaling modality. It described thank, apologise, greet, agree, praise as mostly signaled by text; leave, prevent as mostly signaled by video; complain, criticize as mostly signaled by audio; and comfort, care, joke, taunt, flaunt as requiring multiple modalities together. This class structure is central to the benchmark’s design, because it embeds the claim that human intent is often not purely lexical (Zhang et al., 2022).

3. Annotation procedure, speaker localization, and modality features

Annotation was performed by five trained student annotators with English proficiency. Annotators were given explanations of each class, typical examples, and an annotation platform. Each sample was labeled independently by all five annotators using text, audio, and video together. The final label was determined by majority vote: a sample was kept only if at least 3 of 5 annotators agreed on a non-UNK label, while utterances irrelevant to the intent context were labeled UNK and excluded when not sufficiently agreed upon. The resulting benchmark is imbalanced, reflecting real-world distributions, although every class has at least 50 samples (Zhang et al., 2022).

A distinctive feature of MIntRec is the provision of speaker bounding boxes for video segments. This was introduced because a shot may contain multiple people and the target speaker must be identified to extract the correct visual evidence. The automatic speaker annotation pipeline uses scene detection, a pretrained Faster R-CNN with ResNet-50 backbone for person detection, S3FD for face detection, IoU-based face tracking with threshold 0.5, tolerance of up to 10 consecutive frames of missing faces, and a pretrained TalkNet active speaker detection model to identify the speaker (Zhang et al., 2022).

The process generated more than 120K keyframes with speaker annotations. For validation, the authors built a test set of more than 12K human-annotated keyframes. The reported missing rate of generated keyframes was 2.3%, and among matched keyframes 90.9% of predicted bounding boxes had IoU > 0.9, indicating that the automatic speaker annotation is reliable (Zhang et al., 2022).

For downstream modeling, the benchmark supports standardized multimodal features. Text uses BERT-base-uncased with hidden size 768. Video features are extracted from detected person regions by Faster R-CNN + ResNet-50 and have dimension 256. Audio is extracted with librosa at 16 kHz and represented with wav2vec 2.0 features of dimension 768 (Zhang et al., 2022).

4. Original experimental protocol and benchmark difficulty

The original benchmark split is random by video segments into 1,334 train, 445 validation, and 445 test, corresponding to a 3:1:1 split. The reported evaluation metrics are Accuracy (ACC), Macro F1, Macro Precision, and Macro Recall. Implementation settings include fixed sequence lengths $L_T = 30$, $L_V = 230$, and $L_A = 480$, batch size 16, 100 epochs, hyperparameter tuning by macro F1, and results averaged over 10 runs with seeds 0–9 (Zhang et al., 2022).

The benchmark paper adapted three multimodal fusion baselines: MulT, MISA, and MAG-BERT. On the twenty-class task, the text-only baseline achieved 70.88 ACC. The strongest multimodal results were reported for MAG-BERT with 72.65 ACC, MulT with 72.52 ACC, and MISA with 72.29 ACC. On the binary coarse-grained task, the text-only baseline achieved 88.09 ACC, while MAG-BERT, MulT, and MISA achieved 89.24, 89.19, and 89.21 ACC, respectively. The original conclusion was that adding nonverbal modalities yields consistent gains of about 1% to 2% over text-only performance in both binary and multiclass settings (Zhang et al., 2022).

Human performance remained substantially higher. On the twenty-class task, human annotators achieved 85.51 ACC and 85.07 F1; on the binary task, they achieved 94.72 ACC and 94.67 F1. The human-vs-model gap was described as evidence that MIntRec is still challenging and that contemporary multimodal fusion methods were far from solving real intent understanding (Zhang et al., 2022).

Class-wise analysis in the original paper showed that multimodal cues helped substantially for complain, criticize, taunt, oppose, and joke, all of which depend strongly on tone, facial expression, or body behavior. By contrast, apologise, thank, praise, and agree showed little improvement or even slight degradation with multimodal fusion, because these categories often contain explicit lexical cues such as “sorry,” “thank you,” “yeah,” or “yes” (Zhang et al., 2022). A plausible implication is that the benchmark combines genuinely multimodal cases with many lexically recoverable cases.

5. Text dominance, modality bias, and debiasing

A later reassessment argued that many “multimodal” intent benchmarks are in practice text-dominant rather than truly balanced across text, audio, and video, and used MIntRec-1 (M-1) as a central case study. The paper analyzed MIntRec-1 in three stages: initial model performance, modality-importance analysis, and debiasing (Mullick et al., 22 Aug 2025).

The central empirical result was that a text-only LLM, Mistral-7B, performed best on the original MIntRec-1 benchmark with 82.9% accuracy. This was about 9 percentage points higher than competitive multimodal models such as SDIF: 72.8%, MulT: 71.5%, MAG: 72.7%, and MISA: 71.8%. The same study also reported that text-only LLMs outperformed multimodal LLMs by an even larger margin: GPT-4T: 60.4%, GPT-4V: 59.5%, ClaudeT: 57.7%, and ClaudeV: 59.1% (Mullick et al., 22 Aug 2025).

To quantify modality relevance, the study performed masking over the seven modality combinations $T$, $V$, $A$, $T+V$, $T+A$, $V+A$, and $T+V+A$, and defined

$L_V = 230$0

On MIntRec-1, the distribution of minimal modality requirements was reported as Text only (T): 69.66%, Video only (V): 3.60%, Audio only (A): 4.72%, T+V: 1.35%, T+A: 3.37%, V+A: 0.67%, and T+V+A: 16.63%. Crucially, $L_V = 230$1, meaning that about 91% of MIntRec-1 samples require text for correct classification, either alone or in combination with other modalities (Mullick et al., 22 Aug 2025).

Human evaluation confirmed this pattern. Manual annotation of a subset of MIntRec-1 yielded Text-only: 82.46%, Text+Video: 5.69%, Text+Video+Audio: 5.01%, Text+Audio: 4.56%, and Video-only: 2.28%, with 85.7% inter-annotator agreement. The paper also identified particularly text-biased intent classes, including Agree, Apologise, Greet, Praise, Thank, and Care, and attributed their recoverability to explicit lexical cues such as “sorry,” “thank you,” “okay,” and “yeah” (Mullick et al., 22 Aug 2025).

The paper distinguished explicit textual bias from implicit textual bias. Explicit bias refers to direct lexical signals such as “I am sorry” $L_V = 230$2 Apologise, “Thank you” $L_V = 230$3 Thank, and “Okay”, “yeah”, “right” $L_V = 230$4 Agree. Implicit bias refers to samples in which intent is not explicitly stated but can still be inferred from wording or context. In the automated framework, the minimal sufficient modality subset for each test sample was chosen as the smallest combination that either correctly predicts the label or gives the highest probability for the ground-truth label (Mullick et al., 22 Aug 2025).

The debiasing framework identified textually biased samples using three models: BERT as a text-only model, LLaMA-2-7B as a text-only LLM, and SDIF as a small multimodal model with masked audio/video. A sample was labeled textually biased if it was correctly classified by the majority of these three models. Because the dataset is relatively small, the authors used 5 different splits, dividing the training set into three parts and applying a round-robin scheme across runs (Mullick et al., 22 Aug 2025).

After filtering, about 70% of MIntRec-1 samples were removed. Heavily biased intent categories such as Agree, Apologise, Care, Greet, Praise, and Thank became too small after filtering and were excluded from the final debiased version. The final debiased MIntRec-1 contained 290 train, 107 dev, and 108 test samples and 14 intent categories instead of 20. The paper reported a 74.69% total reduction in the bias-heavy label groupings shown in the debiasing statistics (Mullick et al., 22 Aug 2025).

Debiasing caused a substantial performance collapse across model families. On MIntRec-1, M-7B dropped from 82.9% to 32.4%, BERT from 70.8% to 16.7%, SDIF from 72.8% to 22.2%, and GPT-4V from 59.5% to 52.9%. The paper summarized this as textual models drop by more than 50% and smaller multimodal models drop by 50–60% or more. A random-subset control showed that the debiased set performs much worse than a random subset of equal size, indicating that the degradation is due to removing textually easy samples rather than merely reducing dataset size (Mullick et al., 22 Aug 2025).

This reassessment directly challenged the interpretation of the original benchmark. The paper concluded that MIntRec-1 is not a reliable benchmark for measuring true multimodal understanding in its original form, because text dominates the dataset, a text-only LLM can outperform multimodal systems, most samples do not require video/audio, reported multimodal gains are inflated by text shortcuts, and once bias is removed all models suffer large drops (Mullick et al., 22 Aug 2025).

6. Subsequent methodological developments and evolving uses

Despite the critique of modality bias, MIntRec-1 has remained a widely used benchmark for developing multimodal architectures. One line of work emphasizes richer nonverbal modeling. WDMIR introduces a Wavelet-Driven Multimodal Intent Recognition framework with a Wavelet-driven Fusion Module, Collaborative Representation Module, and Progressive Fusion Module. On MIntRec it reports 75.06 ACC, 74.96 WF1, 75.26 WP, and 72.65 R, improving over SDIF-DA* by +1.13 ACC, +1.07 WF1, +1.08 WP, and +0.99 R. Its main claim is that frequency-domain analysis of audio-video signals improves the extraction of subtle nonverbal semantics (Gong et al., 27 May 2025).

A second line moves from conventional fusion to token-level modulation. DyKen-Hyena is a BERT-based model that inserts multimodal fusion before the main BERT encoder and uses cross-modal attention to generate dynamic, per-token convolutional kernels that modulate textual feature extraction through a Hyena backbone. On the closed-set MIntRec benchmark corresponding to MIntRec-1, it reports Accuracy 73.66, F1 69.26, Precision 70.30, Recall 69.54, Weighted F1 73.05, and Weighted Precision 73.30, with gains of +1.84 Acc, +0.79 F1, and +1.47 WF over the best baseline in that paper. On MIntRec2.0, it reports F1-OOS 38.69, an improvement of +10.46 absolute F1 points over the best baseline for out-of-scope detection (Wang et al., 12 Sep 2025).

Another direction strengthens class-level semantic grounding. MVCL-DAF++ extends MVCL-DAF with Prototype-aware contrastive alignment and Coarse-to-fine dynamic attention fusion. On MIntRec 1.0 / MIntRec-1, it reports ACC 76.18, WF1 75.66, WP 76.17, and R 74.39, outperforming MVCL-DAF by +1.46 ACC, +1.05 WF1, +1.10 WP, and +2.45 Recall. The paper also highlights a +1.05% WF1 improvement in rare-class recognition on MIntRec, and attributes the gain to class prototypes as semantic anchors plus hierarchical fusion of coarse global and fine token-level signals (Huang et al., 22 Sep 2025).

MIntRec has also been used to study incomplete observation. ADMC, an Attention-based Diffusion model for Missing Modalities feature Completion, decouples modality-specific feature extraction from completion and fusion. Under missing-modality evaluation on MIntRec, ADMC reports 33.4 / 29.3 for $L_V = 230$5, 15.4 / 10.2 for $L_V = 230$6, 71.6 / 65.7 for $L_V = 230$7, 34.3 / 29.6 for $L_V = 230$8, 72.3 / 65.8 for $L_V = 230$9, 71.8 / 65.1 for $L_A = 480$0, and 49.8 / 44.3 average WA / UA. In the full-modality enhancement setting, it reports 72.5 WA / 65.4 UA for MMER, compared with 71.2 / 65.0 for jointly training and 71.5 / 65.0 for using ITFN alone (Zhang et al., 8 Jul 2025).

A further line addresses semantic conflict rather than missing data. CDPR, a Cognitive Dual-Pathway Reasoning framework, separates a consensus-driven intuition pathway from a conflict-resolving reasoning pathway, with representation disentanglement and an inconsistency perception mechanism based on semantic prototype matching and statistical calibration. On MIntRec, it reports ACC 75.15, F1 71.04, P 72.01, R 71.08, WF1 74.91, and WP 75.37. In hard categories it reports Taunt: 33.33 F1 and Oppose: 47.06 F1, and under Gaussian noise added to text it remains the strongest method reported in that study, with 55.65 F1 at $L_A = 480$1 and 22.68 F1 at $L_A = 480$2 (Wang et al., 10 May 2026).

Taken together, these works show that MIntRec-1 has served several distinct research functions: as a benchmark for multimodal fusion, as a stress test for nonverbal modeling, as a platform for prototype-guided contrastive learning, as an evaluation bed for missing-modality completion, and as a test case for inconsistency-aware reasoning. At the same time, the modality-bias analysis indicates that strong results on the original benchmark do not by themselves establish robust multimodal understanding (Mullick et al., 22 Aug 2025). This suggests that MIntRec-1 is best understood both as a historically influential multimodal intent dataset and as a benchmark whose evaluation meaning depends strongly on whether textual shortcuts are controlled.