---
title: 'MIntRec-1: Multimodal Intent Benchmark'
url: https://www.emergentmind.com/topics/mintrec-1
type: topic
---

# MIntRec-1: Multimodal Intent Benchmark

MIntRec-1 is the closed-set multimodal intent recognition benchmark introduced as **MIntRec** and later referred to as **MIntRec-1 / MIntRec 1.0** in subsequent work. It contains **2,224 high-quality samples** from **43 videos**, with **20 fine-grained intent classes** grouped into **2 coarse-grained intent classes**, and each sample includes **text, video, and audio**. The dataset was built from the TV series *Superstore* to study intent recognition in real-world conversational scenes rather than in text-only query settings, and it has become a standard benchmark for multimodal intent recognition, robustness under missing modalities, out-of-scope detection, and multimodal inconsistency analysis [2209.04355] [2509.17446].

## 1. Origins and benchmark identity

MIntRec was introduced to address a limitation of prior intent datasets: most existing resources were described as fundamentally limited because they were built almost entirely from **text-only utterances** in constrained task-oriented settings such as voice assistants or customer-service queries. The motivation was that many real-world intents are ambiguous from text alone and can only be disambiguated by combining **text, video, and audio**. In this formulation, multimodal intent recognition is the task of inferring a speaker’s intention from verbal and nonverbal evidence jointly [2209.04355].

The original paper positioned MIntRec against several earlier resources. **ATIS, SNIPS, CLINC150, HWU64, and BANKING77** were characterized as mostly **single-modality text corpora**. **Intentonomy** was described as visual intent analysis based on **images** from social media rather than conversational spoken language, and **MDID** was described as combining image and text from Instagram posts but remaining far from natural spoken dialogue. MIntRec was therefore designed as a benchmark for **real-world conversational scenes** in which intention is expressed through facial expression, body movement, gesture, gaze, and tone of voice, in addition to lexical content [2209.04355].

In later work, the benchmark called **MIntRec** in the original release is explicitly treated as **MIntRec-1 / MIntRec 1.0**. This later nomenclature mainly serves to distinguish the original closed-set classification benchmark from **MIntRec2.0**, which adds **out-of-scope (OOS)** intent detection [2509.17446].

## 2. Corpus construction and intent taxonomy

The dataset is built from the TV series **Superstore**, chosen because it provides a large cast of recurring characters with distinctive behaviors and speaking styles, many everyday scenes with diverse social interactions, and rich multimodal signals including expressions, gestures, and tone. The raw **videos and subtitles** were collected from **YouTube** and **OpenSubtitles**. The construction pipeline extracted utterance **start/end timestamps**, split the raw videos into **utterance-level video segments**, extracted aligned **audio segments** using `moviepy`, and paired each text utterance with its aligned video and audio segment [2209.04355].

The final corpus contains **2,224** samples, **43** videos, **20** fine-grained intent classes, **2** coarse-grained intent classes, and **3 modalities**. Reported corpus statistics include **15,658** total words, **2,562** unique words, average utterance length **7.04** words, maximum utterance length **26**, average video segment length **2.38 s**, and maximum video segment length **9.59 s** [2209.04355].

A major contribution of MIntRec is its intent taxonomy. The two coarse-grained categories are:

- **Express emotions or attitudes**
- **Achieve goals**

The fine-grained classes under **Express emotions or attitudes** are **Complain**, **Praise**, **Apologise**, **Thank**, **Criticize**, **Care**, **Agree**, **Taunt**, **Flaunt**, **Oppose**, and **Joke**. The fine-grained classes under **Achieve goals** are **Inform**, **Advise**, **Arrange**, **Introduce**, **Comfort**, **Leave**, **Prevent**, **Greet**, and **Ask for help** [2209.04355].

The original dataset paper also distinguished classes by their dominant signaling modality. It described **thank, apologise, greet, agree, praise** as mostly signaled by **text**; **leave, prevent** as mostly signaled by **video**; **complain, criticize** as mostly signaled by **audio**; and **comfort, care, joke, taunt, flaunt** as requiring **multiple modalities together**. This class structure is central to the benchmark’s design, because it embeds the claim that human intent is often not purely lexical [2209.04355].

## 3. Annotation procedure, speaker localization, and modality features

Annotation was performed by **five trained student annotators** with English proficiency. Annotators were given explanations of each class, typical examples, and an annotation platform. Each sample was labeled independently by all five annotators using **text, audio, and video** together. The final label was determined by **majority vote**: a sample was kept only if at least **3 of 5** annotators agreed on a non-UNK label, while utterances irrelevant to the intent context were labeled **UNK** and excluded when not sufficiently agreed upon. The resulting benchmark is **imbalanced**, reflecting real-world distributions, although every class has at least **50 samples** [2209.04355].

A distinctive feature of MIntRec is the provision of **speaker bounding boxes** for video segments. This was introduced because a shot may contain multiple people and the target speaker must be identified to extract the correct visual evidence. The automatic speaker annotation pipeline uses **scene detection**, a pretrained **Faster R-CNN** with **ResNet-50** backbone for person detection, **S3FD** for face detection, IoU-based face tracking with threshold **0.5**, tolerance of up to **10 consecutive frames** of missing faces, and a pretrained **TalkNet** active speaker detection model to identify the speaker [2209.04355].

The process generated more than **120K keyframes** with speaker annotations. For validation, the authors built a test set of more than **12K human-annotated keyframes**. The reported missing rate of generated keyframes was **2.3%**, and among matched keyframes **90.9%** of predicted bounding boxes had **IoU > 0.9**, indicating that the automatic speaker annotation is reliable [2209.04355].

For downstream modeling, the benchmark supports standardized multimodal features. **Text** uses **BERT-base-uncased** with hidden size **768**. **Video** features are extracted from detected person regions by **Faster R-CNN + ResNet-50** and have dimension **256**. **Audio** is extracted with **librosa** at **16 kHz** and represented with **wav2vec 2.0** features of dimension **768** [2209.04355].

## 4. Original experimental protocol and benchmark difficulty

The original benchmark split is random by video segments into **1,334 train**, **445 validation**, and **445 test**, corresponding to a **3:1:1** split. The reported evaluation metrics are **Accuracy (ACC)**, **Macro F1**, **Macro Precision**, and **Macro Recall**. Implementation settings include fixed sequence lengths \(L_T = 30\), \(L_V = 230\), and \(L_A = 480\), batch size **16**, **100** epochs, hyperparameter tuning by **macro F1**, and results averaged over **10 runs** with seeds **0–9** [2209.04355].

The benchmark paper adapted three multimodal fusion baselines: **MulT**, **MISA**, and **MAG-BERT**. On the twenty-class task, the **text-only baseline** achieved **70.88** ACC. The strongest multimodal results were reported for **MAG-BERT** with **72.65** ACC, **MulT** with **72.52** ACC, and **MISA** with **72.29** ACC. On the binary coarse-grained task, the text-only baseline achieved **88.09** ACC, while **MAG-BERT**, **MulT**, and **MISA** achieved **89.24**, **89.19**, and **89.21** ACC, respectively. The original conclusion was that adding nonverbal modalities yields consistent gains of about **1% to 2%** over text-only performance in both binary and multiclass settings [2209.04355].

Human performance remained substantially higher. On the twenty-class task, human annotators achieved **85.51** ACC and **85.07** F1; on the binary task, they achieved **94.72** ACC and **94.67** F1. The human-vs-model gap was described as evidence that MIntRec is still challenging and that contemporary multimodal fusion methods were far from solving real intent understanding [2209.04355].

Class-wise analysis in the original paper showed that multimodal cues helped substantially for **complain**, **criticize**, **taunt**, **oppose**, and **joke**, all of which depend strongly on tone, facial expression, or body behavior. By contrast, **apologise**, **thank**, **praise**, and **agree** showed little improvement or even slight degradation with multimodal fusion, because these categories often contain explicit lexical cues such as “sorry,” “thank you,” “yeah,” or “yes” [2209.04355]. A plausible implication is that the benchmark combines genuinely multimodal cases with many lexically recoverable cases.

## 5. Text dominance, modality bias, and debiasing

A later reassessment argued that many “multimodal” intent benchmarks are in practice **text-dominant** rather than truly balanced across text, audio, and video, and used **MIntRec-1 (M-1)** as a central case study. The paper analyzed MIntRec-1 in three stages: initial model performance, modality-importance analysis, and debiasing [2508.16122].

The central empirical result was that a **text-only LLM**, **Mistral-7B**, performed best on the original MIntRec-1 benchmark with **82.9%** accuracy. This was about **9 percentage points higher** than competitive multimodal models such as **SDIF: 72.8%**, **MulT: 71.5%**, **MAG: 72.7%**, and **MISA: 71.8%**. The same study also reported that text-only LLMs outperformed multimodal LLMs by an even larger margin: **GPT-4T: 60.4%**, **GPT-4V: 59.5%**, **ClaudeT: 57.7%**, and **ClaudeV: 59.1%** [2508.16122].

To quantify modality relevance, the study performed masking over the seven modality combinations \(T\), \(V\), \(A\), \(T+V\), \(T+A\), \(V+A\), and \(T+V+A\), and defined
\[
\Sigma_T = T \cup (T+V) \cup (T+A) \cup (T+V+A).
\]
On MIntRec-1, the distribution of minimal modality requirements was reported as **Text only (T): 69.66%**, **Video only (V): 3.60%**, **Audio only (A): 4.72%**, **T+V: 1.35%**, **T+A: 3.37%**, **V+A: 0.67%**, and **T+V+A: 16.63%**. Crucially, **\(\Sigma_T = 91.01\%\)**, meaning that about **91%** of MIntRec-1 samples require text for correct classification, either alone or in combination with other modalities [2508.16122].

Human evaluation confirmed this pattern. Manual annotation of a subset of MIntRec-1 yielded **Text-only: 82.46%**, **Text+Video: 5.69%**, **Text+Video+Audio: 5.01%**, **Text+Audio: 4.56%**, and **Video-only: 2.28%**, with **85.7% inter-annotator agreement**. The paper also identified particularly text-biased intent classes, including **Agree**, **Apologise**, **Greet**, **Praise**, **Thank**, and **Care**, and attributed their recoverability to explicit lexical cues such as “sorry,” “thank you,” “okay,” and “yeah” [2508.16122].

The paper distinguished **explicit textual bias** from **implicit textual bias**. Explicit bias refers to direct lexical signals such as “I am sorry” \(\rightarrow\) **Apologise**, “Thank you” \(\rightarrow\) **Thank**, and “Okay”, “yeah”, “right” \(\rightarrow\) **Agree**. Implicit bias refers to samples in which intent is not explicitly stated but can still be inferred from wording or context. In the automated framework, the minimal sufficient modality subset for each test sample was chosen as the smallest combination that either correctly predicts the label or gives the highest probability for the ground-truth label [2508.16122].

The debiasing framework identified textually biased samples using three models: **BERT** as a text-only model, **LLaMA-2-7B** as a text-only LLM, and **SDIF** as a small multimodal model with masked audio/video. A sample was labeled textually biased if it was correctly classified by the **majority** of these three models. Because the dataset is relatively small, the authors used **5 different splits**, dividing the training set into three parts and applying a round-robin scheme across runs [2508.16122].

After filtering, about **70%** of MIntRec-1 samples were removed. Heavily biased intent categories such as **Agree**, **Apologise**, **Care**, **Greet**, **Praise**, and **Thank** became too small after filtering and were excluded from the final debiased version. The final debiased MIntRec-1 contained **290 train**, **107 dev**, and **108 test** samples and **14 intent categories** instead of 20. The paper reported a **74.69% total reduction** in the bias-heavy label groupings shown in the debiasing statistics [2508.16122].

Debiasing caused a substantial performance collapse across model families. On MIntRec-1, **M-7B** dropped from **82.9%** to **32.4%**, **BERT** from **70.8%** to **16.7%**, **SDIF** from **72.8%** to **22.2%**, and **GPT-4V** from **59.5%** to **52.9%**. The paper summarized this as **textual models drop by more than 50%** and **smaller multimodal models drop by 50–60% or more**. A random-subset control showed that the debiased set performs much worse than a random subset of equal size, indicating that the degradation is due to removing textually easy samples rather than merely reducing dataset size [2508.16122].

This reassessment directly challenged the interpretation of the original benchmark. The paper concluded that **MIntRec-1 is not a reliable benchmark for measuring true multimodal understanding in its original form**, because **text dominates the dataset**, **a text-only LLM can outperform multimodal systems**, **most samples do not require video/audio**, **reported multimodal gains are inflated by text shortcuts**, and once bias is removed **all models suffer large drops** [2508.16122].

## 6. Subsequent methodological developments and evolving uses

Despite the critique of modality bias, MIntRec-1 has remained a widely used benchmark for developing multimodal architectures. One line of work emphasizes richer nonverbal modeling. **WDMIR** introduces a **Wavelet-Driven Multimodal Intent Recognition** framework with a **Wavelet-driven Fusion Module**, **Collaborative Representation Module**, and **Progressive Fusion Module**. On MIntRec it reports **75.06 ACC**, **74.96 WF1**, **75.26 WP**, and **72.65 R**, improving over **SDIF-DA*** by **+1.13 ACC**, **+1.07 WF1**, **+1.08 WP**, and **+0.99 R**. Its main claim is that frequency-domain analysis of audio-video signals improves the extraction of subtle nonverbal semantics [2506.10011].

A second line moves from conventional fusion to token-level modulation. **DyKen-Hyena** is a BERT-based model that inserts multimodal fusion **before** the main BERT encoder and uses cross-modal attention to generate **dynamic, per-token convolutional kernels** that modulate textual feature extraction through a **Hyena** backbone. On the closed-set **MIntRec** benchmark corresponding to MIntRec-1, it reports **Accuracy 73.66**, **F1 69.26**, **Precision 70.30**, **Recall 69.54**, **Weighted F1 73.05**, and **Weighted Precision 73.30**, with gains of **+1.84 Acc**, **+0.79 F1**, and **+1.47 WF** over the best baseline in that paper. On **MIntRec2.0**, it reports **F1-OOS 38.69**, an improvement of **+10.46 absolute F1 points** over the best baseline for out-of-scope detection [2509.09940].

Another direction strengthens class-level semantic grounding. **MVCL-DAF++** extends MVCL-DAF with **Prototype-aware contrastive alignment** and **Coarse-to-fine dynamic attention fusion**. On **MIntRec 1.0 / MIntRec-1**, it reports **ACC 76.18**, **WF1 75.66**, **WP 76.17**, and **R 74.39**, outperforming **MVCL-DAF** by **+1.46 ACC**, **+1.05 WF1**, **+1.10 WP**, and **+2.45 Recall**. The paper also highlights a **+1.05% WF1 improvement in rare-class recognition** on MIntRec, and attributes the gain to class prototypes as semantic anchors plus hierarchical fusion of coarse global and fine token-level signals [2509.17446].

MIntRec has also been used to study incomplete observation. **ADMC**, an **Attention-based Diffusion model for Missing Modalities feature Completion**, decouples modality-specific feature extraction from completion and fusion. Under missing-modality evaluation on MIntRec, ADMC reports **33.4 / 29.3** for \(\{A\}\), **15.4 / 10.2** for \(\{V\}\), **71.6 / 65.7** for \(\{T\}\), **34.3 / 29.6** for \(\{AV\}\), **72.3 / 65.8** for \(\{AT\}\), **71.8 / 65.1** for \(\{VT\}\), and **49.8 / 44.3** average **WA / UA**. In the full-modality enhancement setting, it reports **72.5 WA / 65.4 UA** for **MMER**, compared with **71.2 / 65.0** for jointly training and **71.5 / 65.0** for using ITFN alone [2507.05624].

A further line addresses semantic conflict rather than missing data. **CDPR**, a **Cognitive Dual-Pathway Reasoning** framework, separates a consensus-driven **intuition pathway** from a conflict-resolving **reasoning pathway**, with representation disentanglement and an inconsistency perception mechanism based on semantic prototype matching and statistical calibration. On **MIntRec**, it reports **ACC 75.15**, **F1 71.04**, **P 72.01**, **R 71.08**, **WF1 74.91**, and **WP 75.37**. In hard categories it reports **Taunt: 33.33 F1** and **Oppose: 47.06 F1**, and under Gaussian noise added to text it remains the strongest method reported in that study, with **55.65** F1 at \(\sigma = 0.3\) and **22.68** F1 at \(\sigma = 0.7\) [2605.09468].

Taken together, these works show that MIntRec-1 has served several distinct research functions: as a benchmark for multimodal fusion, as a stress test for nonverbal modeling, as a platform for prototype-guided contrastive learning, as an evaluation bed for missing-modality completion, and as a test case for inconsistency-aware reasoning. At the same time, the modality-bias analysis indicates that strong results on the original benchmark do not by themselves establish robust multimodal understanding [2508.16122]. This suggests that MIntRec-1 is best understood both as a historically influential multimodal intent dataset and as a benchmark whose evaluation meaning depends strongly on whether textual shortcuts are controlled.

Source: https://www.emergentmind.com/topics/mintrec-1