---
title: 'M3HG: Graph-Based Multimodal Emotion-Cause Extraction'
url: https://www.emergentmind.com/topics/m3hg
type: topic
---

# M3HG: Graph-Based Multimodal Emotion-Cause Extraction

M\textsuperscript{3}HG is an end-to-end framework for **Multimodal Emotion Cause Triplet Extraction in Conversations (MECTEC)** that represents a conversation as a **multimodal, multi-scale, and multi-type node heterogeneous graph** and jointly predicts emotion utterances, cause utterances, and emotion categories [2508.18740]. In the formulation considered by the paper, a conversation is given as
\[
C=\{(S_i,\mathbf U_i)\}_{i=1}^n,\quad \mathbf U_i=\{U_i^t,U_i^a,U_i^v\},
\]
where each utterance contains text, audio, and video. The target is to extract triplets
\[
\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,
\]
where \(\mathbf U_j^e\) is an emotion utterance, \(\mathbf U_j^c\) is its cause utterance, and \(y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}\). The model is introduced together with MECAD, a new dataset intended to address the scarcity and limited scenario diversity of prior public MECTEC resources [2508.18740].

## 1. Problem formulation and task-specific difficulties

MECTEC extends earlier emotion-cause analysis tasks from text-only conversations to **multimodal conversations** with **text, audio, and video** [2508.18740]. The task is more difficult than ECAC and ECPEC for three reasons stated in the paper. First, **multiple modalities** contribute to emotional expression and causality. Second, **multi-scale semantics** matter, because relevant cues occur both within individual utterances and across interactions among utterances. Third, **cause utterances may occur after emotion utterances**, which requires modeling global conversation context rather than only historical context [2508.18740].

The paper frames prior MECTEC systems as limited in three corresponding ways. They do not explicitly isolate or model emotional and causal contexts; they do not handle future-cause cases well because they rely heavily on historical context; and they do not adequately combine **intra-utterance semantics** with **inter-utterance semantics** [2508.18740]. The design of M\textsuperscript{3}HG follows directly from these stated deficiencies. This suggests that the model is not merely a multimodal encoder with a graph wrapper, but a task-structured architecture in which graph topology is intended to reflect emotional and causal reasoning.

A common simplification in conversational emotion-cause modeling is to assume that nearby past utterances are sufficient. The paper explicitly rejects that assumption for MECTEC by highlighting long-distance dependencies and future-cause cases [2508.18740]. A plausible implication is that methods optimized primarily for local sequential context may underperform when causal evidence is distributed globally across a dialogue.

## 2. MECAD and the data regime motivating M\textsuperscript{3}HG

Before this work, the only public MECTEC dataset was **ECF**, which is multimodal but highly constrained because all conversations come from the **Friends** TV series [2508.18740]. The paper introduces **MECAD** to address both data scarcity and limited dialogue diversity. MECAD is described as the **first Chinese multi-scenario MECTEC dataset**, built from **56 Chinese TV series**, and intended to provide broader dialogue contexts and better support for generalization [2508.18740].

MECAD includes annotations for emotion labels and cause utterances, and also for **cause types** such as event and expression, as well as the **modal basis** for the cause annotation, namely whether the cause is supported by text, audio, or video [2508.18740]. This gives the dataset a broader scope than triplet extraction alone and makes it relevant to multimodal emotion-cause analysis more generally.

| Statistic | MECAD value |
|---|---:|
| Conversations | 989 |
| Utterances | 10,516 |
| Emotion-cause pairs | 8,077 |
| Speakers | 626 |
| TV series | 56 |
| Train split | 684 convs, 7,516 utterances |
| Val split | 126 convs, 1,168 utterances |
| Test split | 179 convs, 1,832 utterances |
| Average conversation length | about 10.63 utterances |
| Average utterance length | about 18.33 tokens |
| Average relative position of emotion-cause pairs | about 0.69 |
| Annotation reliability | Fleiss’s Kappa = 0.6932 |

The average relative position of emotion-cause pairs is reported as about **0.69**, indicating that many causes are nearby and often in previous utterances, but that the dataset also contains long-distance and future-cause cases [2508.18740]. That property is central to the paper’s motivation for explicit whole-conversation propagation. The combination of multimodal evidence, scenario diversity, and nontrivial causal directionality is the empirical setting in which M\textsuperscript{3}HG is evaluated.

## 3. Unimodal feature extraction and heterogeneous graph construction

M\textsuperscript{3}HG has four main components: **unimodal feature extraction**, **graph construction**, **multi-scale semantic fusion**, and **emotion-cause classification** [2508.18740]. For each utterance, the model extracts text features using **SA-RoBERTa**, audio features using **Wav2Vec2**, and video features using **DenseNet**, producing
\[
\mathbf E^t \in \mathbb R^{n \times d_t},\quad \mathbf E^a \in \mathbb R^{n \times d_a},\quad \mathbf E^v \in \mathbb R^{n \times d_v}.
\]
Modality-specific local context encoding is then applied: text uses multi-head self-attention, while audio and video use GRU-based encoding with residual and normalization:
\[
\mathbf E'^m = LN(\mathbf E^m + GRU(\mathbf E^m)),
\]
\[
\mathbf H^m = LN(\mathbf E^m + \mathbf E'^m + FFN(\mathbf E'^m)), \quad m\in\{a,v\}.
\]
All three modalities are subsequently projected to the same hidden size \(d_h\), yielding \(\mathbf H'^t,\mathbf H'^a,\mathbf H'^v\) [2508.18740].

The conversation is represented as a heterogeneous graph
\[
\mathcal G=(\mathcal V,\mathcal E,\mathcal R),
\]
with four node types:
1. **Utterance super-nodes** \(SN^u\),
2. **Conversation super-node** \(SN^d\),
3. **Emotional context nodes** \(N^e\),
4. **Causal context nodes** \(N^c\) [2508.18740].

The graph is termed **multi-type** because these nodes serve different semantic roles, and **multi-scale** because it includes both **intra-utterance level** structure and **inter-utterance level** structure [2508.18740]. Each utterance super-node is
\[
SN_i^u=\{N_i^t,N_i^a,N_i^v\},
\]
initialized from the modality features, while the conversation super-node is
\[
SN^d=\{N_d^t,N_d^a,N_d^v\},
\]
initialized by averaging modal features to capture global context. The emotional and causal context nodes are initialized from textual representations and later updated with information from other modalities [2508.18740].

The graph uses five relation types,
\[
\mathcal R=\{r_{ss},r_{ds},r_{gc},r_{ec},r_{cc}\},
\]
corresponding to **same-speaker edge**, **different-speaker edge**, **global connection edge**, **emotion connection edge**, and **cause connection edge** [2508.18740]. For each utterance, the model connects it to previous \(K\) utterances, using \(r_{ss}\) when the speaker matches and \(r_{ds}\) otherwise. Each utterance super-node is also linked to the conversation super-node and to its emotional and causal context nodes. In compact form,
\[
\mathcal{V}=\{SN_i^u,N_i^e,N_i^c,SN^d\}_{1\le i\le n}.
\]

This graph design operationalizes the paper’s claim that emotional and causal evidence should be represented explicitly rather than remaining latent inside a single conversational embedding. A plausible implication is that node-type separation reduces interference between emotion-specific and cause-specific signals before joint decision making.

## 4. Multi-scale semantic fusion and prediction mechanism

The core computation in M\textsuperscript{3}HG is **multi-scale semantic fusion** implemented with a **Heterogeneous Graph Attention Network (HGAT)** over meta-paths [2508.18740]. The paper defines two fusion scopes: \(\Phi_{intra}\) for **intra-utterance fusion** and \(\Phi_{inter}\) for **inter-utterance fusion**.

Intra-utterance fusion combines information within one utterance across modalities and across the emotional and causal context nodes. The meta-path set includes modality-to-modality links and modality-to-emotion or modality-to-cause links:
\[
\Phi_{intra}=\{\phi(N^{m_1},N^{m_2},r_{m_1,m_2})\}\cup \{\phi(N^m,N^e,r_{m,e})\},
\]
with \(m_1,m_2,m\in\{t,a,v\}\) [2508.18740]. For a meta-path \(\phi\), node-level attention is defined as
\[
\alpha_{ij}^{\phi}= \frac{ \exp\left(\sigma(\mathbf a_\phi^T [\mathbf H'_i \parallel \mathbf H'_j])\right) }{ \sum_{k\in\mathcal N_i^\phi} \exp\left(\sigma(\mathbf a_\phi^T [\mathbf H'_i \parallel \mathbf H'_k])\right) },
\]
and aggregation as
\[
\mathbf Z_i=\sigma\left(\sum_{j\in\mathcal N_i^\phi}\alpha_{ij}^\phi \mathbf H'_j\right).
\]
This stage is intended to gather emotional and causal cues from all modalities explicitly [2508.18740].

Inter-utterance fusion connects utterance super-nodes to the conversation super-node through \(\Phi_{inter}\), enabling global context aggregation [2508.18740]. The paper identifies this as the key mechanism for handling cases in which the cause appears later in the dialogue, because every utterance can communicate through the global node. After meta-path-based aggregation, the model applies semantic attention to combine meta-path outputs and a **position-wise feed-forward network (PFFN)** after each iteration. The output is the updated emotion and cause node representations,
\[
\mathbf Z_i^e,\quad \mathbf Z_i^c.
\]

Prediction proceeds at three levels. For each utterance \(U_i\), two MLPs predict the emotion label \(\hat y_i^e\) and the cause indicator \(\hat y_i^c\) from \(\mathbf Z_i^e\) and \(\mathbf Z_i^c\), respectively [2508.18740]. For pair extraction, the model predicts whether utterance \(U_i\) is the cause of emotion utterance \(U_j\) by concatenating the emotion representation of \(U_j\), the cause representation of \(U_i\), and a relative position encoding \(RPE_{ij}\):
\[
\hat y_{ij}^{ec} = \sigma\left(MLP(\mathbf Z_j^e \parallel \mathbf Z_i^c \parallel RPE_{ij})\right).
\]
The relative position encoding is **RBF-kernel-based** and is used to model nonlinear positional relations [2508.18740]. A triplet is predicted when an utterance is classified as an emotion utterance, another as a cause utterance, the pair score is positive, and the emotion category is attached to the emotion utterance.

## 5. Training objective, evaluation protocol, and empirical performance

The model is trained jointly on three tasks—emotion prediction, cause utterance prediction, and emotion-cause pair prediction—using **Focal Loss** to address class imbalance [2508.18740]:
\[
\mathcal{L}^\beta = -\frac{1}{N^\beta} \sum_{i=1}^{N^\beta} \alpha^\beta (1-\hat y_i^\beta)^\gamma \log(\hat y_i^\beta), \quad \beta \in \{e,c,ec\}.
\]
Experiments are conducted on **ECF** and **MECAD**. Evaluation uses **F1** on utter-cause-emotion triplets, with two summary metrics: **6 Avg.**, the weighted average F1 across all six emotion categories, and **4 Avg.**, the weighted average F1 across the four main emotions excluding Disgust and Fear [2508.18740]. The paper also reports subtask metrics for **EP** (Emotion Extraction), **ER** (Emotion Recognition), **CE** (Cause Extraction), and **EC** (Emotion-Cause Pair Extraction).

The baselines listed in the paper are **MC-ECPE-2steps**, **HiLo**, **ECPE-2D**, **RankCP**, **UECA-Prompt**, **SHARK**, and **GPT-4o (5-shot)** [2508.18740]. Implementation settings include RoBERTa-large for ECF and Chinese RoBERTa for MECAD as text encoders, Wav2Vec2 for audio, DenseNet for video, a graph context window \(K=3\), AdamW, batch size 16, learning rate \(5\times 10^{-6}\), and 50 training epochs [2508.18740].

On **ECF**, with all three modalities, M\textsuperscript{3}HG achieves **6 Avg = 40.07** and **4 Avg = 41.96**, outperforming all baselines [2508.18740]. The paper reports gains over the previous best end-to-end model **HiLo** of **+21.28%** in 6 Avg and **+17.17%** in 4 Avg. Even in the text-only setting, M\textsuperscript{3}HG obtains **6 Avg = 37.46** and **4 Avg = 39.95**, with additional improvement from adding audio and video.

On **MECAD**, M\textsuperscript{3}HG also obtains the best reported results, with **6 Avg = 32.82** and **4 Avg = 34.59** [2508.18740]. These results surpass **SHARK**, the second-best model, by about **19%** on 6 Avg and **15.34%** on 4 Avg. The paper further states that **GPT-4o** performs noticeably worse than M\textsuperscript{3}HG on MECAD, indicating that few-shot LLM prompting is not yet competitive for this task under the reported setup.

## 6. Ablation evidence, behavior on difficult cases, and stated limitations

The ablation study is presented as strong support for the architectural choices in M\textsuperscript{3}HG [2508.18740]. Removing the emotional and causal context nodes \(N^e\) and \(N^c\) causes notable performance degradation, which the paper interprets as evidence that explicit modeling of emotional and causal contexts is important. Removing **intra-utterance fusion** causes a larger drop than removing **inter-utterance fusion**, indicating that within-utterance multimodal fusion is especially critical. Removing all modules produces the largest degradation.

On ECF, the reported values are:
- full model: **40.07 / 41.96**
- without all modules: **36.81 / 38.57**
- without \(N^e, N^c\): **38.13 / 40.11**
- without inter-fusion: **39.56 / 41.14**
- without intra-fusion: **39.12 / 40.86** [2508.18740]

The paper reports similar trends on MECAD. It also examines the effect of the graph context window \(K\), finding that performance improves as \(K\) increases up to **3** and then declines, making \(K=3\) the best setting [2508.18740]. This is consistent with the implementation choice used in the main experiments.

Additional analysis shows that M\textsuperscript{3}HG is especially effective for conversations with more than 10 utterances, where many baselines lose global context [2508.18740]. It also significantly outperforms **SHARK** and **GPT-4o** on cases where the cause appears after the emotion utterance, which the paper attributes to the conversation super-node. Across subtasks—EP, ER, CE, and EC—the model performs best, suggesting that the gains are not confined to the final triplet extraction layer.

The stated limitations are threefold: **no external knowledge integration**, **input length constraints**, and **possible multimodal conflict / error propagation** because modalities may provide inconsistent signals [2508.18740]. The proposed future directions are to integrate external knowledge, leverage LLMs for deeper semantic extraction, and improve robustness for long conversations and modality conflicts. These limitations are structurally aligned with the model’s scope: M\textsuperscript{3}HG improves multimodal graph reasoning within the observed conversation, but does not yet augment that reasoning with knowledge beyond the encoded inputs.

## 7. Position within multimodal emotion-cause analysis

Within the problem setting defined by the paper, M\textsuperscript{3}HG is best characterized as a **graph-based multimodal reasoning framework** that explicitly separates and then fuses **emotion context**, **cause context**, **utterance-level semantics**, **conversation-level semantics**, and **text/audio/video cues** [2508.18740]. Its central novelty is the combination of **multi-type nodes** with **multi-scale heterogeneous graph fusion**.

The model’s significance in the paper rests on two linked contributions. The first is methodological: an explicit graph design for joint reasoning over emotion and cause, rather than direct triplet prediction from undifferentiated conversational representations. The second is empirical: strong gains over the listed state-of-the-art baselines on both ECF and MECAD, together with evidence of improved behavior on long conversations and future-cause cases [2508.18740].

A plausible implication is that M\textsuperscript{3}HG establishes a template for MECTEC systems in which graph topology is used to encode task structure directly: utterance-level multimodal evidence, explicit emotion and cause nodes, and a global conversation node for nonlocal propagation. In the context of the paper, that architecture is presented as a response to the main unresolved issues in MECTEC—data scarcity, lack of scenario diversity, insufficient explicit emotion-cause modeling, and weak handling of global conversational dependencies [2508.18740].

Source: https://www.emergentmind.com/topics/m3hg