M3HG: Graph-Based Multimodal Emotion-Cause Extraction

• The paper introduces M3HG, an end-to-end graph-based framework that jointly extracts emotion utterances, cause utterances, and emotion categories in multimodal conversations.
• It employs a heterogeneous graph with multi-type nodes and multi-scale semantic fusion, integrating text, audio, and video modalities for robust analysis.
• Experiments on ECF and MECAD demonstrate significant performance gains, effectively addressing local context limitations and future-cause challenges.

M\textsuperscript{3}HG is an end-to-end framework for Multimodal Emotion Cause Triplet Extraction in Conversations (MECTEC) that represents a conversation as a multimodal, multi-scale, and multi-type node heterogeneous graph and jointly predicts emotion utterances, cause utterances, and emotion categories (Liang et al., 26 Aug 2025). In the formulation considered by the paper, a conversation is given as

$$C=\{(S_i,\mathbf U_i)\}_{i=1}^n,\quad \mathbf U_i=\{U_i^t,U_i^a,U_i^v\},$$

where each utterance contains text, audio, and video. The target is to extract triplets

$\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$

where $\mathbf U_j^e$ is an emotion utterance, $\mathbf U_j^c$ is its cause utterance, and $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$. The model is introduced together with MECAD, a new dataset intended to address the scarcity and limited scenario diversity of prior public MECTEC resources (Liang et al., 26 Aug 2025).

1. Problem formulation and task-specific difficulties

MECTEC extends earlier emotion-cause analysis tasks from text-only conversations to multimodal conversations with text, audio, and video (Liang et al., 26 Aug 2025). The task is more difficult than ECAC and ECPEC for three reasons stated in the paper. First, multiple modalities contribute to emotional expression and causality. Second, multi-scale semantics matter, because relevant cues occur both within individual utterances and across interactions among utterances. Third, cause utterances may occur after emotion utterances, which requires modeling global conversation context rather than only historical context (Liang et al., 26 Aug 2025).

The paper frames prior MECTEC systems as limited in three corresponding ways. They do not explicitly isolate or model emotional and causal contexts; they do not handle future-cause cases well because they rely heavily on historical context; and they do not adequately combine intra-utterance semantics with inter-utterance semantics (Liang et al., 26 Aug 2025). The design of M\textsuperscript{3}HG follows directly from these stated deficiencies. This suggests that the model is not merely a multimodal encoder with a graph wrapper, but a task-structured architecture in which graph topology is intended to reflect emotional and causal reasoning.

A common simplification in conversational emotion-cause modeling is to assume that nearby past utterances are sufficient. The paper explicitly rejects that assumption for MECTEC by highlighting long-distance dependencies and future-cause cases (Liang et al., 26 Aug 2025). A plausible implication is that methods optimized primarily for local sequential context may underperform when causal evidence is distributed globally across a dialogue.

2. MECAD and the data regime motivating M\textsuperscript{3}HG

Before this work, the only public MECTEC dataset was ECF, which is multimodal but highly constrained because all conversations come from the Friends TV series (Liang et al., 26 Aug 2025). The paper introduces MECAD to address both data scarcity and limited dialogue diversity. MECAD is described as the first Chinese multi-scenario MECTEC dataset, built from 56 Chinese TV series, and intended to provide broader dialogue contexts and better support for generalization (Liang et al., 26 Aug 2025).

MECAD includes annotations for emotion labels and cause utterances, and also for cause types such as event and expression, as well as the modal basis for the cause annotation, namely whether the cause is supported by text, audio, or video (Liang et al., 26 Aug 2025). This gives the dataset a broader scope than triplet extraction alone and makes it relevant to multimodal emotion-cause analysis more generally.

Statistic	MECAD value
Conversations	989
Utterances	10,516
Emotion-cause pairs	8,077
Speakers	626
TV series	56
Train split	684 convs, 7,516 utterances
Val split	126 convs, 1,168 utterances
Test split	179 convs, 1,832 utterances
Average conversation length	about 10.63 utterances
Average utterance length	about 18.33 tokens
Average relative position of emotion-cause pairs	about 0.69
Annotation reliability	Fleiss’s Kappa = 0.6932

The average relative position of emotion-cause pairs is reported as about 0.69, indicating that many causes are nearby and often in previous utterances, but that the dataset also contains long-distance and future-cause cases (Liang et al., 26 Aug 2025). That property is central to the paper’s motivation for explicit whole-conversation propagation. The combination of multimodal evidence, scenario diversity, and nontrivial causal directionality is the empirical setting in which M\textsuperscript{3}HG is evaluated.

3. Unimodal feature extraction and heterogeneous graph construction

M\textsuperscript{3}HG has four main components: unimodal feature extraction, graph construction, multi-scale semantic fusion, and emotion-cause classification (Liang et al., 26 Aug 2025). For each utterance, the model extracts text features using SA-RoBERTa, audio features using Wav2Vec2, and video features using DenseNet, producing

$$\mathbf E^t \in \mathbb R^{n \times d_t},\quad \mathbf E^a \in \mathbb R^{n \times d_a},\quad \mathbf E^v \in \mathbb R^{n \times d_v}.$$

Modality-specific local context encoding is then applied: text uses multi-head self-attention, while audio and video use GRU-based encoding with residual and normalization: $\mathbf E'^m = LN(\mathbf E^m + GRU(\mathbf E^m)),$

$$\mathbf H^m = LN(\mathbf E^m + \mathbf E'^m + FFN(\mathbf E'^m)), \quad m\in\{a,v\}.$$

All three modalities are subsequently projected to the same hidden size $d_h$, yielding $\mathbf H'^t,\mathbf H'^a,\mathbf H'^v$ (Liang et al., 26 Aug 2025).

The conversation is represented as a heterogeneous graph

$\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$0

with four node types:

1. Utterance super-nodes $\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$1,
2. Conversation super-node $\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$2,
3. Emotional context nodes $\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$3,
4. Causal context nodes $\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$4 (Liang et al., 26 Aug 2025).

The graph is termed multi-type because these nodes serve different semantic roles, and multi-scale because it includes both intra-utterance level structure and inter-utterance level structure (Liang et al., 26 Aug 2025). Each utterance super-node is

$\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$5

initialized from the modality features, while the conversation super-node is

$\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$6

initialized by averaging modal features to capture global context. The emotional and causal context nodes are initialized from textual representations and later updated with information from other modalities (Liang et al., 26 Aug 2025).

The graph uses five relation types,

$\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$7

corresponding to same-speaker edge, different-speaker edge, global connection edge, emotion connection edge, and cause connection edge (Liang et al., 26 Aug 2025). For each utterance, the model connects it to previous $\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$8 utterances, using $\langle \mathbf U_j^e,\mathbf U_j^c,y_j^e\rangle,$9 when the speaker matches and $\mathbf U_j^e$0 otherwise. Each utterance super-node is also linked to the conversation super-node and to its emotional and causal context nodes. In compact form,

$\mathbf U_j^e$1

This graph design operationalizes the paper’s claim that emotional and causal evidence should be represented explicitly rather than remaining latent inside a single conversational embedding. A plausible implication is that node-type separation reduces interference between emotion-specific and cause-specific signals before joint decision making.

4. Multi-scale semantic fusion and prediction mechanism

The core computation in M\textsuperscript{3}HG is multi-scale semantic fusion implemented with a Heterogeneous Graph Attention Network (HGAT) over meta-paths (Liang et al., 26 Aug 2025). The paper defines two fusion scopes: $\mathbf U_j^e$2 for intra-utterance fusion and $\mathbf U_j^e$3 for inter-utterance fusion.

Intra-utterance fusion combines information within one utterance across modalities and across the emotional and causal context nodes. The meta-path set includes modality-to-modality links and modality-to-emotion or modality-to-cause links: $\mathbf U_j^e$4 with $\mathbf U_j^e$5 (Liang et al., 26 Aug 2025). For a meta-path $\mathbf U_j^e$6, node-level attention is defined as

$\mathbf U_j^e$7

and aggregation as

$\mathbf U_j^e$8

This stage is intended to gather emotional and causal cues from all modalities explicitly (Liang et al., 26 Aug 2025).

Inter-utterance fusion connects utterance super-nodes to the conversation super-node through $\mathbf U_j^e$9, enabling global context aggregation (Liang et al., 26 Aug 2025). The paper identifies this as the key mechanism for handling cases in which the cause appears later in the dialogue, because every utterance can communicate through the global node. After meta-path-based aggregation, the model applies semantic attention to combine meta-path outputs and a position-wise feed-forward network (PFFN) after each iteration. The output is the updated emotion and cause node representations,

$\mathbf U_j^c$0

Prediction proceeds at three levels. For each utterance $\mathbf U_j^c$1, two MLPs predict the emotion label $\mathbf U_j^c$2 and the cause indicator $\mathbf U_j^c$3 from $\mathbf U_j^c$4 and $\mathbf U_j^c$5, respectively (Liang et al., 26 Aug 2025). For pair extraction, the model predicts whether utterance $\mathbf U_j^c$6 is the cause of emotion utterance $\mathbf U_j^c$7 by concatenating the emotion representation of $\mathbf U_j^c$8, the cause representation of $\mathbf U_j^c$9, and a relative position encoding $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$0: $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$1 The relative position encoding is RBF-kernel-based and is used to model nonlinear positional relations (Liang et al., 26 Aug 2025). A triplet is predicted when an utterance is classified as an emotion utterance, another as a cause utterance, the pair score is positive, and the emotion category is attached to the emotion utterance.

5. Training objective, evaluation protocol, and empirical performance

The model is trained jointly on three tasks—emotion prediction, cause utterance prediction, and emotion-cause pair prediction—using Focal Loss to address class imbalance (Liang et al., 26 Aug 2025): $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$2 Experiments are conducted on ECF and MECAD. Evaluation uses F1 on utter-cause-emotion triplets, with two summary metrics: 6 Avg., the weighted average F1 across all six emotion categories, and 4 Avg., the weighted average F1 across the four main emotions excluding Disgust and Fear (Liang et al., 26 Aug 2025). The paper also reports subtask metrics for EP (Emotion Extraction), ER (Emotion Recognition), CE (Cause Extraction), and EC (Emotion-Cause Pair Extraction).

The baselines listed in the paper are MC-ECPE-2steps, HiLo, ECPE-2D, RankCP, UECA-Prompt, SHARK, and GPT-4o (5-shot) (Liang et al., 26 Aug 2025). Implementation settings include RoBERTa-large for ECF and Chinese RoBERTa for MECAD as text encoders, Wav2Vec2 for audio, DenseNet for video, a graph context window $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$3, AdamW, batch size 16, learning rate $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$4, and 50 training epochs (Liang et al., 26 Aug 2025).

On ECF, with all three modalities, M\textsuperscript{3}HG achieves 6 Avg = 40.07 and 4 Avg = 41.96, outperforming all baselines (Liang et al., 26 Aug 2025). The paper reports gains over the previous best end-to-end model HiLo of +21.28% in 6 Avg and +17.17% in 4 Avg. Even in the text-only setting, M\textsuperscript{3}HG obtains 6 Avg = 37.46 and 4 Avg = 39.95, with additional improvement from adding audio and video.

On MECAD, M\textsuperscript{3}HG also obtains the best reported results, with 6 Avg = 32.82 and 4 Avg = 34.59 (Liang et al., 26 Aug 2025). These results surpass SHARK, the second-best model, by about 19% on 6 Avg and 15.34% on 4 Avg. The paper further states that GPT-4o performs noticeably worse than M\textsuperscript{3}HG on MECAD, indicating that few-shot LLM prompting is not yet competitive for this task under the reported setup.

6. Ablation evidence, behavior on difficult cases, and stated limitations

The ablation study is presented as strong support for the architectural choices in M\textsuperscript{3}HG (Liang et al., 26 Aug 2025). Removing the emotional and causal context nodes $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$5 and $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$6 causes notable performance degradation, which the paper interprets as evidence that explicit modeling of emotional and causal contexts is important. Removing intra-utterance fusion causes a larger drop than removing inter-utterance fusion, indicating that within-utterance multimodal fusion is especially critical. Removing all modules produces the largest degradation.

On ECF, the reported values are:

• full model: 40.07 / 41.96
• without all modules: 36.81 / 38.57
• without $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$7: 38.13 / 40.11
• without inter-fusion: 39.56 / 41.14
• without intra-fusion: 39.12 / 40.86 (Liang et al., 26 Aug 2025)

The paper reports similar trends on MECAD. It also examines the effect of the graph context window $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$8, finding that performance improves as $$y_j^e \in \{\text{Anger, Disgust, Fear, Joy, Sadness, Surprise}\}$$9 increases up to 3 and then declines, making $$\mathbf E^t \in \mathbb R^{n \times d_t},\quad \mathbf E^a \in \mathbb R^{n \times d_a},\quad \mathbf E^v \in \mathbb R^{n \times d_v}.$$0 the best setting (Liang et al., 26 Aug 2025). This is consistent with the implementation choice used in the main experiments.

Additional analysis shows that M\textsuperscript{3}HG is especially effective for conversations with more than 10 utterances, where many baselines lose global context (Liang et al., 26 Aug 2025). It also significantly outperforms SHARK and GPT-4o on cases where the cause appears after the emotion utterance, which the paper attributes to the conversation super-node. Across subtasks—EP, ER, CE, and EC—the model performs best, suggesting that the gains are not confined to the final triplet extraction layer.

The stated limitations are threefold: no external knowledge integration, input length constraints, and possible multimodal conflict / error propagation because modalities may provide inconsistent signals (Liang et al., 26 Aug 2025). The proposed future directions are to integrate external knowledge, leverage LLMs for deeper semantic extraction, and improve robustness for long conversations and modality conflicts. These limitations are structurally aligned with the model’s scope: M\textsuperscript{3}HG improves multimodal graph reasoning within the observed conversation, but does not yet augment that reasoning with knowledge beyond the encoded inputs.

7. Position within multimodal emotion-cause analysis

Within the problem setting defined by the paper, M\textsuperscript{3}HG is best characterized as a graph-based multimodal reasoning framework that explicitly separates and then fuses emotion context, cause context, utterance-level semantics, conversation-level semantics, and text/audio/video cues (Liang et al., 26 Aug 2025). Its central novelty is the combination of multi-type nodes with multi-scale heterogeneous graph fusion.

The model’s significance in the paper rests on two linked contributions. The first is methodological: an explicit graph design for joint reasoning over emotion and cause, rather than direct triplet prediction from undifferentiated conversational representations. The second is empirical: strong gains over the listed state-of-the-art baselines on both ECF and MECAD, together with evidence of improved behavior on long conversations and future-cause cases (Liang et al., 26 Aug 2025).

A plausible implication is that M\textsuperscript{3}HG establishes a template for MECTEC systems in which graph topology is used to encode task structure directly: utterance-level multimodal evidence, explicit emotion and cause nodes, and a global conversation node for nonlocal propagation. In the context of the paper, that architecture is presented as a response to the main unresolved issues in MECTEC—data scarcity, lack of scenario diversity, insufficient explicit emotion-cause modeling, and weak handling of global conversational dependencies (Liang et al., 26 Aug 2025).