Learning Robust Pair Confidence for Multimodal Emotion-Cause Pair Extraction

Abstract: Multimodal emotion-cause pair extraction (MECPE) requires reliable pair confidence over candidate pairs. Existing pair scorers commonly use pair-level cross entropy over valid candidates, which treats links mostly independently. This leaves the relative confidence geometry among competing causes under-constrained, allowing gold pairs to stay close to hard negatives or rely on incidental non-gold context. We study this vulnerability as pair-confidence brittleness and propose RPCL (Robust Pair Confidence Learning), a training-only framework for pair-confidence learning. RPCL encourages pair confidence to be both discriminative and stable: gold pairs are separated from row-wise hard negatives through a confidence-difference margin constraint, and clean pair predictions are aligned with predictions from a corrupted view where non-gold contextual utterance representations are partially corrupted. The original clean pair scorer and decoding pipeline are used unchanged at inference time. On ECF, MECAD, and MEC4, RPCL improves the three-seed mean Pair F1 over a matched base model by 2.58 to 2.83 percentage points in the full text-audio-video setting, and improves mean Pair AUPRC on all three datasets. Diagnostic analysis further shows larger gold-negative confidence gaps and lower margin-violation severity. These results suggest that explicitly shaping pair confidence is an effective training strategy for MECPE.

• The paper introduces Robust Pair Confidence Learning (RPCL), which combines adaptive row-wise margin ranking with corrupted-context consistency to improve pair-confidence geometry without changing inference.
• RPCL raises three-seed mean Pair F1 by 2.58–2.83 percentage points and Pair AUPRC by 1.63–2.62 points over matched baselines in full text-audio-video settings across ECF, MECAD, and MEC⁴.
• The method increases gold-minus-hard-negative confidence gaps, remains effective across modality configurations, and shows that both hard-negative mining and protected context corruption contribute to performance gains.

Overview

This paper addresses a training-level weakness in multimodal emotion-cause pair extraction (MECPE): pair-confidence brittleness. Standard MECPE systems supervise candidate emotion-cause pairs with pair-level cross entropy, which treats each candidate largely independently. The authors argue that this leaves the relative confidence geometry among competing causes for the same emotion utterance under-constrained, allowing gold pairs to remain close to hard negatives or to depend on incidental non-gold context. They propose RPCL (Robust Pair Confidence Learning), a training-only framework that adds two constraints to a standard pair-scoring backbone while leaving the inference pipeline entirely unchanged. On ECF, MECAD, and MEC$^4$, RPCL improves the three-seed mean Pair F1 over a matched base model by 2.58–2.83 percentage points in the full text-audio-video (TAV) setting and improves Pair AUPRC on all three datasets (2606.18893).

Motivation and problem formulation

MECPE requires, for each dialogue, identifying which utterances express emotions and which utterances cause them, forming emotion-cause pairs. The task is structured: multiple candidate causes can be locally plausible for a single emotion utterance, while only a small annotated subset is correct. Existing supervision via cross entropy over valid candidates evaluates each candidate only through its own binary label and does not enforce that a gold cause outrank the strongest non-gold alternatives within the same emotion row.

The paper builds on a generic backbone that produces multimodal utterance representations $h_t$, emotion and cause logits, and pair logits $s_{ij}$ for valid candidates, from which the positive-pair confidence $p_{ij}$ is derived via softmax. The pair scorer is treated as a black-box module, and both proposed constraints act on it without architectural modification.

The RPCL method

RPCL combines two training-time constraints.

Row-conditioned margin ranking (CDMR): For each emotion row $i$ containing both gold and non-gold candidates, the method mines the top-$k$ hard negatives by current pair confidence (with no gradient through the discrete selection) and enforces a hinge constraint that the gold-minus-negative confidence difference exceeds a margin. The margin is adaptive: it scales exponentially with the stop-gradiented contrast between the cause classifier's confidence on the negative and the gold cause, so that more cause-like negatives demand larger separation. This focuses pressure precisely where confidence is most likely to be brittle.

Corrupted-context pair stability (CCPS): A corrupted dialogue view is constructed by zeroing non-gold utterance representations (those not appearing in any annotated pair) with Bernoulli probability $\rho$, while protecting gold utterances. Because gold-pair evidence is preserved, the original labels remain valid for the corrupted view. The corrupted branch is trained with pair cross entropy plus an $L_2$ alignment to the clean pair distributions, where the clean prediction is the stop-gradiented reference. This discourages reliance on incidental non-gold context without pulling the clean branch toward noisier predictions.

The full objective adds both terms to the standard supervised loss (emotion CE + cause CE + pair CE). A single hyperparameter setting ($m_0=0.05$, $k=8$, $h_t$0, $h_t$1, $h_t$2, $h_t$3) is used across all datasets, modality settings, and seeds. At inference, the clean scorer and original thresholding/decoding pipeline are used unchanged, so any gain is attributable to the training objective rather than decoding or operating-point changes.

Main results

In the TAV setting, RPCL yields consistent gains over the matched base model:

Dataset	Base F1	RPCL F1	Δ F1	Base AUPRC	RPCL AUPRC	Δ AUPRC
ECF	55.71	58.29	+2.58	54.83	56.46	+1.63
MECAD	49.90	52.49	+2.59	46.05	48.28	+2.23
MEC$h_t$4	35.85	38.68	+2.83	28.02	30.64	+2.62

The largest gains occur on MEC$h_t$5, where the base model is weakest, suggesting the confidence constraints are most useful in the hardest evaluated setting. Against two conventional objective controls — fixed-margin ranking and utterance-dropout consistency — RPCL achieves the best Pair F1 on all three datasets and the best AUPRC on MECAD and MEC$h_t$6; fixed-margin ranking is the sole exception on ECF AUPRC, indicating fixed margins can sharpen ranking but RPCL gives stronger balanced extraction. The authors conclude that adaptive row-wise separation and protected corrupted-context stability are complementary rather than reducible to either standard objective.

The published-system comparison is presented with an explicit caveat: compared systems differ in architecture, modalities, features, and protocols. Notably, on MECAD, RPCL attains 52.49 Pair F1 versus 50.27 for M$h_t$7HG, but with a markedly different precision-recall profile (precision 40.95, recall 73.09), and on MEC$h_t$8 the M$h_t$9F system (44.79 F1) remains stronger under a different architecture. The paper therefore treats this comparison as literature context rather than an isolated component test.

Diagnostic and ablation analysis

Confidence diagnostics align with the proposed mechanism: RPCL increases the mean gold-minus-negative pair-probability gap by 4.72, 1.69, and 3.46 percentage points on ECF, MECAD, and MEC$s_{ij}$0 respectively. Gold-pair confidence rises on all datasets while hard-negative and all-candidate margin-violation severity decreases. Precision-recall movement differs by dataset — ECF and MEC$s_{ij}$1 mainly gain recall, MECAD mainly gains precision — indicating the method does not simply bias predictions toward more positives.

Across modality configurations (T, T+A, T+V, T+A+V), matched RPCL-Base gains are positive for both Pair F1 and Pair AUPRC, showing the objective is not tied to complete multimodal evidence. Ablations confirm both components contribute: CDMR alone and CCPS alone each improve over Base on all datasets, and partial removals (dropping the adaptive margin, top-$s_{ij}$2 selection, consistency alignment, or corrupted-view supervision) each weaken performance, with the full objective best on all reported metrics.

Limitations

The paper concedes three limitations. First, RPCL is a training objective for pair-scoring backbones, not a new encoder or decoder, and may be complementary to stronger architectures — the M$s_{ij}$3F result on MEC$s_{ij}$4 shows architecture can still dominate. Second, the corrupted-context constraint operates at the representation level and does not cover realistic noise such as ASR errors, missing visual frames, domain shift, or cultural variation. Third, predictions reflect annotated emotion-cause links only, not underlying internal causes of emotion, and should not ground high-stakes decisions. An additional open question is whether the gains transfer to backbones whose pair scorers differ structurally from the generic interface assumed here.

Conclusion

The paper reframes MECPE training as reliable pair-confidence learning, identifying brittleness in independent candidate classification and addressing it with row-conditioned margin ranking and corrupted-context stability. The consistent F1 and AUPRC gains, unchanged inference pipeline, and supporting confidence diagnostics indicate that explicitly shaping the pair-confidence surface is an effective, lightweight training strategy for multimodal ECPE, distinct from representation- or decoder-side improvements.