E-THER: Empathic Therapy Benchmark

• E-THER is a multimodal, Person-Centered Therapy-based dataset that benchmarks empathic AI by detecting mismatches between verbal expressions and visual cues.
• The dataset comprises 18 authentic therapeutic sessions with dense annotations across video, audio, and text to capture nuanced emotional and engagement metrics.
• Its training methodology, including context dropout and incongruence-aware loss weighting, enhances model sensitivity to latent affect beyond surface-level emotion recognition.

E-THER, expanded in the paper as “Empathic THERapy Conversations,” is a multimodal dataset for benchmarking empathic AI that is grounded in Person-Centered Therapy (PCT) and explicitly organized around verbal-visual incongruence detection in client-counsellor interactions. It is presented as the first PCT-grounded multimodal dataset with multidimensional annotations for verbal-visual incongruence detection, addressing a limitation of prior empathic AI resources that emphasize surface-level emotion recognition without systematically modeling cases in which verbal expressions do not align with underlying emotional states (Tahir et al., 2 Sep 2025).

1. Conceptual foundation

E-THER is built on a PCT framework associated in the paper with Carl Rogers’ emphasis on empathy, unconditional positive regard, and congruence. Within this framing, “congruence” denotes consistency between communicated and underlying affective state, while “verbal-visual incongruence” refers to cases in which spoken content and facially expressed emotion diverge. The dataset uses PCT not as a prescriptive therapy protocol, but as a framework for annotation, response evaluation, and benchmarking authentic empathic communication (Tahir et al., 2 Sep 2025).

The central research motivation is that current empathic AI systems often fail when verbal reports understate, mask, or conflict with visually expressed affect. E-THER therefore treats incongruence not as annotation noise but as a clinically meaningful signal. In the paper’s formulation, this enables training and evaluation of systems that aim at “genuine rather than performative empathic capabilities,” rather than merely detecting explicit emotion words or producing formulaic empathic language (Tahir et al., 2 Sep 2025).

This orientation differentiates E-THER from datasets centered on surface emotion recognition or text-only conversational empathy. A plausible implication is that the benchmark is intended not only to score sentiment recognition, but to test whether multimodal systems can remain sensitive to latent affective tension under therapeutic constraints.

2. Corpus composition and data structure

The corpus consists of authentic therapeutic material rather than crowdsourced or acted dialogue. It contains 18 actual therapeutic sessions conducted by a registered clinical psychologist with 18 participants, spanning ages 19–72 and 6 ethnicities. The modalities are synchronized video, audio, and textual transcripts, and each dialogue turn is paired with the video frame at speech onset to preserve the visual context associated with the utterance (Tahir et al., 2 Sep 2025).

At corpus scale, the dataset comprises 1,578 utterances, 789 turn pairs, and approximately 5 hours of data. Annotation is dense rather than sparse: each dialogue pair carries 5 annotated dimensions, yielding 3,945 annotations in total.

Aspect	Specification	Note
Sessions	18 actual therapeutic sessions	not crowdsourced, not acted
Participants	18 participants	age 19–72; 6 ethnicities
Modalities	video, audio, textual transcripts	synchronized; frame at speech onset
Scale	1,578 utterances; 789 turn pairs; ~5 hours	3,945 annotations
Annotation format	5 annotated dimensions per dialogue pair	789 pairs × 5 dimensions

The paper contrasts this design with resources such as EmpatheticDialogues, ESConv, MEDIC, MESC, and MELD, arguing that E-THER is unique in combining multimodality, therapeutic grounding, systematic incongruence annotation, and clinical authenticity. In the comparison reported there, E-THER is also characterized by high annotation intensity relative to dataset size, reflecting an explicit quality-over-quantity design (Tahir et al., 2 Sep 2025).

3. Annotation scheme and reliability

The annotation framework has three core components: verbal-visual incongruence class, engagement level, and emotional state represented in valence-arousal-dominance (VAD) space. The incongruence taxonomy is mutually exclusive and contains three classes: no incongruence, minimizing incongruence, and contradiction incongruence. “No incongruence” denotes alignment between verbal and visual emotion. “Minimizing incongruence” denotes downplaying of distress in speech while facial cues indicate stronger emotion. “Contradiction incongruence” denotes direct conflict between facial expression and verbal content, such as smiling while discussing trauma (Tahir et al., 2 Sep 2025).

The paper reports that incongruent instances comprise approximately 20% of annotated dialogue pairs. Engagement is annotated on a continuous 0–1 scale, subdivided into 0–0.3 low, 0.4–0.7 moderate, and 0.8–1 high. Emotional state is encoded with VAD labels, which the paper describes as supporting compatibility with other emotion models while quantifying emotion in both modalities (Tahir et al., 2 Sep 2025).

Quality assurance is organized as expert, theory-informed annotation rather than crowd labeling. Three expert annotators, trained on post-clinical literature, performed the annotations. Consensus was then confirmed by a certified clinical psychologist, who reviewed 12.7% of the corpus. The reported agreement was 83%, with Cohen’s $\kappa \approx 0.74$–$0.76$ (Tahir et al., 2 Sep 2025).

The annotation design is significant because it elevates misalignment itself to a first-class supervised target. In effect, the benchmark operationalizes a therapeutic notion of incongruence in a form that can be used for both discriminative analysis and generative model evaluation.

4. Training methodology and model adaptation

E-THER is used to fine-tune open-source vision-LLMs, specifically IDEFICS2-8B, VideoLLaVA-7B, and BLIP2, with LoRA fine-tuning for efficiency. The training procedure described in the paper comprises three main elements: self-supervised emotion understanding, context dropout, and incongruence-aware loss weighting (Tahir et al., 2 Sep 2025).

In self-supervised emotion understanding, VAD labels are masked so that the model must infer emotion across modalities. In context dropout, explicit emotion in the context text is randomly obscured, forcing the model to rely on subtle, non-explicit cues. The third component, incongruence-aware loss weighting, increases emphasis on difficult samples in which visual and textual signals diverge.

The weighting scheme is given as

$w_i = 1 + s_i^{\,\gamma}, \quad s_i \in [0,1],$

with composite score

$$s_i = \mathrm{clip}\left( \frac{\|\hat{\mathbf{e}}^{(v)}_i - \hat{\mathbf{e}}^{(t)}_i\|_2}{\tau_e} + \lambda \big(1 - \langle \hat{\mathbf{z}}^{(v)}_i, \hat{\mathbf{z}}^{(t)}_i\rangle\big), 0, 1 \right),$$

and batch loss

$$\mathcal{L}(\theta) = \frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} w_i\,\ell_i.$$

Here, $\hat{\mathbf{e}}^{(v)}_i$ and $\hat{\mathbf{e}}^{(t)}_i$ are VAD-predicted vectors for visual and textual modalities, $\hat{\mathbf{z}}^{(v)}_i$ and $\hat{\mathbf{z}}^{(t)}_i$ are normalized embeddings, $\tau_e$ is the batch median VAD mismatch, and $0.76$0 is a balancing parameter. The paper states that higher $0.76$1, up to 2, is assigned to high-incongruence samples (Tahir et al., 2 Sep 2025).

Methodologically, this makes E-THER more than a static benchmark. It is also a training resource that induces cross-modal attunement by penalizing failures on misaligned cases more heavily than failures on verbally explicit ones.

5. Evaluation protocol and reported empirical results

The evaluation framework is aligned with therapeutic and empathic criteria rather than relying exclusively on semantic overlap. The principal metrics are Empathic Authenticity, Responsive Engagement, Therapeutic Concision, and PCT Adherence. BERTScore is also used for semantic validation, but not as the primary criterion of empathic quality (Tahir et al., 2 Sep 2025).

The paper reports that E-THER-trained models outperform GPT-4V and baseline VLMs on the empathy-aligned metrics. For Empathic Authenticity, VideoLLaVA is reported at 91.8 versus 72.0 for GPT-4V, and IDEFICS2 at 87.3 versus 72.0. For Therapeutic Concision and PCT Adherence, the reported improvement is approximately 5–8% in absolute terms. These results are reported with $0.76$2-values $0.76$3 and Cohen’s $0.76$4 up to 1.01 (Tahir et al., 2 Sep 2025).

The paper also emphasizes a divergence between semantic adequacy and therapeutic adequacy. BLIP2 reportedly attains a BERTScore of 0.81 while still producing superficial, repetitive responses, and the authors therefore argue that BERTScore alone fails to capture genuine empathy differences (Tahir et al., 2 Sep 2025).

Ablation findings further specify the role of the dataset’s design. Removing the visual modality caused the largest drop in PCT Adherence and Therapeutic Concision, by 8–9%, which the paper interprets as evidence that multimodality is vital for genuine empathy. Removing incongruence weighting degraded Responsive Engagement and overall effectiveness, and qualitative analysis indicated reversion to more generic, less context-sensitive responses. Heatmaps of information travel additionally showed increased cross-modal integration in incongruence-weighted training, especially in fusion layers (Tahir et al., 2 Sep 2025).

6. Research significance, scope, and common misunderstandings

E-THER is positioned as a benchmark for multimodal empathic reasoning under therapeutic conditions, not as a generic emotion dataset and not as a substitute for psychotherapy. One recurrent misunderstanding addressed in the paper is the assumption that empathic competence can be measured from text alone. E-THER is explicitly constructed against that assumption: it makes verbal-visual mismatch central, and its ablation results attribute substantial performance losses to removal of the visual stream (Tahir et al., 2 Sep 2025).

A second misunderstanding is that therapeutic response quality can be adequately measured by semantic similarity. The reported BLIP2 result shows why the paper rejects that premise: high BERTScore can coexist with superficial or repetitive therapeutic language. In this benchmark, the relevant target is not lexical resemblance to references, but authentic, context-sensitive, therapeutically bounded response behavior (Tahir et al., 2 Sep 2025).

A third point concerns dataset scale. Relative to large conversational corpora, E-THER is modest in duration and number of turns, but the paper explicitly frames it as a dense, clinically authentic, quality-over-quantity resource. This suggests that the intended contribution is not breadth of everyday dialogue coverage, but depth of multimodal and theory-grounded annotation.

Within empathic AI research, E-THER therefore serves three functions simultaneously: a clinically grounded corpus, an annotation schema for verbal-visual incongruence, and an evaluation framework that operationalizes PCT-aligned response quality. Its main contribution is not merely the presence of video or therapy transcripts in isolation, but the integration of therapeutic theory, authentic interactional data, and multimodal supervision into a single benchmark for systems intended to model empathy beyond surface affect recognition (Tahir et al., 2 Sep 2025).