EWH Dataset: Emotion, Causality & World Modeling

• The EWH Dataset is a multimodal resource that fuses emotion, causality, and dynamic state prediction to model why actions occur and how they affect future states.
• It employs a fully automated LMM pipeline to annotate temporally grounded 5-tuple events, with quality control showing up to an 8% performance drop when emotional cues are omitted.
• EWH bridges gaps in traditional world models by operationalizing causal chains between perceptual, emotional, and action states, offering a robust testbed for social reasoning research.

The Emotion-Why-How (EWH) Dataset is a multimodal resource designed to incorporate emotion as a core factor within the causal reasoning process, enabling the study of both why actions occur (motivated by emotional and observable states) and how emotional transitions shape subsequent world states. EWH operationalizes theory-of-mind principles within an empirical benchmark, systematically linking perceptual state, emotional state, action, future emotion, and future state. It fills a critical gap in world modeling by fusing affect, causality, and dynamic prediction, thereby supporting robust evaluation of models’ ability to reason about the interplay between perception, emotion, and behavior (Song et al., 30 Dec 2025).

1. Conceptual Framework and Motivation

Traditional world models and video understanding benchmarks focus predominantly on physical and logical dynamics. However, in real-world social contexts, human behavior is deeply shaped by affective states—emotion influences both immediate actions and the evolution of subsequent observable states. The primary motivation behind EWH is to embed emotional state and causally linked transitions as first-class entities within a world modeling and causal reasoning corpus. This infrastructure is intended to allow systems not only to identify observable regularities but also to dissect the “emotional logic” underlying behavior—specifically, explaining why an action was performed, and how that action (in concert with emotion) affects the world state over time. A plausible implication is broader applicability of world models to domains requiring theory-of-mind capabilities or social reasoning (Song et al., 30 Dec 2025).

2. Data Structure and Sample Representation

Each entry in EWH is structured as a temporally grounded 5-tuple: $$\left((\mathcal{S}_0, \mathcal{E}_0),\,A,\;(\mathcal{S}_1, \mathcal{E}_1) \right)$$ where:

• $$\mathcal{S}_t = (\mathcal{V}_t, \mathcal{A}_t, \mathcal{I}_t)$$ denotes the multimodal snapshot (video clip $\mathcal{V}_t$, audio $\mathcal{A}_t$, key still frame $\mathcal{I}_t$) at time $t$,
• $\mathcal{E}_t$ is the inferred emotional state at time $t$,
• $A$ is a natural-language description of the emotion-driven action bridging $t=0$ to $t=1$.

The “Why–How” causal chain operationalizes two relations:

1. Why: $$((\mathcal{S}_0, \mathcal{E}_0)) \xrightarrow{\text{Why}} A$$ — the causal factor underpinning the action.
2. How: $$(\mathcal{E}_0, A) \xrightarrow{\text{How}} (\mathcal{E}_1 \xrightarrow{} \mathcal{S}_1)$$ — the transition of emotional and world state induced by the action (Song et al., 30 Dec 2025).

Emotion labels are not restricted to a fixed discrete set; instead, they are inferred via a Large Multimodal Model (LMM) that integrates semantic features and facial-expression analysis, supporting both canonical emotion categories (such as joy or anger) and more nuanced affective states.

3. Dataset Generation and Annotation Pipeline

EWH diverges from traditional manually annotated corpora, being constructed entirely via automated LMMs. The LMM pipeline performs the following:

• Identifies emotion-driven video, audio, and keyframe segments,
• Infers the current emotional state ($\mathcal{E}_0$) from visual and auditory content,
• Generates a natural-language action description $A$,
• Integrates $A$ and $\mathcal{E}_0$ to predict the new emotional state $\mathcal{E}_1$ and the resultant observable state $\mathcal{S}_1$.

No human annotators, labeling guidelines, or inter-annotator agreement metrics are referenced in the dataset's construction. Quality control is implemented through cross-validation with vision, audio, and facial-expression submodels to ensure the selected segments reflect genuinely emotionally salient events. This marks a significant methodological difference compared to manually curated resources such as CEAC (Liu et al., 2019).

4. Formal Definitions and Causal Modeling

Let $\mathcal{S}$ denote the space of multimodal observations, $\mathcal{E}$ the space of emotional states, and $\mathcal{A}$ the space of action descriptions. Each element is structured as

$$s \;=\; \left( (\mathcal{S}_0, \mathcal{E}_0),\,A,\,(\mathcal{S}_1, \mathcal{E}_1) \right)$$

The causal structure can be summarized as: $$(\mathcal{S}_0, \mathcal{E}_0) \xrightarrow{\text{Why}} A \xrightarrow{\text{Emotion Transition}} \mathcal{E}_1 \xrightarrow{\text{How}} \mathcal{S}_1$$ The conditional factorization in the associated Large Emotional World Model (LEWM) is: $$p(\mathcal{S}_{t+1},\,\mathcal{E}_{t+1}\,|\,\mathcal{S}_t,\,\mathcal{E}_t,\,A_t) = p(\mathcal{E}_{t+1}\,|\,h_t) \cdot p(\mathcal{S}_{t+1}\,|\,h_t,\,\mathcal{E}_{t+1})$$ where $$h_t = [\mathrm{Enc}_S(\mathcal{S}_t) ; \mathrm{Enc}_A(A_t) ; \mathrm{Enc}_E(\mathcal{E}_t)]$$ (Song et al., 30 Dec 2025).

5. Tasks and Evaluation Metrics

EWH supports two primary inference tasks:

• Action inference (Why): Given $(\mathcal{S}_0, \mathcal{E}_0)$, predict $A$.
• Emotion evolution (How): Given $(\mathcal{S}_0, \mathcal{E}_0, A)$, predict $(\mathcal{E}_1, \mathcal{S}_1)$.

Model performance is assessed using:

• Accuracy and F1-score on emotion classification (aligned with standards such as MELD benchmark),
• Predictive and reconstruction accuracy for future states (e.g., pixel or feature error for $\mathcal{S}_1$).

Removal of affective information from EWH samples leads to an observed degradation (up to an 8% drop) in emotion classification accuracy, which demonstrates the utility of the emotional state for reasoning. LEWM trained on EWH maintains comparable performance on non-affective reasoning tasks (e.g., HellaSwag, MMLU) while improving prediction of emotion-driven behavior (Song et al., 30 Dec 2025).

6. Relation to Prior Datasets and Benchmarks

EWH differs fundamentally from preceding resources. Prior world modeling datasets (such as Sora and Daydreamer) focus on physical-object dynamics and continuity, ignoring affect as a generative factor. Text- and dialogue-based emotion corpora (e.g., MELD) restrict themselves to unimodal, typically conversational, cues and lack explicit causal-chaining between world state, emotional state, and action. EWH uniquely provides aligned sequences of $(\mathcal{S}_0, \mathcal{E}_0)$, $A$, $(\mathcal{E}_1, \mathcal{S}_1)$, and does so in a fully multimodal context (Song et al., 30 Dec 2025). This positions EWH as an integrated testbed for both causal and affective world reasoning.

7. Limitations and Plausible Directions

EWH’s construction is LMM-driven, with no reported manual annotation or inter-annotator reliability metrics. There is no disclosure of overall sample counts, emotion-class distributions, or explicit data splits. All annotations are inferred via the LMM’s semantic understanding and submodel validation, rather than human consensus. This suggests a need for future work to establish gold-standard benchmarks with controlled splits and human-validated quality assessment. A plausible implication is that, while highly scalable, LMM-based datasets may encode the biases and limitations of their underlying vision/LLMs and are subject to shifts as LMMs evolve (Song et al., 30 Dec 2025).