Embodied LVLM Emotion Narratives (ELENA)

• ELENA is a framework that transforms embodied affect signals from images, skeletons, and robotics into structured, multi-layered emotion narratives.
• It integrates explicit bodily cues and trigger-centered reasoning to associate posture, gesture, and context with specific emotional states.
• Studies extend ELENA from static images to dynamic motion tokens, using contrastive learning and causal regulation to boost classification accuracy and narrative quality.

Searching arXiv for the cited ELENA-related papers to ground the article in current sources. Embodied LVLM Emotion Narratives (ELENA) denotes a family of methods in which embodied affective evidence is converted into structured language that names an emotion, localizes the bodily cues supporting it, and links those cues to internal sensations, contextual causes, or later actions. In the direct visual formulation, ELENA maps an input image $I$ to $(L, E_e, E_i, N, B)$, where $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$ is the predicted Ekman label, $E_e$ are explicit embodied descriptions focusing on visible body parts, $E_i$ are implicit descriptions of internal sensations or unobserved parts, $N$ is a coherent narrative weaving embodied cues with scene context, and $B$ is a list of identifiable body parts cited by the model (Saim et al., 23 Sep 2025). Across related work, this basic schema is extended from static images to 3D skeleton sequences, trigger-centered “why” explanations, episodic affective recall, and embodied action selection, so ELENA functions both as a concrete framework and as a broader design pattern for grounded affective reasoning (Lu et al., 2024).

1. Conceptual basis and representational scope

ELENA is grounded in a shift from flat label prediction to multi-layered affect interpretation. The visual ELENA framework explicitly targets posture, gesture, limb orientation, torso tension, and inferred somatovisceral sensations, particularly when facial cues are unavailable or unreliable, and uses a single, unified structured prompt to elicit all outputs in one forward pass (Saim et al., 23 Sep 2025). A complementary formalization comes from Emotion Interpretation, where the query space is $\mathcal{Q}=\mathcal{X}\times\mathcal{E}$, the generative map is $G:\mathcal{Q}\to\mathcal{P}(\mathcal{S})$, and the trigger set is $T=G(x,e)\subseteq\mathcal{S}$; triggers may be free-form sentences or concise tags and include $(L, E_e, E_i, N, B)$0, $(L, E_e, E_i, N, B)$1, $(L, E_e, E_i, N, B)$2, and $(L, E_e, E_i, N, B)$3, with explicit/implicit distinctions (Lin et al., 10 Apr 2025). Within this broader perspective, ELENA is not restricted to “what emotion is present” but also asks which bodily, environmental, social, and contextual factors justify the inference.

A second representational anchor is the body-sensation map. In the fMRI study on somatosensory and motor contributions to emotion representation, participants colored body silhouettes to indicate where they felt sensations elicited by images; those maps were downsampled to $(L, E_e, E_i, N, B)$4, smoothed via radial dilation, flattened, averaged per stimulus, and compared by correlation distance $(L, E_e, E_i, N, B)$5 with Fisher normalization (Reddan et al., 2024). Whole-brain searchlight RSA linked this geometry to bilateral primary somatosensory cortex, primary motor cortex, precuneus, insula, and medial prefrontal cortex. This suggests ELENA can be evaluated not only by lexical correctness but also by whether its internal embeddings and narratives preserve the representational geometry of embodied feeling.

Theoretical accounts of embodiment further widen the scope. “On the Computational Modeling of Meaning: Embodied Cognition Intertwined with Emotion” argues that an agent must “be embodied,” “interact using speech,” “be physically co-located with language speakers,” “distinguish concrete and abstract concepts,” and “use affect and emotion,” while treating embodiment as “being able to perceive and enact in the world” and insisting that affect and cognition are inseparable in language acquisition (Kennington, 2023). In ELENA terms, this places bodily perception, action, memory, and emotion narration in a single pipeline rather than in separate recognition and generation modules.

2. Skeleton motion as a tokenized language

A decisive extension of ELENA replaces image-only affect reading with motion-language modeling. In “Understanding Emotional Body Expressions via LLMs,” the skeleton sequence is formalized as $(L, E_e, E_i, N, B)$6, where each frame is $(L, E_e, E_i, N, B)$7, and temporal length is standardized to $(L, E_e, E_i, N, B)$8 frames per sequence by padding shorter sequences with preceding frames and downsampling longer ones following CTR-GCN practices (Lu et al., 2024). A GCN-based encoder maps motion to $(L, E_e, E_i, N, B)$9 and then to multi-granularity tokens: a global semantic token $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$0, temporal tokens $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$1 derived by temporal pooling, and spatial tokens $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$2 derived by spatial pooling. Each token is projected from $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$3 to the LLaMA-7B embedding dimension $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$4 through a learnable linear layer, then concatenated with text prompts such as “#Human: <Skeleton><SkeletonFeature></Skeleton> Can you tell me the emotion of this person? #Assistant:”.

This tokenization is paired with a unified skeleton token module (UST) for heterogeneous datasets. Spatial and temporal token sequences are padded to a shared length $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$5 and masked by $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$6, allowing datasets with different joint counts, frame counts, capture rates, and label spaces to occupy a shared “skeleton language” space. Skeleton-language alignment is further enforced through contrastive learning against CLIP text embeddings using $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$7 for spatio-temporal tokens and $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$8 for semantic tokens. The LLM backbone remains frozen except for LoRA adapters with rank $$L \in \{\text{Happiness, Sadness, Anger, Fear, Disgust, Surprise, Neutral}\}$$9, $E_e$0, and dropout $E_e$1, while the skeleton encoder is also frozen during LLM adaptation.

The empirical profile is bifurcated. Semantic tokens are better for recognition, whereas spatio-temporal tokens are better for description. Combining spatial and temporal tokens improves classification accuracy across Emilya, KDAE, and EGBM, and UST improves transfer especially with semantic tokens. On the state-of-the-art comparison, EAI-LLM reaches 71.17% on KDAE and 66.97% on EGBM, exceeding CTR-GCN on both datasets, while remaining below CTR-GCN and GAP on Emilya; importantly, it also generates explanations. On narrative metrics under identical prompts over videos derived from skeletons, EAI-LLM obtains ROUGE 0.2018, BLEU 0.1177, and METEOR 0.2148, compared with GPT-4 at 0.1282, 0.0686, and 0.2006, and Gemini 1.5 Pro at 0.1007, 0.0596, and 0.1823 (Lu et al., 2024). The training order also matters: recognition then description causes catastrophic forgetting relative to description then recognition, with average accuracy drops of 37.06% for semantic tokens and 61.86% for spatio-temporal tokens when description is fine-tuned after recognition.

In ELENA, this skeleton work operationalizes a strong claim: body motion can be treated as a first-class language with its own tokens, and narration quality improves when the model preserves spatio-temporal detail instead of collapsing the sequence into a single label.

3. Visual ELENA, facial bias, and trigger-centered explanation

The image-based ELENA framework is explicitly zero-shot and diagnostic. It uses face masking via YuNet with confidence threshold $E_e$2, complete occlusion of detected face regions, and a structured prompt requiring strict JSON output. Cross-attention maps from Llama-3.2-11B-Vision are summarized as $E_e$3, and facial bias is characterized by a Facial Attention Ratio, $E_e$4 (Saim et al., 23 Sep 2025). The core finding is persistent facial-region bias: deeper cross-attention layers focus strongly on the mouth in unmasked images, and under masking the model may still attend to the masked facial area or diffuse its attention rather than reallocating it to limbs and torso.

Nevertheless, structured embodied prompting improves classification substantially in face-masked settings. On HECO Masked, macro F1 rises from 16.9 to 31.5 for Gemini 2.5 Flash, from 14.4 to 22.5 for Llama-3.2-11B, and from 14.3 to 23.6 for Gemma-3-12B. On EMOTIC Masked, the gains are 15.6 to 26.0, 17.1 to 20.7, and 17.2 to 24.8, respectively. On BESST Masked, Llama-3.2-11B improves from 22.2 to 37.8 and Gemma-3-12B from 32.3 to 41.6. Textual body-part mentions also shift sharply: in HECO, Head/Face falls from 40.63% in Normal to 4.71% in Masked, while Limbs rise from 17.14% to 50.18% and Torso from 13.21% to 22.98% (Saim et al., 23 Sep 2025). These results establish ELENA as a method for redirecting descriptive focus from face-centric priors toward embodied cues, even though attention adaptation remains incomplete.

Emotion Interpretation generalizes this from “which body parts matter” to “why the emotion arises.” EIBench contains 1,615 basic EI samples and 50 complex EI samples derived from CAER-S and EmoSet, with trigger sets organized into ten causal categories and annotated through the Coarse-to-Fine Self-Ask (CFSA) pipeline. CFSA proceeds through Initial Question Preprocessing, General Self-Ask Thinking, Scenario Self-Ask Thinking, and Emotion Summarization, followed by human refinement to remove hallucinations, add commonsense, and prune irrelevant triggers (Lin et al., 10 Apr 2025). In the basic EI setting, CFSA with LLaVA-NEXT (34B) reaches approximately 68.81/68.04 overall recall under LLaMA-3/ChatGPT criteria; in the complex EI setting, performance drops markedly, with LLaVA-1.5 (13B) at 38.10/39.53 and ChatGPT-4o at 39.27/39.57. Long-Term Coherence remains high for many models, typically 80–86%, showing that fluent explanations may still miss causal correctness.

Together, these lines of work distinguish two layers of ELENA output. One layer anatomizes the emotion through body parts, posture, and somatic implication; the other layer attributes triggers across atmosphere, social interactions, body movements, facial expressions, objects, and implicit context. This suggests that full ELENA systems require both embodied grounding and explicit trigger reasoning.

4. Memory, somatic marker analogues, and affective recall

A later extension of ELENA addresses memory not as a store of facts alone but as a store of felt episodes. “The Echo Amplifies the Knowledge: Somatic Marker Analogues in LLMs via Emotion Vector Re-Injection” distinguishes semantic memory from episodic memory through Tulving’s noetic versus autonoetic framing and operationalizes a somatic marker analogue in Gemma 3 1B-IT using Gemma Scope 2 sparse autoencoders (Glover, 9 May 2026). The system identifies 310 emotion-exclusive features at layer 22 under differential activation thresholds of activation $E_e$5 on emotional probes and $E_e$6 on neutral probes, constructs episode-specific echo vectors from the top-$E_e$7 absolute deviations with $E_e$8, and re-injects a norm-scaled residual vector at layer 22 when a layer-7 context match fires under BDN-mean-sub similarity.

The memory store contains a context key, emotion echo payload, semantic label, and metadata. At retrieval time, four conditions are compared: A (no memory), B (semantic labels), C (emotion echo), and BC (semantic + echo). In emotional orientation, the threat-rating regression slope on contextual similarity is 0.80 for C versus 0.56 for A, with permutation-test $E_e$9; B and BC reach 1.20 and 1.13, both with $E_i$0 versus A. In decision-making at $E_i$1 and $E_i$2 per condition, good choices are A = 20%, C = 22% (n.s.), B = 52% ($E_i$3, $E_i$4), and BC = 80% ($E_i$5, $E_i$6), with BC versus B at $E_i$7, $E_i$8 (Glover, 9 May 2026). The central empirical conclusion is exact: the echo changes how the model feels independently, but changes what it does only when combined with knowledge.

For ELENA, the relevance is direct. Narrative continuity across sessions requires more than retrieval of prior labels or summaries; it requires a mechanism by which prior affective stance can be reactivated when a new embodied context resembles an old one. A plausible implication is that ELENA narratives can become temporally coherent not only by remembering “what happened” but also by reinstating the affective gradients that bias interpretation and action.

5. Mechanistic emotional circuits and inference-time regulation

Mechanistic work on LVLMs provides a causal account of how emotional information flows inside ELENA-like systems. “Interpreting and Enhancing Emotional Circuits in Large Vision-LLMs via Cross-Modal Information Flow” introduces a steering-vector-based causal attribution framework and the training-free intervention VEENA, centered on an Adapt–Aggregate–Execute mechanism (Zhang et al., 21 May 2026). In shallow layers, visual tokens are adapted into language-compatible semantics; in middle layers, sentiment-specific attention heads aggregate emotional cues into the query token; and in deep layers, emotion-general pathways execute narrative generation. The routing pivot is reported at $E_i$9, where steering impact peaks, visual-token entropy dips, and dominance transfers from $N$0 to $N$1.

The intervention acts at three loci. Steering vectors $N$2 are injected as $N$3 with $N$4, primarily at the query position around $N$5. Pre-softmax head scaling uses $N$6 to strengthen upstream sentiment-specific heads during prefill and downstream universal heads during decoding. Neuron excitation uses $N$7 on Top-30 neurons traced per head. On MER-UniBench with Qwen3-VL-4B-Instruct, the baseline mean across nine datasets is 58.1%; VEE only adds 5.2%, ENA only adds 3.3%, and VEENA reaches 64.8%, a total gain of 6.7%, with the largest gain of 9.2% on OV-MERD+. Latency increases by 1.82–9.46% ms/token depending on selection scale, with state-of-the-art performance at approximately 3% parameter intervention using Top-10 heads and Top-30 neurons (Zhang et al., 21 May 2026).

This mechanistic account matters because ELENA is vulnerable to emotional hallucinations and diffuse affective priors. VEENA shows that causal control can be exercised without full retraining by regulating identified heads and neurons at the layers where emotional intent is actually routed. In encyclopedic terms, it supplies the circuit-level infrastructure for making ELENA’s narratives more faithful to visual evidence while preserving general vision-language competence.

6. Embodied interfaces, robotic realization, and persistent limitations

Several systems instantiate ELENA principles at the interface and robotics level. E3VA uses ASR to convert user speech to text, ChatGPT 3.5 Turbo for response generation and sentiment extraction, a sentiment prompt that outputs JSON with mood in $N$8 and intensity in $N$9, and a Unity avatar with blendshapes for seven facial expressions, lip-sync from TTS audio amplitude, thinking and breathing animations, and a decay function returning the facial state to neutral (Kulkarni et al., 25 Feb 2026). In an exploratory pilot study with 12 participants, SUS is 77.71 (SD 17.30) and mean UES is 3.8 (SD 0.6), with qualitative feedback favoring chat history, ease of use, and expressiveness, and identifying response latency as the main complaint. E3VA does not include longitudinal affect tracking, but it provides a practical architecture for expressive turn-level interaction.

EVOLVE pushes embodiment beyond avatars to robot behavior selection. It uses camera images interpreted by a vision-LLM and an LLM that, through structured prompting, selects an emoji, a motion primitive, a color palette, and a short explanation; the target platform includes an OLED display, an LED strip, and a differential-drive base with two wheels (Sinclair et al., 2024). The work emphasizes atomic actions for motion and color, LLM verification of candidate responses, and qualitative alignment on EmoSet examples such as contentment, excitement, and fear. It also records a salient failure mode: color selection may follow image colors rather than affect, motivating grayscale preprocessing as a mitigation.

These interface systems sit alongside persistent limitations that recur across ELENA research. Static-image ELENA reports facial bias, attention adaptation failure, difficulty with visually subtle classes such as Disgust, and dataset mismatch when bodily cues are weak or distant (Saim et al., 23 Sep 2025). Skeleton-based ELENA reports hallucinations and ambiguity, including outputs such as “This person is expressing anxiety or fear,” which are conceptually related but marked as errors under strict label matching, as well as privacy concerns for motion-capture data and the risk of culturally specific gesture misinterpretation (Lu et al., 2024). E3VA explicitly lacks baselines and inferential statistics, and leaves ethics out of scope while relying on a prompt instructing the agent not to reveal it is non-human (Kulkarni et al., 25 Feb 2026). More foundationally, embodied-cognition work insists that a language-learning agent must be physically co-located, act in its environment, and ground language in affect and perception rather than in text alone (Kennington, 2023).

The convergent picture is therefore dual. ELENA now comprises concrete techniques for structured embodied narration, motion tokenization, trigger extraction, episodic affective recall, and causal circuit regulation. At the same time, the field remains constrained by face-centric pretraining, weak temporal grounding in some settings, incomplete multimodal memory, cross-cultural variability in bodily expression, and unresolved questions of privacy, transparency, and safe deployment.