EmoACT: Computational Emotion Modeling

• EmoACT is a computational framework that applies Affect Control Theory’s EPA model to recognize, synthesize, and track emotions in diverse human-machine interactions.
• It leverages multimodal inputs, impression detection, and lexicon mapping to dynamically update EPA vectors and generate appropriate affective responses.
• The framework underpins applications in chatbots, human-robot interaction, and text messaging, improving emotional appropriateness and perceived agency.

EmoACT is an umbrella term for computational frameworks that integrate Affect Control Theory (ACT) into the recognition, synthesis, and tracking of emotions in artificial agents. These approaches use ACT’s mathematical model of socially situated emotion—anchored by the Evaluation-Potency-Activity (EPA) space—to generate, interpret, or track affective states during human-machine and human-human interaction. Applications span human-robot interaction (HRI), conversational agents, chatbot dialogue, and emotion estimation in multimedia. Key research focuses include real-time synthesis of robot emotional displays, emotionally-conditioned response generation in dialogue, and trajectory tracking in text messaging, leveraging advances in affective computing, deep learning, and affective linguistics.

1. Theoretical Foundations: Affect Control Theory

ACT provides the core formalism: every relevant social identity, behavior, or object is associated with an EPA vector, encoding affective meaning on three axes—Evaluation (E, good-bad), Potency (P, strong-weak), and Activity (A, active-passive)—with empirically derived value ranges (typically –4 to +4). In any interaction, identities (e.g., "teacher"), behaviors (e.g., "praise"), and settings are characterized by fundamental EPA profiles. When an event occurs (such as an utterance or action), impression-formation equations compute transient EPA impressions. The squared difference between these impressions and fundamental EPAs (deflection) is minimized in typical social behavior:

$D = \sum_{i\in\{E,P,A\}} w_i (f_i - t_i)^2$

where $f$ is the fundamental sentiment, $t$ is the transient impression, and $w_i$ are empirically determined weights. This principle—minimizing deflection—is foundational for ACT-inspired computational models (Corrao et al., 16 Apr 2025, Asghar et al., 2020, Mostafavi et al., 2021).

2. Computational Architectures and Pipelines

EmoACT frameworks architecturally translate ACT into agent pipelines that typically follow this progression: perception/appraisal of user affect → mapping to real-time EPA → emotion generation (according to ACT) → expression or response.

• Perception Module: Gathers multimodal cues—facial affect (often with SDKs such as MorphCast), gaze direction, distance—to dynamically update the agent’s impression vector ($imp_E,\,imp_P,\,imp_A$).
• Impression Detection: Applies appraisal updates to impression vectors based on user cues. For example, a change in user's facial affect or gaze increments corresponding EPA components, according to defined gain parameters ($\alpha_E,\,\beta_P,\,\gamma_E,\,\alpha_A$).
• Emotion Generation: Computes the agent’s emotion as an EPA vector, typically via equations using the discrepancy between the agent’s identity vector ($id$) and the current impression vector:
  • $emo_E = imp_E - id_E + 1 + (imp_A - id_A)\delta$
  • $emo_P = imp_P - id_P - (imp_A - id_A)$
  • $emo_A = imp_A + id_A$
• Emotion Expression: Maps continuous EPA to prototypical discrete emotion labels (e.g., Anger, Fear, Happiness, Sadness) by cosine similarity, and triggers multimodal cues (animation, LED, speech) (Corrao et al., 16 Apr 2025).
• Dialogue Integration: In dialogue agents, ACT-derived EPA is injected into neural generation models (e.g., prepending or conditioning GRUs/CVAEs on EPA target vectors) (Asghar et al., 2020).

These stages are implemented as modular, platform-agnostic microservices, supporting real-time interaction with hardware platforms such as Softbank Robotics Pepper (Corrao et al., 16 Apr 2025).

3. Multimodal Input, Appraisal Mechanisms, and Lexicon Extension

EmoACT extends classic ACT in two key technical aspects:

• Multimodal Input: Beyond text, systems ingest continuous multimodal streams—camera-derived gaze/distance and real-time facial affect. Appraisal functions update the impression EPA based on these modalities.
• Lexicon Expansion: To process contemporary communication (social media text, emojis), EmoACT frameworks map Word2Vec and Emoji2Vec embeddings into EPA space. This is accomplished by fitting a regression matrix $f$0 from embedding space to the EPA space using a seed lexicon with known affective ratings. For each token (word or emoji), this yields a continuous EPA estimate:

$f$1

where $f$2 incorporates second-order regression for accuracy. Emoji amalgamation with identities is further supported by specialized equations, enabling dynamic identity shifts in response to affective modifiers (Mostafavi et al., 2021).

4. Dialogue Systems and Emotionally Aligned Response Generation

ACT-based models are used for emotionally-aware neural dialogue systems, enabling emotionally aligned response generation. The prototypical pipeline involves:

1. S2EPA: Mapping a sentence (input prompt) to an EPA vector, typically via an emoji soft-labeling system (e.g., DeepMoji) followed by EPA aggregation.
2. ACT Prediction: Using ACT impression-formation equations to predict the respondent’s ideal EPA vector, given both interactants’ identities and the prompt EPA.
3. EPA2S: Conditioning a sequence-to-sequence or CVAE-based neural dialogue model on the target EPA, by either concatenation at input, or through conditional latent variables.

Empirical evaluations on open-domain dialogue datasets show that ACT-informed models generate responses with higher emotional appropriateness as judged by human annotators, with the best models aligning both content and affect in ~27% of test cases—surpassing baselines lacking ACT conditioning (Asghar et al., 2020).

5. Emotion Synthesis and Expression in Human-Robot Interaction

In real-world HRI, EmoACT direct emotion synthesis and expression on physical robots is achieved through continuous EPA tracking and mapping to agent-specific expressive channels. Key findings include:

• Frequency of Display: High-frequency emotional displays (EPA-driven animation/LED feedback after every utterance) significantly raise scores for robot agency and emotional experience compared to low-frequency feedback.
• Perceptual Impact: The visibility and naturalness of synthetic emotion depends critically on the calibration and frequency of feedback loops between perceptual appraisal and expression modules.
• Implementation: On the Pepper robot (NAOqi OS), external servers host Impression Detection and Emotion Generation, with output EPA mapped to validated pose/LED/rich speech cues.
• Statistical Validation: Human user studies (N=14 and N=16) confirm that frequent, ACT-controlled displays raise ratings for both emotional and cognitive agency dimensions, as opposed to control or infrequent feedback (Corrao et al., 16 Apr 2025).

6. Emotion Tracking in Text and Emoji-based Messaging

EmoACT supports automated tracking of emotional state trajectories in text- and emoji-rich online messaging. The workflow includes:

• Token-level EPA Mapping: All message tokens, including previously out-of-vocabulary emojis, are mapped to EPA vectors through the designed projection-and-regression pipeline.
• ACT Impression-change Equations: Actor–Behavior–Object models are applied per message turn to propagate transient impression updates in EPA space.
• Deflection Monitoring: Deflection is tracked at each turn, revealing emotional spikes during negative behaviors (“grouse at,” “criticize”) and reductions during positive actions (“uplift,” “gratify”).

Empirical analysis shows that user-specific affective trajectories (e.g., persistently lower Evaluation for “angry” users) and co-variation of chatbot EPA and deflection can be captured in a fully automated fashion, even as the vocabulary and emoji lexicon shifts over time (Mostafavi et al., 2021).

7. Limitations, Evaluation, and Future Directions

• Lexicon Quality: Emoji2Vec and word embedding affective mappings are constrained by the semantic noise in their tags and coverage; more accurate EPA lexica for emojis require dedicated surveys.
• Deflection as Optimization: While most implementations compute deflection passively, integration as a direct loss function or reward in RL-based agents remains a target for further research.
• Agency and Social Perception: Experiments indicate that emotional agency (ability to feel, react emotionally) is perceived independently from anthropomorphism or likeability, suggesting distinct avenues for optimizing HRI with EmoACT strategies.
• Robustness: Frameworks are robust to lexical sparsity (as few as one mood tag per message/track), and can be ported across platforms by recalibrating identity EPA and appraisal gains (Saari et al., 2013, Corrao et al., 16 Apr 2025).
• Interpretability: By grounding affect in EPA space and standard ACT equations, EmoACT retains high interpretability compared to black-box sentiment models.

EmoACT frameworks, by embedding ACT’s mathematically principled approach to affect into artificial agents, enable precise modeling, real-time synthesis, and dynamic tracking of emotion in diverse domains—dialogue, social robotics, and multimodal interaction—advancing both the affective intelligence and social grace of synthetic agents (Corrao et al., 16 Apr 2025, Asghar et al., 2020, Mostafavi et al., 2021, Saari et al., 2013).