MojiTalk: Emotion-Controlled Dialogue Generation

• MojiTalk is a framework that uses naturally occurring emojis as fine-grained emotion labels for generating contextual dialogue.
• It employs models like reinforced CVAE and Transformer-based architectures with modulated layer normalization for precise emotion control.
• Empirical evaluations show improved fluency and emotion accuracy, with metrics like perplexity and accuracy underscoring its performance.

MojiTalk is an emotion-controlled neural response generation framework and dataset designed to address the challenge of generating fluent, contextually relevant conversational responses aligned with specific affective states as represented by emojis. By leveraging the natural use of emojis in public Twitter conversations, MojiTalk produces a large-scale, fine-grained emotion-labeled dataset and a suite of conditional language generation models, including reinforced conditional variational autoencoders (R-CVAE) and, in subsequent research, Transformer-based architectures with emotion modulation. This approach enables large-scale empirical exploration of emotional text generation and precise control over affect in neural conversational agents (Zhou et al., 2017, Liu et al., 2021).

1. Problem Formulation and Motivation

Generation of emotional language is fundamental for building empathetic and engaging natural language processing agents. Historically, progress in this area has been limited by small datasets with coarse sentiment labels (e.g., positive/negative or 6–8 discrete emotions) and the infeasibility of manually annotating large conversational corpora with fine-grained affective signals. MojiTalk circumvents these limitations by exploiting the fact that Twitter users frequently annotate their messages with emojis indicative of nuanced emotions. This provides a scalable, automatic labeling mechanism for conversational data, supporting conditional generation tasks where the target emotion is specified by any of 64 emoji classes (Zhou et al., 2017).

2. Dataset Collection and Processing

Corpus Construction

• Data Source: Twitter streaming API, filtered for two-turn conversations collected from August 12–14, 2017.
• Inclusion criteria: Responses must:
  • Be in English
  • Contain at least one of 64 selected standalone emojis
  • Exclude URLs, images, videos
  • Have at least three alphabetic words
• Emoji Label Assignment:
  • If multiple emoji types occur, select the most frequent; in case of ties, favor the rarer emoji across the corpus for finer emotion granularity.
• Preprocessing:
  • Remove mentions and hashtags
  • Tokenize punctuation and emojis adjacent to words
  • Reduce repeated letters, punctuation, and emojis to two copies max
  • Restrict to a 20,000-word vocabulary, mapping OOV tokens to <UNK>

Dataset Statistics

Split	Size	Emojis	Max/Min Class Freq
Train	596,959	64	184,500 / 130
Dev/Test	32,600 ea	64	Zipfian distribution

The resulting corpus contains >660K (context, response, emoji) triples exhibiting a highly skewed, Zipf-like emoji frequency distribution. Each example consists of a context tweet, a target emoji, and the corresponding response tweet, e.g.:

$v_e$7 (Zhou et al., 2017, Liu et al., 2021)

3. Model Architectures

3.1 Recurrent Seq2Seq with Emoji Conditioning

• Encoder: Bidirectional GRU processes the context, producing a context vector $v_o$
• Emoji Embedding: Each emoji has a learnable embedding $e$ projected to $v_e$
• Decoder: Single-layer GRU with global attention; at each step, input is $[v_o; v_e]$ plus previous token

3.2 Conditional Variational Autoencoder (CVAE)

• Condition: $c = [v_o; v_e]$
• Recognition network $q_R(z|x,c)$: Bi-GRU over gold response $x$, outputs $\mu_R(c, x), \sigma_R(c, x)$ (diagonal Gaussian)
• Prior network $p_P(z|c)$: MLP over $c$, outputs $e$0
• Latent variable $e$1 via reparameterization
• Decoder $e$2: Same GRU, initial state and input augmented with $e$3

3.3 Reinforcement Learning Augmentation

• Emoji classifier $e$4 scores the generated output for correctness of emotion
• Reward $e$5 for generated response $e$6
• Baseline $e$7 compares to true response
• Hybrid Objective: Combines variational loss with policy gradient (REINFORCE), using $e$8

3.4 Modulated Transformer-Based Models

The "Modulating LLMs with Emotions" framework injects a one-hot 64-dim vector $e$9 (corresponding to the target emoji class) into each Transformer layer using modulated layer normalization (Mod-LN):

$v_e$0

where $v_e$1 are learned via compact two-layer MLPs per layer, effectively modulating network activations on a per-emoji basis. Encoder-decoder pairs used BERT or GPT-2 backbones, with minimal architectural changes beyond Mod-LN (Liu et al., 2021).

4. Training and Objective Functions

4.1 Loss Functions

• (Conditional) Evidence Lower Bound:

$v_e$2

• Bag-of-Words Loss: Used to counter posterior collapse,

$v_e$3

• Reinforcement Loss: REINFORCE with reward based on classifier $v_e$4, hybridized with generative losses as above.
• Decoding: Top-$v_e$5 sampling ($v_e$6) in Mod-LN models for diversity.

4.2 Optimization Details

• Optimizers: AdamW with custom warmup/decay, batch sizes of 32/GPU across 4 GPUs
• Early stopping: Monitored on dev perplexity

5. Evaluation and Empirical Results

5.1 Automatic Metrics

Key results on the test set (CVAE/R-CVAE from (Zhou et al., 2017), Mod-LN from (Liu et al., 2021)):

Model	Perplexity	Top-1 EMJ Acc	Top-5 EMJ Acc	BERTScore F1	TTR-1	TTR-2	Avg Len	Sent. Diversity
Seq2Seq (Base)	130.6	33.9%	58.1%	—	0.0061	0.0199	—	Low
CVAE	36.9	41.4%	75.1%	—	0.0191	0.1310	—	Modest
R-CVAE	38.3	42.1%	77.3%	80.8%	0.0160	0.1180	9.2	Mid
Mod-LN (BERT, 100%)	50.72	33.6%	72.2%	83.5%	0.059	0.390	14.3	High
Mod-LN (GPT-2, 100%)	36.31	34.4%	82.5%	85.4%	0.057	0.400	12.5	High
Human	—	—	—	—	0.059	0.430	11.7	Highest

All advanced models, especially Mod-LN with autoregressive GPT-2 backbones, demonstrate increased fluency, coherence, and emotion control. Mod-LN models consistently outperform R-CVAE in low-resource scenarios as well.

5.2 Human Evaluation

• Pairwise comparison (relevance, emotion fit): R-CVAE and Mod-LN models were judged at near-human levels for both emotion control and relevance.
• Emotion control/readability ratings (7-pt scale, Mod-LN): Mod-LN ARLM (GPT-2) achieves 5.40—5.43 in emotion control, and 5.20—5.32 in readability (human reference: 5.62/5.34).
• Turing-style test: 18% of annotators rated machine outputs as human, 27% could not distinguish.

6. Analysis, Limitations, and Qualitative Observations

Dataset and Annotation Noise

• The emoji distribution is Zipfian, resulting in underrepresentation for certain emotions (e.g., fear, disgust).
• Single emoji labels may introduce noise due to polysemy, irony, or context-dependence in emoji usage.
• Twitter-only data may limit generalizability to other conversational settings or multi-turn dialogue.

Generative Behavior

• RNN base models tended to default to generic constructions (e.g., "I'm ..."), while CVAE and especially RL-augmented CVAE produced more varied and affect-aligned outputs.
• Mod-LN Transformer models demonstrated context-contingent, affect-rich, idiomatic language that aligns well with target emoji classes.

Qualitative Illustration

Generated responses targeting different emojis appropriately reflected joy (😂), sadness (😢), anger (😡), and love (😍) using both lexical and affective cues, e.g.:

$v_e$8 (Liu et al., 2021)

7. Impact, Extensions, and Research Directions

MojiTalk—both as a dataset and suite of models—has served as a primary benchmark for research on controllable emotion generation in dialogue. Subsequent advances, such as Mod-LN, demonstrate that transformer architectures augmented with per-emoji modulation outperform earlier CVAE-based approaches, matching or surpassing human-level outputs on multiple criteria even in low-resource regimes (Liu et al., 2021).

Key research directions identified include:

• Expanding the dataset beyond Twitter and incorporating emotion labels with less ambiguity and greater coverage of rare affective states
• Adapting conditional affect control to multi-turn or task-oriented dialogue systems
• Enhancing latent-variable and hierarchical generative architectures to further improve relevance and emotional fidelity
• Ongoing accumulation and curation of new conversational data for improved balance and generalizability

MojiTalk establishes that naturally occurring emojis can serve as scalable, weakly supervised emotion labels, enabling the training of conditional neural response generators with high affective controllability and diversity (Zhou et al., 2017, Liu et al., 2021).