---
title: 'MojiTalk: Emotion-Controlled Dialogue Generation'
url: https://www.emergentmind.com/topics/mojitalk
type: topic
---

# MojiTalk: Emotion-Controlled Dialogue Generation

MojiTalk is an emotion-controlled neural response generation framework and dataset designed to address the challenge of generating fluent, contextually relevant conversational responses aligned with specific affective states as represented by emojis. By leveraging the natural use of emojis in public Twitter conversations, MojiTalk produces a large-scale, fine-grained emotion-labeled dataset and a suite of conditional language generation models, including reinforced conditional variational autoencoders (R-CVAE) and, in subsequent research, Transformer-based architectures with emotion modulation. This approach enables large-scale empirical exploration of emotional text generation and precise control over affect in neural conversational agents [1711.04090][2108.07886].

## 1. Problem Formulation and Motivation

Generation of emotional language is fundamental for building empathetic and engaging natural language processing agents. Historically, progress in this area has been limited by small datasets with coarse sentiment labels (e.g., positive/negative or 6–8 discrete emotions) and the infeasibility of manually annotating large conversational corpora with fine-grained affective signals. MojiTalk circumvents these limitations by exploiting the fact that Twitter users frequently annotate their messages with emojis indicative of nuanced emotions. This provides a scalable, automatic labeling mechanism for conversational data, supporting conditional generation tasks where the target emotion is specified by any of 64 emoji classes [1711.04090].

## 2. Dataset Collection and Processing

### Corpus Construction

- **Data Source**: Twitter streaming API, filtered for two-turn conversations collected from August 12–14, 2017.
- **Inclusion criteria**: Responses must:
  - Be in English
  - Contain at least one of 64 selected standalone emojis
  - Exclude URLs, images, videos
  - Have at least three alphabetic words
- **Emoji Label Assignment**:
  - If multiple emoji types occur, select the most frequent; in case of ties, favor the rarer emoji across the corpus for finer emotion granularity.
- **Preprocessing**:
  - Remove mentions and hashtags
  - Tokenize punctuation and emojis adjacent to words
  - Reduce repeated letters, punctuation, and emojis to two copies max
  - Restrict to a 20,000-word vocabulary, mapping OOV tokens to `<UNK>`

### Dataset Statistics

| Split    | Size      | Emojis | Max/Min Class Freq |
|----------|-----------|--------|--------------------|
| Train    | 596,959   | 64     | 184,500 / 130      |
| Dev/Test | 32,600 ea | 64     | Zipfian distribution|

The resulting corpus contains >660K (context, response, emoji) triples exhibiting a highly skewed, Zipf-like emoji frequency distribution. Each example consists of a context tweet, a target emoji, and the corresponding response tweet, e.g.:

```
Context: "I can't believe finals are over"
Target emoji: 😂
Response: "Same here lol I'm so relieved 😂"
```
[1711.04090][2108.07886]

## 3. Model Architectures

### 3.1 Recurrent Seq2Seq with Emoji Conditioning

- **Encoder**: Bidirectional GRU processes the context, producing a context vector $v_o$
- **Emoji Embedding**: Each emoji has a learnable embedding $e$ projected to $v_e$
- **Decoder**: Single-layer GRU with global attention; at each step, input is $[v_o; v_e]$ plus previous token

### 3.2 Conditional Variational Autoencoder (CVAE)

- **Condition**: $c = [v_o; v_e]$
- **Recognition network** $q_R(z|x,c)$: Bi-GRU over gold response $x$, outputs $\mu_R(c, x), \sigma_R(c, x)$ (diagonal Gaussian)
- **Prior network** $p_P(z|c)$: MLP over $c$, outputs $\mu_P(c), \sigma_P(c)$
- **Latent variable** $z\sim N(\mu, \operatorname{diag}(\sigma^2))$ via reparameterization
- **Decoder** $p_D(x|z,c)$: Same GRU, initial state and input augmented with $z$

### 3.3 Reinforcement Learning Augmentation

- **Emoji classifier** $C(\cdot)$ scores the generated output for correctness of emotion
- **Reward** $R = C_e(x')$ for generated response $x'$
- **Baseline** $r = C_e(x)$ compares to true response
- **Hybrid Objective**: Combines variational loss with policy gradient (REINFORCE), using $\mathcal{L}''(x,c) = \mathcal{L}'(x,c) - \lambda\,\mathcal{J}'(\theta), \lambda=1$

### 3.4 Modulated Transformer-Based Models

The "Modulating Language Models with Emotions" framework injects a one-hot 64-dim vector $c$ (corresponding to the target emoji class) into each Transformer layer using modulated layer normalization (Mod-LN):

\[
\widetilde{x}^{(l)} = \mathrm{MLP}_\gamma^{(l)}(c) \odot \frac{x^{(l)} - \mu}{\sigma + \epsilon} + \mathrm{MLP}_\beta^{(l)}(c)
\]

where $[\gamma^{(l)}(c), \beta^{(l)}(c)]$ are learned via compact two-layer MLPs per layer, effectively modulating network activations on a per-emoji basis. Encoder-decoder pairs used BERT or GPT-2 backbones, with minimal architectural changes beyond Mod-LN [2108.07886].

## 4. Training and Objective Functions

### 4.1 Loss Functions

- **(Conditional) Evidence Lower Bound:**
  \[
  \log p(x|c) \geq -\mathcal{L}(x,c),\quad \mathcal{L}(x,c) = \mathrm{KL}[q_R(z|x,c) \parallel p_P(z|c)] - \mathbb{E}_{z\sim q_R}[\log p_D(x|z,c)]
  \]
- **Bag-of-Words Loss:** Used to counter posterior collapse,
  \[
  \mathcal{L}'(x, c) = \mathcal{L}(x, c) + \mathbb{E}_{z \sim q_R}[\log p(x_{bow}|z, c)]
  \]
- **Reinforcement Loss:** REINFORCE with reward based on classifier $C$, hybridized with generative losses as above.
- **Decoding:** Top-$k$ sampling ($k=10$) in Mod-LN models for diversity.

### 4.2 Optimization Details

- **Optimizers**: AdamW with custom warmup/decay, batch sizes of 32/GPU across 4 GPUs
- **Early stopping**: Monitored on dev perplexity

## 5. Evaluation and Empirical Results

### 5.1 Automatic Metrics

Key results on the test set (CVAE/R-CVAE from [1711.04090], Mod-LN from [2108.07886]):

| Model                | Perplexity | Top-1 EMJ Acc | Top-5 EMJ Acc | BERTScore F1 | TTR-1 | TTR-2 | Avg Len | Sent. Diversity |
|----------------------|------------|---------------|---------------|--------------|-------|-------|---------|----------------|
| Seq2Seq (Base)       | 130.6      | 33.9%         | 58.1%         | —            | 0.0061| 0.0199| —       | Low            |
| CVAE                 | 36.9       | 41.4%         | 75.1%         | —            | 0.0191| 0.1310| —       | Modest         |
| R-CVAE               | 38.3       | 42.1%         | 77.3%         | 80.8%        | 0.0160| 0.1180| 9.2     | Mid            |
| Mod-LN (BERT, 100%)  | 50.72      | 33.6%         | 72.2%         | 83.5%        | 0.059 | 0.390 | 14.3    | High           |
| Mod-LN (GPT-2, 100%) | 36.31      | 34.4%         | 82.5%         | 85.4%        | 0.057 | 0.400 | 12.5    | High           |
| Human                | —          | —             | —             | —            | 0.059 | 0.430 | 11.7    | Highest        |

All advanced models, especially Mod-LN with autoregressive GPT-2 backbones, demonstrate increased fluency, coherence, and emotion control. Mod-LN models consistently outperform R-CVAE in low-resource scenarios as well.

### 5.2 Human Evaluation

- **Pairwise comparison** (relevance, emotion fit): R-CVAE and Mod-LN models were judged at near-human levels for both emotion control and relevance.
- **Emotion control/readability ratings (7-pt scale, Mod-LN)**: Mod-LN ARLM (GPT-2) achieves 5.40—5.43 in emotion control, and 5.20—5.32 in readability (human reference: 5.62/5.34).
- **Turing-style test**: 18% of annotators rated machine outputs as human, 27% could not distinguish.

## 6. Analysis, Limitations, and Qualitative Observations

### Dataset and Annotation Noise

- The emoji distribution is Zipfian, resulting in underrepresentation for certain emotions (e.g., fear, disgust).
- Single emoji labels may introduce noise due to polysemy, irony, or context-dependence in emoji usage.
- Twitter-only data may limit generalizability to other conversational settings or multi-turn dialogue.

### Generative Behavior

- RNN base models tended to default to generic constructions (e.g., "I'm ..."), while CVAE and especially RL-augmented CVAE produced more varied and affect-aligned outputs.
- Mod-LN Transformer models demonstrated context-contingent, affect-rich, idiomatic language that aligns well with target emoji classes.

### Qualitative Illustration

Generated responses targeting different emojis appropriately reflected joy (😂), sadness (😢), anger (😡), and love (😍) using both lexical and affective cues, e.g.:

```
Context: "My car broke down in the rain…"
Target emoji: 😢
Generated: "Oh no, that's awful. I'm really sorry you have to deal with that. 😢"
```
[2108.07886]

## 7. Impact, Extensions, and Research Directions

MojiTalk—both as a dataset and suite of models—has served as a primary benchmark for research on controllable emotion generation in dialogue. Subsequent advances, such as Mod-LN, demonstrate that transformer architectures augmented with per-emoji modulation outperform earlier CVAE-based approaches, matching or surpassing human-level outputs on multiple criteria even in low-resource regimes [2108.07886].

Key research directions identified include:

- Expanding the dataset beyond Twitter and incorporating emotion labels with less ambiguity and greater coverage of rare affective states
- Adapting conditional affect control to multi-turn or task-oriented dialogue systems
- Enhancing latent-variable and hierarchical generative architectures to further improve relevance and emotional fidelity
- Ongoing accumulation and curation of new conversational data for improved balance and generalizability

MojiTalk establishes that naturally occurring emojis can serve as scalable, weakly supervised emotion labels, enabling the training of conditional neural response generators with high affective controllability and diversity [1711.04090][2108.07886].

Source: https://www.emergentmind.com/topics/mojitalk