Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm (1708.00524v2)

Abstract: NLP tasks are often limited by scarcity of manually annotated data. In social media sentiment analysis and related tasks, researchers have therefore used binarized emoticons and specific hashtags as forms of distant supervision. Our paper shows that by extending the distant supervision to a more diverse set of noisy labels, the models can learn richer representations. Through emoji prediction on a dataset of 1246 million tweets containing one of 64 common emojis we obtain state-of-the-art performance on 8 benchmark datasets within sentiment, emotion and sarcasm detection using a single pretrained model. Our analyses confirm that the diversity of our emotional labels yield a performance improvement over previous distant supervision approaches.

Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm

Overview

The paper, authored by Bjarke Felbo et al., presents an approach leveraging the ubiquity of emojis in social media texts to pretrain models on emotional representations. The authors argue that distant supervision techniques in NLP, such as using emoticons or hashtags, can be extended to include a more diverse set of noisy labels to learn richer emotional representations. This extension is pivotal as it allows the construction of a more generalizable model that performs well across various domains and NLP tasks related to sentiment, emotion, and sarcasm detection.

Pretraining on Emoji Data

A large-scale dataset of 1.2 billion English-language tweets, each containing one of 64 common emojis, serves as the foundation for pretraining. Tweets containing URLs or other non-relevant content are filtered out, and specific preprocessing steps such as token normalization are applied. This pretraining utilizes an LSTM-based architecture named DeepMoji, consisting of bidirectional LSTMs and an attention mechanism to capture the semantic and emotional nuances of the texts.

Benchmarks and Performance

The pretrained DeepMoji model is evaluated across eight benchmark datasets spanning five domains, focusing on tasks such as sentiment analysis, emotion detection, and sarcasm detection. Results show that DeepMoji outperforms state-of-the-art models significantly across these tasks. Notably, for emotion datasets, DeepMoji achieves higher F1 scores, and for sentiment tasks, it shows superior accuracy compared to previous models utilizing simpler emoticon-based pretraining methods.

Some key results include:

• F1 score of 0.57 on the PsychExp dataset, compared to the state-of-the-art score of 0.45.
• Accuracy of 0.93 on the SS-Youtube dataset, significantly improving upon the previous benchmark.

Transfer Learning and Chain-Thaw Approach

A significant contribution of the paper is the introduction of the "chain-thaw" transfer learning method. This method incrementally fine-tunes each layer of the pretrained model one at a time, starting from the top layers down to the embeddings. This approach mitigates overfitting on small datasets and allows for better utilization of pretrained knowledge. Comparative analysis confirms that chain-thaw consistently provides the best performance over other approaches like last (only fine-tuning the final layer) and full (fine-tuning the entire network simultaneously).

Importance of Emoji Diversity

Through hierarchical clustering on the correlation matrix of emoji predictions, the paper demonstrates that the model effectively captures similarities and nuances in emoji usage. The diversity in emotional labels (64 types of emojis) leads to richer and more transferable emotional representations compared to models pretrained on binary sentiment datasets with positive and negative emoticons.

Theoretical and Practical Implications

The paper has substantial theoretical implications as it highlights the capability of diverse, weakly supervised datasets in training robust NLP models. Practically, the DeepMoji model showcases a substantial advance, particularly in user-generated content analysis where data is typically noisy and diverse. The ability to generalize across different domains means these models can be deployed in varied real-world applications without extensive retraining.

Future Directions

The research opens up several avenues for future exploration. Firstly, the paper suggests deeper investigations into model architecture, particularly the impact of attention mechanisms and skip-connections on transfer learning. There is also potential in exploring other rich, diverse data sources beyond emojis for pretraining NLP models. Further, ethical considerations around the use of social media data require continuous examination, particularly with respect to data privacy and representation fairness.

In conclusion, the paper by Felbo et al. presents a robust method for leveraging abundant social media data through the use of emojis to pretrain models for detecting sentiment, emotion, and sarcasm. The introduction of the chain-thaw transfer learning approach presents further advancements in fine-tuning techniques, enhancing the model’s performance across varied NLP tasks. The findings contribute significantly to the ongoing development of generalizable, high-performing NLP systems.