TimeLMs: Diachronic Language Models from Twitter

Abstract: Despite its importance, the time variable has been largely neglected in the NLP and LLM literature. In this paper, we present TimeLMs, a set of LLMs specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based LLMs' capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift.

• The paper introduces a diachronic modeling approach that updates language models with Twitter data to reflect evolving linguistic trends.
• It employs continual learning with quarterly updates using 4.2 million tweets and NVIDIA V100 GPUs to maintain temporal relevance.
• Evaluations on the TweetEval benchmark show that TimeLMs outperform static models by reducing performance drops on recent data by over 10%.

TimeLMs: Diachronic LLMs from Twitter

The paper "TimeLMs: Diachronic LLMs from Twitter" addresses the crucial aspect of temporal dynamics in NLP models, emphasizing the need for temporal adaptability and specialization in LLMs to effectively handle real-time, evolving data. Current LLMs, such as BERT and RoBERTa, are typically trained on static corpora, which can limit their effectiveness in contexts where language evolves rapidly, such as social media platforms like Twitter.

Introduction and Motivation

The principle motivation for developing TimeLMs is the inherent limitation within static pre-training paradigms, which neglect temporal changes, thereby affecting a model's capability to generalize to new, unseen data. The rapid evolution of discourse, especially in social media, necessitates an approach where models are consistently updated with recent data to remain effective. Time-awareness is particularly crucial for understanding emerging topics and concept drifts, such as those witnessed during periods of significant global events like the COVID-19 pandemic. The authors argue that LLMs should continually adapt to mirror the current linguistic landscape to sustain their applicability and reliability.

Methodology

TimeLMs are developed using a diachronic approach, training on Twitter data collected over specific time-periods. Initially, a base model using data from up to the end of 2019 is established. From this base, continual learning methods are employed, updating the models quarterly with new data sets obtained via the Twitter Academic API. During each training phase, data preprocessing steps include filtering for active users (to avoid bot-generated noise), deduplication, and normalization. Models are trained on NVIDIA V100 GPUs, and each periodic update uses 4.2 million additional tweets, maintaining a contemporary relevance while building on historical data.

Evaluation

TimeLMs are evaluated through both downstream task performance and pseudo-perplexity measurements on time-specific test sets. The TweetEval benchmark, a well-known Twitter-based evaluation suite, confirms these models' competitiveness and validates their ability to perform across multiple NLP tasks encompassing sentiment analysis, hate speech detection, and more. Moreover, pseudo-perplexity evaluation reveals that as models are trained on more recent data, they exhibit better metrics, indicating improvements in language comprehension and adaptation to new linguistic patterns. Degradation analysis demonstrates a performance decline of over 10% when outdated models are tasked with recent data, reinforcing the need for continuous model updates.

Impact and Future Directions

This research underscores a significant shift towards incorporating time-dependent variables into LLMs, which is essential for applications demanding high temporal accuracy, like temporally-aware sentiment analysis or sociolinguistic research. Practically, TimeLMs can empower better real-time content moderation and trend analysis on platforms that are heavily time-sensitive.

Looking forward, an exciting avenue suggested involves integrating explicit temporal markers into models, potentially enhancing their capacity to contextually differentiate linguistic samples based on time. By exploring time-conditioned embeddings and temporally aware self-attention mechanisms, future temporal LLMs could be better positioned to comprehend and predict the drift of language over time.

In conclusion, "TimeLMs: Diachronic LLMs from Twitter" paves the way for dynamic NLP practices that not only preserve historical linguistic contexts but also adapt to contemporary speaking trends, making them indispensable tools in both academic and applied AI fields.