TwitterCorpus: Twitter Text Adaptation Corpus

• TwitterCorpus is a large-scale, text-only corpus of 2.8 billion English tweets curated to adapt language models to Twitter's informal and symbol-rich language.
• The corpus is preprocessed using NLTK’s TweetTokenizer and normalization techniques for mentions, URLs, and emojis to optimize masked language model pretraining.
• It plays a critical role in a two-stage training pipeline, enhancing rumor detection and other social media tasks with measurable performance improvements.

TwitterCorpus is a large-scale, pure-text corpus of English tweets curated for continue pretraining of pretrained LLMs on social-media text. It was introduced as the Stage-1 training resource for SoLM, a Twitter-tailored BERT-base model for rumor detection, and contains approximately 2.8 billion English tweets, or 269 GB of uncompressed text, collected from 2015 to 2022. The corpus is explicitly designed to mitigate corpus mismatch between generic pretraining data and Twitter language, improve handling of social symbols such as mentions, hashtags, URLs, and emojis, and serve as the textual counterpart to propagation-aware supervision introduced later through the UTwitter and UWeibo datasets (Cui et al., 10 Aug 2025).

1. Definition, scope, and design objectives

TwitterCorpus was created to address three deficiencies that the authors identify in conventional PLM pretraining for social-media tasks: mismatch between pretraining corpora and social text, inadequate handling of social symbols, and pretraining objectives that do not model user engagement structure. Its immediate target task is rumor detection, but the paper also situates it as potentially useful for content recommendation, social network analysis, and user behavior analysis (Cui et al., 10 Aug 2025).

The corpus is English-only and consists of “source tweets” only. It does not contain comments, replies, or conversation trees, and therefore does not itself encode propagation structure. That distinction is central to its design: TwitterCorpus supplies large-scale domain-matched text for Masked Language Modeling, whereas structural supervision is deferred to UTwitter and UWeibo in a second pretraining stage. The corpus spans 2015–2022, and its post-length distribution reflects short-form social writing: 57.92% of tweets contain fewer than 20 tokens, while only 0.01% exceed 100 tokens (Cui et al., 10 Aug 2025).

This design places TwitterCorpus in the category of domain-adaptation corpora rather than task-annotated benchmarks. A plausible implication is that its principal value lies in shifting a PLM’s lexical, orthographic, and symbol-level priors toward Twitter-native usage before any task-specific supervision is applied.

2. Data acquisition and normalization pipeline

The corpus was extracted from the Twitter Stream Grab maintained by Archive Team. The authors report that they extracted 2.8 billion English tweets from 2015 to 2022 from this public archive. The paper specifies English filtering, but does not report additional corpus-level controls such as deduplication, spam removal, bot filtering, or treatment of protected or deleted accounts for TwitterCorpus itself (Cui et al., 10 Aug 2025).

Preprocessing is optimized for pretraining rather than linguistic annotation. Raw tweet text is segmented with NLTK’s TweetTokenizer. Mentions and URLs are canonicalized to the special tokens <@user> and <url>, and emojis are translated into textual tokens using the Python emoji package. The authors also state that they standardize different fonts before tokenization. The SoLM vocabulary derived for this setup has size 52,000 and includes seven special tokens: [UNK], [SEP], [PAD], [CLS], [MASK], <@user>, and <url>. The maximum positional encoding length is 128, a cap chosen to reflect typical tweet length (Cui et al., 10 Aug 2025).

TwitterCorpus is released as text-only, uncompressed data, but the paper does not describe a record schema, file format, or example instance. It also does not specify a license. Access is provided through a Mega download link, and the accompanying code for SoLM and Post Engagement Prediction is released separately on GitHub (Cui et al., 10 Aug 2025).

3. Role in the SoLM training stack

TwitterCorpus is the first stage of a two-stage continue-pretraining pipeline. In Stage 1, a BERT-base encoder is trained from scratch on TwitterCorpus using Masked Language Modeling in order to internalize Twitter-specific text distributions and symbol handling. In Stage 2, the resulting model is further pretrained on UTwitter with MLM plus Post Engagement Prediction, a multi-task objective intended to inject propagation-structure semantics into the encoder (Cui et al., 10 Aug 2025).

The relation between the datasets is complementary. TwitterCorpus is broad and text-only, covering 2015–2022 without conversation structure. UTwitter and UWeibo, by contrast, are unlabeled claim-conversation datasets with explicit propagation trees. UTwitter contains 204,922 unlabeled claims and about 17 million tweets, with an average of about 82.5 posts per claim; UWeibo contains 209,549 unlabeled claims and about 11 million posts, with an average of about 50.5 posts per claim. These resources are used to derive the three PEP sub-tasks: Root Prediction, Branch Prediction, and Parent Prediction (Cui et al., 10 Aug 2025).

The final PEP objective is defined as

$$\mathcal{L}_{\mathrm{PEP}}=\alpha\,\mathcal{L}_{\mathrm{RoP}}+\beta\,\mathcal{L}_{\mathrm{BrP}}+\gamma\,\mathcal{L}_{\mathrm{PaP}}, \quad \alpha=\beta=\gamma=1.$$

Within this pipeline, TwitterCorpus is responsible for the generic social-text adaptation step, while UTwitter/UWeibo provide structurally grounded self-supervision. SoLM training uses Hugging Face Transformers and AdamW. For Stage 1 on TwitterCorpus, the reported settings are batch size 8,000, peak learning rate 0.0004, 40 epochs, and warmup over the first 4 epochs. Full SoLM training takes about 14 days on 8× A800 80GB SXM GPUs (Cui et al., 10 Aug 2025).

4. Empirical contribution to rumor detection

The empirical rationale for TwitterCorpus is supported by ablation results within the SoLM framework. Across Weibo, DRWeibo, Twitter15, Twitter16, and PHEME, PEP-enhanced PLMs improve baseline rumor detection performance by 1.0–3.7 percentage points, measured as accuracy on balanced datasets and macro-F1 on PHEME. For SoLM specifically, adding the Stage-2 PEP objective to the MLM-pretrained model yields gains of up to +2.0 on Twitter15, +1.5 on Twitter16, and +1.7 on PHEME (Cui et al., 10 Aug 2025).

The Stage-1 corpus itself also shows measurable value. When the MLM pretraining stage on TwitterCorpus is removed, performance drops by 0.8–1.1 points in the reported BiGCN experiments on Twitter15 and Twitter16. Removing Stage-2 PEP causes a larger degradation of 1.6–1.9 points, and among the PEP subtasks the paper reports that Parent Prediction and Root Prediction are the most impactful, with the ordering $\mathrm{PaP} \ge \mathrm{RoP} > \mathrm{BrP}$ (Cui et al., 10 Aug 2025).

Few-shot experiments reinforce the same pattern. For labeled sample sizes between 10 and 140, SoLM substantially improves BiGCN and GACL on Twitter15 and Twitter16, with the advantage narrowing as the number of labeled examples increases. This suggests that large-scale Twitter-domain adaptation via TwitterCorpus contributes especially when supervised signal is scarce (Cui et al., 10 Aug 2025).

5. Position within the landscape of Twitter corpora

TwitterCorpus occupies a different niche from smaller, annotation-heavy Twitter datasets. Corpora such as TACO, which annotates 1,814 tweets from 200 conversations for argument mining, and ILiAD, which provides a fully annotated English Twitter corpus with morphology, syntax, lemmas, n-grams, and interactive visualizations, are designed for interpretive or task-specific analysis rather than foundation-model pretraining (Feger et al., 2024, Gonzalez, 2024). Likewise, “A Corpus of English-Hindi Code-Mixed Tweets for Sarcasm Detection” provides 5,250 code-mixed tweets labeled for sarcasm/irony and token-level language identification, serving as a benchmark resource for code-mixed NLP rather than a large-domain adaptation corpus (Swami et al., 2018).

It also differs from Twitter knowledge bases such as TweetsKB and TweetsCOV19. Those resources expose semantically enriched tweet metadata, entities, sentiment, and related annotations through RDF-based schemas while deliberately withholding tweet text to comply with platform policies. TweetsKB contains approximately 1.56 billion English, non-retweet tweets from 2013–2017, and TweetsCOV19 contains 8,151,524 original English tweets about COVID-19 from October 2019 to April 2020 (Fafalios et al., 2018, Dimitrov et al., 2020). By contrast, TwitterCorpus is explicitly text-centric and intended for direct MLM pretraining.

Within the narrower family of social-media pretraining corpora, the authors position TwitterCorpus against BERTweet, TimeLMs, XLM-T, and TwHIN-BERT. The paper notes that TwitterCorpus is larger than BERTweet’s 850 million tweets and is paired with a second-stage propagation-aware objective absent from text-only pretraining regimes. This suggests that its novelty is not scale alone, but scale combined with a division of labor between text adaptation and engagement-structure learning (Cui et al., 10 Aug 2025).

6. Limitations, access conditions, and research implications

Several limitations are explicit. TwitterCorpus is English-only, and its 2015–2022 time window introduces temporal drift and regional-language skew. It is text-only, excluding images and videos, which remain important in multimodal rumor detection. It also contains no replies or tree structures, so propagation-aware modeling requires external datasets such as UTwitter and UWeibo. The paper does not report plans to refresh the corpus beyond 2022, and it does not specify licensing or a formal schema (Cui et al., 10 Aug 2025).

The ethical framing emphasizes that only publicly available content was collected and that personally identifiable information was removed prior to release for academic research. At the same time, the paper does not enumerate detailed Twitter Terms-of-Service compliance procedures for redistribution. This leaves the governance model less formalized than in corpora that distribute only tweet IDs or derived annotations, such as the Spanish–English borrowing/codeswitching corpus, TweetsKB, or TweetsCOV19 (Mellado et al., 2022, Fafalios et al., 2018, Dimitrov et al., 2020).

From a methodological perspective, TwitterCorpus is best understood as infrastructure for social-domain PLM adaptation. It does not offer task labels, linguistically layered annotations, or a fixed benchmark split. Instead, it provides large-scale exposure to Twitter-native lexical variation, informal orthography, emoji usage, and normalized social symbols. The empirical results reported for SoLM indicate that this exposure is not redundant with propagation-aware pretraining, but a distinct component whose removal measurably harms downstream rumor detection (Cui et al., 10 Aug 2025).