---
title: EXIST 2024 Tweet Dataset Overview
url: https://www.emergentmind.com/topics/exist-2024-tweet-dataset
type: topic
---

# EXIST 2024 Tweet Dataset Overview

Searching arXiv for the specified paper to ground the article in the source record.
The dataset described in "A Public Dataset Tracking Social Media Discourse about the 2024 U.S. Presidential Election on Twitter/X" is the first release of a large-scale corpus of election-related discourse on $\mathbb{X}$, also identified as Twitter/X, centered on the 2024 U.S. Presidential Election [2411.00376]. It comprises 22 million publicly available posts collected on X.com between May 1, 2024, and July 31, 2024, using a custom-built scraper. The collection strategy uses targeted keywords linked to key political figures, events, and emerging issues, aligning retrieval with the election cycle in order to capture evolving public sentiment and the dynamics of political engagement on social media. The dataset is positioned as a resource for studying political discourse, election-related narratives, and misinformation, and it is accompanied by a preliminary frequency-based analysis of keywords and hashtags.

## 1. Scope, temporal coverage, and access

The corpus contains 22 million publicly available posts gathered during a three-month interval from May 1, 2024, to July 31, 2024 [2411.00376]. The documented collection window spans several campaign landmarks, including the first Democratic primary debates in June 2024, the RNC and DNC conventions in July 2024, the assassination attempt on Donald Trump on July 13, 2024, the naming of JD Vance as Trump’s running mate on July 15, 2024, and early campaign rallies, slogans, and emerging third-party movements.

Repository access is provided through GitHub at `https://github.com/sinking8/usc-x-24-us-election`. The dataset is distributed in JSON for full metadata dumps and in CSV for tabular analyses. This combination of temporal specificity and dual-format release suggests utility both for full-fidelity reconstruction of platform-native objects and for downstream statistical workflows.

Because the collection period is bounded and event-dense, the dataset is especially suited to studying short-horizon changes in online political discourse. A plausible implication is that analyses can be synchronized with discrete campaign shocks and organizational milestones rather than being limited to coarse election-season aggregates.

## 2. Collection architecture and scraping workflow

Data collection is performed by a custom engine named **X-Scraper**, which harvests posts directly from X.com’s web UI via a headless Chromium driver [2411.00376]. The scraper emulates a real browser, rotates through multiple accounts, and continuously queries for new content while respecting X.com’s rate limits.

The documented high-level workflow is:

1. Initialize logger, browser context, cookie store  
2. Loop over time intervals $\Delta t$ (e.g., 1 hour)  
3. For each interval and each keyword batch, compose a query string  
4. Navigate to X.com/search with the query parameters  
5. Intercept JSON responses, extract tweet objects  
6. Append new entries to disk, deduplicate via “patience” checks  

The paper also provides simplified pseudocode for the core scraping loop, including time-slice partitioning, keyword batching, account iteration, JSON interception, duplicate removal, CSV appending, and browser teardown. In methodological terms, the system combines browser automation with response interception rather than relying on a conventional platform API. This suggests a design optimized for large-scale, repeated retrieval under interface-mediated constraints.

## 3. Query construction, rate limits, and deduplication

Each search string uses grouped OR clauses over tracked keywords and hashtags together with explicit temporal bounds via `until:` and `since:` parameters [2411.00376]. The documented query structure is:

```text
(“thedemocrats” OR “DNC” OR “Kamala Harris” OR …)
  until:YYYY-MM-DD_hh:mm:ss_UTC
  since:YYYY-MM-DD_hh:mm:ss_UTC
```

Keywords and hashtags are grouped into comma-separated OR clauses. Appendix A.2 is identified as containing the full list of 60+ tracked terms, with examples including Biden, Trump2024, MAGA, bidenharris2024, Third Party, and RFK Jr.

Rate-limit handling is made explicit. Each account is capped at approximately $R_0 \simeq 2{,}300$ posts per hour, and with $N$ accounts the total throughput is

$$
R_{\mathrm{total}} = N \times R_0.
$$

Deduplication and stopping criteria are also specified. The scraper monitors output-file size at each iteration; if no new posts arrive for $P$ consecutive checks, scraping on that query stops. Before appending each batch, `tweet_id` is checked against an in-memory hash to prevent duplicates.

One interpretive caution follows directly from this design: the dataset is not described as an unconstrained sample of all election discourse on X, but as a keyword-targeted corpus. This suggests that observed prevalence patterns reflect both platform activity and the query vocabulary used to retrieve it.

## 4. Data model, normalization, and export structure

Each post is represented as a JSON object and can also be exported to CSV [2411.00376]. The primary fields include identifiers, text forms, timestamps, author information, engagement counts, thread metadata, hashtags, user mentions, shared links, media attachments, reply metadata, embedded quoted or retweeted objects, and additional contextual attributes such as location, user, viewCount, and cash_app_handle.

| Field name | Data type | Description |
|---|---|---|
| `id` | string | Unique tweet identifier |
| `text` | string | Normalized tweet text |
| `rawContent` | string | Original unnormalized text |
| `epoch` | integer | Creation time as Unix timestamp |
| `epoch_dt` | datetime | Human-readable timestamp |
| `user_id` | string | Tweet author’s user ID |
| `lang` | string | ISO language code |
| `replyCount` | integer | Number of replies |
| `retweetCount` | integer | Number of retweets |
| `likeCount` | integer | Number of likes |
| `quoteCount` | integer | Number of quotes |
| `conversationId` | string | ID of the conversation thread |
| `hashtags` | list[string] | Hashtags mentioned |
| `mentionedUsers` | list[string] | Usernames mentioned |
| `links` | list[string] | URLs shared |
| `media` | list[object] | Media attachments |
| `in_reply_to_*` | string | Reply metadata |
| `quotedTweet`, `retweetedTweet` | object | Embedded tweet objects |

All fields are nullable as objects when absent. In CSV exports, lists are encoded as pipe-delimited strings. Prior to export, text is normalized to NFC Unicode and zero-width characters are removed.

The coexistence of normalized and raw text, together with nullable embedded objects and thread identifiers, indicates support for both linguistic preprocessing and structural reconstruction. A plausible implication is that the corpus can be used for analyses ranging from lexical frequency estimation to conversation-level and retweet-level graph construction without requiring extensive schema redesign.

## 5. Preliminary descriptive statistics

The paper reports an initial frequency analysis for the May–July 2024 subcorpus and identifies the most prominent keywords and hashtags [2411.00376]. The top 10 keywords are:

| Keyword | Count |
|---|---:|
| Biden | 11,070,686 |
| Trump | 5,730,951 |
| MAGA | 2,648,869 |
| Joe | 2,136,631 |
| President | 1,671,385 |
| GOP | 1,560,280 |
| Donald | 1,534,099 |
| Harris | 1,169,948 |
| kamala | 984,206 |
| conservative | 806,766 |

The top 10 hashtags are:

| Hashtag | Count |
|---|---:|
| `#maga` | 559,722 |
| `#trump2024` | 341,792 |
| `#trump` | 170,923 |
| `#bidenharris2024` | 166,669 |
| `#biden` | 165,277 |
| `#donaldtrump` | 40,780 |
| `#biden2024` | 39,588 |
| `#gop` | 33,615 |
| `#joebiden` | 31,620 |
| `#usa` | 30,764 |

A high-level engagement summary is also reported across the full 22 million tweets:

| Metric | Mean | Std. Dev. | Median |
|---|---:|---:|---:|
| `retweetCount` | 5.1 | 23.4 | 0 |
| `likeCount` | 12.7 | 57.3 | 1 |
| `replyCount` | 0.8 | 3.5 | 0 |
| `quoteCount` | 0.3 | 2.1 | 0 |

The paper notes that medians of zero reflect the long-tail nature of social media engagement. In analytic terms, the combination of very high-frequency political keywords and zero-median engagement metrics is consistent with a corpus that mixes a relatively small number of highly amplified posts with a very large mass of low-engagement content.

## 6. Research uses and analytical implications

The dataset is presented as supporting a wide range of political-communication and computational-social-science inquiries [2411.00376]. The examples explicitly identified are misinformation propagation, sentiment dynamics, network analysis, bot and inauthentic-activity detection, topic modeling and event detection, and cross-platform coordination.

The paper specifies these application modes as follows:

- **Misinformation propagation**: trace rumor cascades around the July 13 assassination attempt  
- **Sentiment dynamics**: model shifts in public mood before/after conventions using temporal sentiment classifiers  
- **Network analysis**: reconstruct retweet/mention graphs to detect echo chambers and cross-ideological bridges  
- **Bot and inauthentic-activity detection**: compare verified vs. unverified accounts over time  
- **Topic modeling & event detection**: identify emerging subthemes such as Third Party and KAG with dynamic LDA or BERTopic  
- **Cross-platform coordination**: link the X.com data to companion TikTok and Telegram datasets for multi-modal analyses  

These use cases follow directly from the dataset’s scale, temporal coverage, and metadata richness. At the same time, one analytical constraint is implicit in the collection design: because retrieval is based on tracked terms and publicly available posts, any claim about platform-wide discourse should be interpreted relative to that observation strategy. This does not diminish the value of the resource; rather, it defines the conditions under which inference is best grounded. Within those conditions, the dataset offers a technically detailed and operationally transparent basis for studying election-related discourse in the lead-up to the 2024 U.S. Presidential Election.

Source: https://www.emergentmind.com/topics/exist-2024-tweet-dataset