EXIST 2024 Tweet Dataset Overview

• EXIST 2024 Tweet Dataset is a large-scale corpus of 22 million public posts from X.com collected during key events of the 2024 U.S. Presidential Election.
• It employs a custom headless browser scraper with rate-limit handling and deduplication to ensure high-fidelity data retrieval across a dense campaign period.
• The dataset supports diverse analyses such as sentiment dynamics, network reconstruction, misinformation tracking, and cross-platform coordination.

Searching arXiv for the specified paper to ground the article in the source record. The dataset described in "A Public Dataset Tracking Social Media Discourse about the 2024 U.S. Presidential Election on Twitter/X" is the first release of a large-scale corpus of election-related discourse on $\mathbb{X}$, also identified as Twitter/X, centered on the 2024 U.S. Presidential Election (Balasubramanian et al., 2024). It comprises 22 million publicly available posts collected on X.com between May 1, 2024, and July 31, 2024, using a custom-built scraper. The collection strategy uses targeted keywords linked to key political figures, events, and emerging issues, aligning retrieval with the election cycle in order to capture evolving public sentiment and the dynamics of political engagement on social media. The dataset is positioned as a resource for studying political discourse, election-related narratives, and misinformation, and it is accompanied by a preliminary frequency-based analysis of keywords and hashtags.

1. Scope, temporal coverage, and access

The corpus contains 22 million publicly available posts gathered during a three-month interval from May 1, 2024, to July 31, 2024 (Balasubramanian et al., 2024). The documented collection window spans several campaign landmarks, including the first Democratic primary debates in June 2024, the RNC and DNC conventions in July 2024, the assassination attempt on Donald Trump on July 13, 2024, the naming of JD Vance as Trump’s running mate on July 15, 2024, and early campaign rallies, slogans, and emerging third-party movements.

Repository access is provided through GitHub at https://github.com/sinking8/usc-x-24-us-election. The dataset is distributed in JSON for full metadata dumps and in CSV for tabular analyses. This combination of temporal specificity and dual-format release suggests utility both for full-fidelity reconstruction of platform-native objects and for downstream statistical workflows.

Because the collection period is bounded and event-dense, the dataset is especially suited to studying short-horizon changes in online political discourse. A plausible implication is that analyses can be synchronized with discrete campaign shocks and organizational milestones rather than being limited to coarse election-season aggregates.

2. Collection architecture and scraping workflow

Data collection is performed by a custom engine named X-Scraper, which harvests posts directly from X.com’s web UI via a headless Chromium driver (Balasubramanian et al., 2024). The scraper emulates a real browser, rotates through multiple accounts, and continuously queries for new content while respecting X.com’s rate limits.

The documented high-level workflow is:

1. Initialize logger, browser context, cookie store
2. Loop over time intervals $\Delta t$ (e.g., 1 hour)
3. For each interval and each keyword batch, compose a query string
4. Navigate to X.com/search with the query parameters
5. Intercept JSON responses, extract tweet objects
6. Append new entries to disk, deduplicate via “patience” checks

The paper also provides simplified pseudocode for the core scraping loop, including time-slice partitioning, keyword batching, account iteration, JSON interception, duplicate removal, CSV appending, and browser teardown. In methodological terms, the system combines browser automation with response interception rather than relying on a conventional platform API. This suggests a design optimized for large-scale, repeated retrieval under interface-mediated constraints.

3. Query construction, rate limits, and deduplication

Each search string uses grouped OR clauses over tracked keywords and hashtags together with explicit temporal bounds via until: and since: parameters (Balasubramanian et al., 2024). The documented query structure is:

(“thedemocrats” OR “DNC” OR “Kamala Harris” OR …) until:YYYY-MM-DD_hh:mm:ss_UTC since:YYYY-MM-DD_hh:mm:ss_UTC

Keywords and hashtags are grouped into comma-separated OR clauses. Appendix A.2 is identified as containing the full list of 60+ tracked terms, with examples including Biden, Trump2024, MAGA, bidenharris2024, Third Party, and RFK Jr.

Rate-limit handling is made explicit. Each account is capped at approximately $R_0 \simeq 2{,}300$ posts per hour, and with $N$ accounts the total throughput is

$R_{\mathrm{total}} = N \times R_0.$

Deduplication and stopping criteria are also specified. The scraper monitors output-file size at each iteration; if no new posts arrive for $P$ consecutive checks, scraping on that query stops. Before appending each batch, tweet_id is checked against an in-memory hash to prevent duplicates.

One interpretive caution follows directly from this design: the dataset is not described as an unconstrained sample of all election discourse on X, but as a keyword-targeted corpus. This suggests that observed prevalence patterns reflect both platform activity and the query vocabulary used to retrieve it.

4. Data model, normalization, and export structure

Each post is represented as a JSON object and can also be exported to CSV (Balasubramanian et al., 2024). The primary fields include identifiers, text forms, timestamps, author information, engagement counts, thread metadata, hashtags, user mentions, shared links, media attachments, reply metadata, embedded quoted or retweeted objects, and additional contextual attributes such as location, user, viewCount, and cash_app_handle.

Field name	Data type	Description
id	string	Unique tweet identifier
text	string	Normalized tweet text
rawContent	string	Original unnormalized text
epoch	integer	Creation time as Unix timestamp
epoch_dt	datetime	Human-readable timestamp
user_id	string	Tweet author’s user ID
lang	string	ISO language code
replyCount	integer	Number of replies
retweetCount	integer	Number of retweets
likeCount	integer	Number of likes
quoteCount	integer	Number of quotes
conversationId	string	ID of the conversation thread
hashtags	list[string]	Hashtags mentioned
mentionedUsers	list[string]	Usernames mentioned
links	list[string]	URLs shared
media	list[object]	Media attachments
in_reply_to_*	string	Reply metadata
quotedTweet, retweetedTweet	object	Embedded tweet objects

All fields are nullable as objects when absent. In CSV exports, lists are encoded as pipe-delimited strings. Prior to export, text is normalized to NFC Unicode and zero-width characters are removed.

The coexistence of normalized and raw text, together with nullable embedded objects and thread identifiers, indicates support for both linguistic preprocessing and structural reconstruction. A plausible implication is that the corpus can be used for analyses ranging from lexical frequency estimation to conversation-level and retweet-level graph construction without requiring extensive schema redesign.

5. Preliminary descriptive statistics

The paper reports an initial frequency analysis for the May–July 2024 subcorpus and identifies the most prominent keywords and hashtags (Balasubramanian et al., 2024). The top 10 keywords are:

Keyword	Count
Biden	11,070,686
Trump	5,730,951
MAGA	2,648,869
Joe	2,136,631
President	1,671,385
GOP	1,560,280
Donald	1,534,099
Harris	1,169,948
kamala	984,206
conservative	806,766

The top 10 hashtags are:

Hashtag	Count
#maga	559,722
#trump2024	341,792
#trump	170,923
#bidenharris2024	166,669
#biden	165,277
#donaldtrump	40,780
#biden2024	39,588
#gop	33,615
#joebiden	31,620
#usa	30,764

A high-level engagement summary is also reported across the full 22 million tweets:

Metric	Mean	Std. Dev.	Median
retweetCount	5.1	23.4	0
likeCount	12.7	57.3	1
replyCount	0.8	3.5	0
quoteCount	0.3	2.1	0

The paper notes that medians of zero reflect the long-tail nature of social media engagement. In analytic terms, the combination of very high-frequency political keywords and zero-median engagement metrics is consistent with a corpus that mixes a relatively small number of highly amplified posts with a very large mass of low-engagement content.

6. Research uses and analytical implications

The dataset is presented as supporting a wide range of political-communication and computational-social-science inquiries (Balasubramanian et al., 2024). The examples explicitly identified are misinformation propagation, sentiment dynamics, network analysis, bot and inauthentic-activity detection, topic modeling and event detection, and cross-platform coordination.

The paper specifies these application modes as follows:

• Misinformation propagation: trace rumor cascades around the July 13 assassination attempt
• Sentiment dynamics: model shifts in public mood before/after conventions using temporal sentiment classifiers
• Network analysis: reconstruct retweet/mention graphs to detect echo chambers and cross-ideological bridges
• Bot and inauthentic-activity detection: compare verified vs. unverified accounts over time
• Topic modeling & event detection: identify emerging subthemes such as Third Party and KAG with dynamic LDA or BERTopic
• Cross-platform coordination: link the X.com data to companion TikTok and Telegram datasets for multi-modal analyses

These use cases follow directly from the dataset’s scale, temporal coverage, and metadata richness. At the same time, one analytical constraint is implicit in the collection design: because retrieval is based on tracked terms and publicly available posts, any claim about platform-wide discourse should be interpreted relative to that observation strategy. This does not diminish the value of the resource; rather, it defines the conditions under which inference is best grounded. Within those conditions, the dataset offers a technically detailed and operationally transparent basis for studying election-related discourse in the lead-up to the 2024 U.S. Presidential Election.