RooseBERT: Political NLP Model

• RooseBERT is a pre-trained language model specialized for political debates, trained on a custom 5 GB corpus spanning 1946 to 2024.
• It employs both continued pre-training and from-scratch approaches with a domain-specific vocabulary to enhance tasks like sentiment analysis, named entity recognition, and argument mining.
• Empirical results show RooseBERT outperforms general-purpose models in perplexity and downstream performance, confirming its effectiveness for computational political discourse analysis.

Searching arXiv for RooseBERT and cited related models to ground the article with current paper IDs. Searching for the RooseBERT paper. Searching for ConfliBERT and PoliBERTweet. RooseBERT is a domain-specific pre-trained LLM for political discourse, introduced to address the linguistic specificity of political debates and politics-related discussions, including hidden communication strategies and implicit arguments. It was trained on a custom 5 GB English-language corpus spanning 1946 through 2024 and evaluated on four downstream tasks central to political debate analysis: named entity recognition, sentiment analysis, argument component detection and classification, and argument relation prediction and classification. The reported results show improvements over general-purpose LLMs on these tasks, positioning RooseBERT as a specialized BERT-family model for computational political discourse analysis (Dore et al., 5 Aug 2025).

1. Scope, naming, and conceptual position

RooseBERT is a pre-trained LLM specialized for political debate and speech corpora rather than a modification of the BERT encoder itself. The model is presented in Base and Large configurations and follows BERT’s Transformer encoder design; the principal novelty lies in domain-specific pre-training on political data and, for one variant, a custom vocabulary derived from that domain (Dore et al., 5 Aug 2025).

A recurring misconception is to conflate RooseBERT with similarly named BERT variants developed for unrelated purposes. In particular, it is distinct from RomeBERT, which addresses robust training of multi-exit BERT models and focuses on dynamic early exiting, gradient-regularized self-distillation, and inference-time efficiency rather than political language modelling (Geng et al., 2021). RooseBERT instead targets political discourse understanding, with downstream evaluation centered on entity recognition, sentiment, and argument mining (Dore et al., 5 Aug 2025).

The paper distinguishes multiple RooseBERT variants through the labels “cont” and “scr.” The text explicitly identifies the “scr” variant as “from-scratch” and notes that it uses a custom WordPiece tokenizer trained on the concatenation of all eight corpus sources. This suggests that “cont” denotes continued pre-training on the political corpus, whereas “scr” denotes training with a domain-specific vocabulary built from scratch (Dore et al., 5 Aug 2025).

2. Corpus construction and preprocessing

RooseBERT was trained on a custom-assembled 5 GB corpus of English-language transcripts drawn from eight major sources of “real-world” debates and speeches. The corpus comprises roughly 8 000 debate “documents,” each of which may contain several sub-debates or topical segments. The temporal coverage extends from 1946 through 2024, and the sources are heterogeneous across electoral, parliamentary, and multilateral settings (Dore et al., 5 Aug 2025).

The eight sources are as follows:

• US Presidential Debates (1960 – 2024): harvested from the American Presidency Project, including primary and general election debates.
• United Nations General Debate Corpus (UNGDC, 1946 – 2023): over 10 000 speeches from 202 member states.
• House of Commons Parliamentary Debates (1979 – 2019): more than 50 000 debates and 2.5 million speeches.
• Australian Parliamentary Debates (1998 – 2022): House of Representatives.
• EUSpeech (2007 – 2015): 18 403 speeches by EU leaders in English.
• ParlEE Plenary Speeches (2009 – 2019): eight EU chambers, retaining only UK and Ireland.
• ParlScot (up to 2021): 1.8 million spoken contributions to the Scottish Parliament.
• UN Security Council Debates (1992 – 2023): 106 302 speeches from 6 233 meetings (Dore et al., 5 Aug 2025).

Preprocessing consisted of stripping HTML and markup, removing hyperlinks or boilerplate, normalizing punctuation and whitespace, and splitting text into sentences for batching while respecting speaker turns. For the from-scratch variant, a custom WordPiece tokenizer was trained on the concatenation of all eight sources. The paper explicitly notes that, although it refers to “sub-debates” in the abstract, no additional discourse segmentation beyond speaker-turn and sentence splitting was applied. This design ensured that each turn was seen in context, up to the maximum sequence lengths used during training (Dore et al., 5 Aug 2025).

3. Architecture and learning objectives

RooseBERT retains exactly the same Transformer encoder design as BERT in both Base and Large configurations. The Base model has 12 Transformer layers, hidden size 768, 12 attention heads, and approximately 110 million parameters. The Large model has 24 layers, hidden size 1024, 16 attention heads, and approximately 340 million parameters. Apart from the domain corpus and, for the “scr” variant, the vocabulary, the architecture is unchanged (Dore et al., 5 Aug 2025).

A concise summary of the released configurations is given below.

Variant	Encoder specification	Distinguishing feature
Base	12 layers, hidden size 768, 12 heads, ≈ 110 M parameters	cont and scr variants
Large	24 layers, hidden size 1024, 16 heads, ≈ 340 M parameters	cont and scr variants

The pre-training objective follows BERT’s Masked Language Modeling (MLM) setup. If $X=(x_1,\dots,x_T)$ is a token sequence and $M\subset\{1,\dots,T\}$ is the set of masked positions, the loss is

$$\mathcal{L}_{\mathrm{MLM}} \;=\; -\sum_{i\in M}\log P_\theta(x_i\mid x_{\backslash M})\,.$$

Here $x_{\backslash M}$ denotes the sequence with masked tokens at indices $M$ (Dore et al., 5 Aug 2025).

The model initially also adopted the Next Sentence Prediction (NSP) objective,

$$\mathcal{L}_{\mathrm{NSP}} =- \sum_{j}\log P_\theta\bigl(\mathrm{IsNext}\mid \, s_j^{(1)}, s_j^{(2)}\bigr)\,,$$

but early experiments found that NSP did not yield gains. The final RooseBERT models therefore optimize only MLM. The paper states that omitting $\mathcal{L}_{\mathrm{NSP}}$ both accelerated training and slightly improved downstream performance, aligning the final design more closely with later BERT-family practice (Dore et al., 5 Aug 2025).

4. Pre-training regime and systems setup

All pre-training runs used HuggingFace’s Transformers library with DeepSpeed ZeRO-2, mixed precision in FP16, and gradient accumulation. Training was conducted on eight NVIDIA A100 GPUs. The Base-size runs used a grid search to identify the best-performing configuration (Dore et al., 5 Aug 2025).

For Base-size RooseBERT, the reported hyperparameters were: 150 000 total steps for RooseBERT-cont and 250 000 for RooseBERT-scr; batch size of 2 048 tokens per step; learning rate $1\times 10^{-4}$ with 10 000 warmup steps followed by linear decay; weight decay 0.01; Adam with $\beta_1=0.9$, $\beta_2=0.98$, and $M\subset\{1,\dots,T\}$0; sequence lengths allocated as 80% of steps at 128 tokens and 20% at 512 tokens; and masking probability 15%, with random mask token, random word, or unchanged token as in BERT (Dore et al., 5 Aug 2025).

The reported training times for Base size were 18 hours on eight A100s for RooseBERT-cont at 150 K steps and 31 hours on eight A100s for RooseBERT-scr at 250 K steps. For the Large-size variants, the schedule for warmup, decay, and masking remained identical, but the total number of steps was reduced to 125 000 for both cont and scr because of compute constraints. Total Large-model training time was approximately 35 hours on eight A100s (Dore et al., 5 Aug 2025).

These details matter because the paper frames RooseBERT not as an architectural departure from BERT but as a domain-adapted pre-training program carried out with a modern large-scale training stack. A plausible implication is that the observed gains are attributable primarily to corpus specialization and vocabulary adaptation rather than to new encoder mechanics.

5. Fine-tuning tasks and benchmark design

RooseBERT was evaluated against BERT-base, BERT-large, ConfliBERT, and PoliBERTweet on four downstream tasks relevant to political debate analysis. The experimental protocol adds either a token-classification head for named entity recognition and argument-component tagging or a sequence-classification head for sentiment analysis and argument-relation tasks. Fine-tuning always used the publicly released train, development, and test splits and a small hyperparameter sweep over learning rates $M\subset\{1,\dots,T\}$1, weight decay $M\subset\{1,\dots,T\}$2, batch sizes $M\subset\{1,\dots,T\}$3, epochs $M\subset\{1,\dots,T\}$4, and maximum sequence lengths $M\subset\{1,\dots,T\}$5, fixed at 256 and 128 for Large and PoliBERTweet variants respectively (Dore et al., 5 Aug 2025).

The downstream tasks are summarized below.

Task	Dataset	Reported metric
Named Entity Recognition	CrossNER-Politics	macro-F1
Sentiment Analysis	ParlVote	F1 on test
Argument Component Detection & Classification	ElecDeb60to20	macro-F1
Argument Relation Prediction & Classification	ElecDeb60to20	macro-F1

The task-specific annotations are nontrivial in scale. CrossNER-Politics contains six entity types—Politician, Person, Org., Party, Election, and Country—and approximately 5 000 sentences across train, development, and test. ParlVote contains 34 010 UK parliamentary speeches labelled Positive versus Negative, with balanced classes of approximately 17.7 K positive and 15.7 K negative instances. ElecDeb60to20 for argument components is built from 44 televised U.S. presidential debates from 1960 to 2020 and contains 38 667 argument components, comprising 25 078 claims and 13 589 premises, evaluated with token-level BIO tagging and Claim/Premise labels. The relation task uses 25 524 annotated relations, including 21 689 support and 3 835 attack instances, and classifies each gold component pair as Support or Attack (Dore et al., 5 Aug 2025).

6. Empirical results, interpretation, and limitations

On a held-out 1% test split of the combined political corpora, RooseBERT achieved the lowest reported perplexity among the compared models. For cased perplexity, the table reports: BERT-base 32 317.8, ConfliBERT-cont 4.418, ConfliBERT-scr 4.299, PoliBERTweet 131.451, RooseBERT-cont 2.580, and RooseBERT-scr 2.789. For uncased perplexity, it reports: BERT-base 25 107.0, ConfliBERT-cont 4.960, ConfliBERT-scr 4.223, RooseBERT-cont 2.651, and RooseBERT-scr 2.848. The paper interprets this as confirmation that RooseBERT provides the strongest fit to the political-debate distribution among the evaluated models (Dore et al., 5 Aug 2025).

For Base-size downstream results, the reported averages over five seeds show the following. On named entity recognition, BERT-base reaches 0.516, ConfliBERT ranges from 0.492 to 0.518, PoliBERT-t reaches 0.338, Roose-cont reaches 0.477, and Roose-scr reaches 0.499. On sentiment, BERT-base reaches 0.690, ConfliBERT 0.691, PoliBERT-t 0.654, Roose-cont 0.726, and Roose-scr 0.728. On argument components, BERT-base reaches 0.466, ConfliBERT 0.464, PoliBERT-t 0.443, Roose-cont 0.473, and Roose-scr 0.480. On argument relations, BERT-base reaches 0.662, ConfliBERT 0.659, PoliBERT-t 0.667, Roose-cont 0.663, and Roose-scr 0.674. The paper further states that RooseBERT-scr-uncased significantly outperforms all baselines on sentiment with $M\subset\{1,\dots,T\}$6 versus BERT-uncased and ConfliBERT-scr, on argument component detection with $M\subset\{1,\dots,T\}$7, and on argument relation classification with $M\subset\{1,\dots,T\}$8 (Dore et al., 5 Aug 2025).

For Large-size models, the reported macro-F$M\subset\{1,\dots,T\}$9 values are: NER, BERT-large 0.594, Roose-cont 0.553, Roose-scr 0.572; Sentiment, BERT-large 0.691, Roose-cont 0.718, Roose-scr 0.718; Argument Component, BERT-large 0.469, Roose-cont 0.475, Roose-scr 0.475; and Argument Relation, BERT-large 0.576, Roose-cont 0.499, Roose-scr 0.695. The paper highlights that RooseBERT-scr-cased (Large) improves Argument Relation to 0.695 macro-F$$\mathcal{L}_{\mathrm{MLM}} \;=\; -\sum_{i\in M}\log P_\theta(x_i\mid x_{\backslash M})\,.$$0, described as a +2.1 point absolute gain over BERT-large (Dore et al., 5 Aug 2025).

The study also reports a prompt-based comparison against gemma-3-4B-it, Mistral-7B, Llama-3-8B, and GPT-4-nano under zero-shot and few-shot settings. None of these models matched RooseBERT on the argument-mining tasks, even when given three demonstration examples. Within the paper’s interpretation, the largest gains are concentrated in sentiment analysis and argument mining, where specialized vocabulary such as “rebuttal,” “endorsement,” and “bureaucrat,” together with cross-speaker interactions, are presented as especially important (Dore et al., 5 Aug 2025).

The limitations are explicit. All corpora and annotations are English-only. The model relies on standard MLM, leaving span-masking or discourse-aware objectives as open directions. Large models produce only modest gains in some settings while incurring substantial compute cost, motivating future exploration of parameter-efficient fine-tuning such as LoRA and adapters. The release of the 5 GB political corpus and the RooseBERT weights—Base and Large, cont and scr, cased and uncased—is framed as infrastructure for further work in computational political discourse analysis (Dore et al., 5 Aug 2025).