Args.me: Benchmark Corpus for Argument Mining

• Args.me is a benchmark corpus comprising 48,798 premise–conclusion pairs from online socio-political debates, enabling nuanced evaluation of argument polarity classification.
• The dataset’s construction involves targeted crawling and minimal preprocessing from major debate portals to preserve natural argumentative structures.
• State-of-the-art LLM and hybrid models tested on Args.me show marked improvements, underscoring its significance for refining automated argument mining techniques.

Args.me is a benchmark corpus designed for evaluating automated argument classification systems in the domain of user-generated socio-political debate. The dataset provides a large, heterogeneous collection of real-world pro/con arguments, with each entry consisting of a conclusion (thesis) and one or more supporting or opposing premises. Its architecture, construction methodology, and utilization in LLM benchmarks have rendered it central to recent studies in argument mining (Pietroń et al., 11 Jul 2025, Pietron et al., 2024).

1. Construction and Domain Coverage

Args.me (version 1.0, cleaned) was assembled through targeted crawling and extraction heuristics across four major online debate portals: Debatewise, IDebate.org, Debatepedia, and Debate.org. The corpus contains 48,798 premise–conclusion pairs originating primarily from Idebate.org, Debatepedia, and Debatewise. The central objective is to facilitate the development and evaluation of models that can accurately classify the polarity—support (PRO) or attack (CON)—of premises relative to their conclusions within the context of socio-political debates (e.g., "Should cloning be allowed?", "Is nuclear energy safe?").

The construction pipeline involves minimal preprocessing: premises and conclusions are directly extracted and paired, with no additional sentence splitting, syntactic parsing, or schema transformations beyond the identification and separation of premises from conclusions. This corpus-level granularity ensures intact argumentative structure while retaining maximal naturalistic language features (Pietroń et al., 11 Jul 2025).

Sub-Corpus	#Conclusions	#Premises	#PRO Premises	#CON Premises
Idebate.org	5,011	13,248	6,701	6,547
Debatepedia	10,314	21,197	15,791	5,406
Debatewise	5,992	14,353	8,514	5,839
Total	21,317	48,798	30,006	17,792

2. Annotation Schema and Data Properties

Each record in Args.me comprises a conclusion (C) and at least one premise ($p_i$). Human annotators label each premise as PRO ($p_i \vdash C$) or CON ($p_i \nvdash C$), according to explicit instructions defining a premise as "a minimal unit of text providing evidence or reasoning with respect to the conclusion" (Ajjour et al., 2019). No “non-argument” or “none” label exists; every premise is by construction assumed to be argumentative, even cases where restated conclusions or off-topic remarks might inadvertently be included due to heuristic extraction (Pietroń et al., 11 Jul 2025, Pietron et al., 2024).

The original guidelines relied partly on automatic or semi-automatic cue-based heuristics per debate portal, focusing on lexical markers and contextual patterns to assign polarity. This enforced strict binary relation labeling but also introduced limits regarding granularity and potential context loss (Pietron et al., 2024).

3. Benchmarking Protocols and Data Usage

For evaluation, Args.me typically presents the premise–conclusion pair as natural language input to argument classification models. In recent studies, including those employing LLMs (e.g., LLaMA, DeepSeek, GPT-4o), no explicit train/dev/test splits are provided in the corpus itself. Each premise–thesis pair from the three targeted sub-corpora is processed by the model in prompt-based fashion, with downstream metrics computed on the merged dataset. In workflow configurations using transformer models such as BERT or DistilBERT, preprocessing follows standard tokenization protocols (e.g., BPE or WordPiece); premise and conclusion are concatenated using simple wrappers or prompts without supplementary structure (Pietroń et al., 11 Jul 2025, Pietron et al., 2024).

Hybrid architectures augment compact transformers with LLM refinement: when confidence is low (as measured by class probability margins above a dataset-defined threshold), a natural-language prompt is generated and delegated to ChatGPT-4 for decision refinement. This routing aims to balance classification throughput against the computational and API costs of large models (Pietron et al., 2024).

4. Performance of State-of-the-Art Models

Recent benchmarking on Args.me, spanning compact to large-scale LLMs, yields the following micro-averaged results on the PRO-vs.-CON polarity task:

Model	Accuracy (%)	Precision (%)	Recall (%)	F1 (%)
LLaMA-1B	39.4	46.1	45.8	33.3
LLaMA-3B	60.1	61.0	60.5	58.0
LLaMA-8B	72.5	73.5	72.8	70.4
LLaMA-70B (v3.3)	86.5	85.2	86.5	85.0
DeepSeek-70B	88.9	88.0	88.4	88.0
DeepSeek-R1	90.1	93.2	88.3	90.3
GPT-4o	87.2	85.5	86.5	83.7
BERT + ChatGPT-4*	89.6	—	—	91.3

(*Metric computed with both premise and conclusion as input in (Pietron et al., 2024).)

Providing both premise and conclusion text as input significantly boosts classification effectiveness across model classes. For example, DistilBERT achieves F1 ≈ 69.3% with paired inputs (vs. 68.9% with premise alone), while the BERT + ChatGPT-4 hybrid attains F1 = 91.3%. By comparison, earlier transformer-based and LSTM-based models fell short by 10–20 percentage points, demonstrating the advantage of recent LLM-driven and hybrid architectures (Pietron et al., 2024).

5. Observed Error Patterns and Systematic Shortcomings

Systematic evaluation of LLM and transformer models on Args.me reveals a set of recurring error modalities:

• Dataset Imbalance: For example, Debatepedia has 1.67 times more PRO than CON premises; this correlates with elevated PRO false-positive rates in low-performing models.
• Conditional/Hedged Language: Phrases containing modality (e.g., "may", "could") often mislead models like DeepSeek-R1, which sometimes invert the intended polarity.
• Rhetorical Questions: Such forms, prevalent in Debatewise, significantly disrupt pattern-matching in DeepSeek-R1 but are partially robust under GPT-4o classification.
• Implicit Critique: Cases invoking implicit critique ("Gun companies fund elections…") can be erroneously categorized as neutral or the wrong polarity, exposing LLM limits in pragmatic inference.
• Minimal Premises: Very terse arguments ("Cloning can help save lives.") are erroneously labeled as non-argument by smaller models (e.g., LLaMA-1B), a sign of over-filtering (Pietroń et al., 11 Jul 2025, Pietron et al., 2024).

Weaknesses are also attributed to the data itself. The absence of a “non-argument” label, the conflation of all non-conclusion text as argumentative premises (even restated conclusions or off-topic sequences), and minimal publicly documented annotation guidelines reduce interpretability and model reliability, especially in borderline cases.

6. Future Directions and Recommendations

To address gaps identified in both data and modeling, prominent recommendations include:

1. Annotation Enrichment: Publish comprehensive protocols covering the minimal definition of "premise," handling of restated conclusions, and assignment of "non-argument" labels to off-topic or otherwise irrelevant premises.
2. Corpus Structuring: Implement standardized train/dev/test splits to facilitate reproducible fine-tuning, systematic ablation, and robust error analysis.
3. Label Taxonomy Expansion: Introduce a “non-argument” or “none” category, and incorporate challenging instances (rhetorical, hedged, sarcastic, or adversarial) to enhance model robustness.
4. Contextual Augmentation: Employ retrieval-augmented generation (RAG) or related context-enriching techniques in prompting frameworks to reduce systematic misclassification.
5. Model Enhancement: Advance prompt engineering (e.g., via Chain-of-Thoughts, Tree of Thoughts search), improve handling of logical operators and pragmatic cues, and explore alternative LLM architectures to decrease reliance on proprietary APIs (Pietroń et al., 11 Jul 2025, Pietron et al., 2024).

A plausible implication is that as dataset and annotation practices mature in tandem with modeling innovation, systematic errors—such as those arising from ambiguous, implicit, or underspecified statements—will diminish, yielding further gains in argument mining reliability.

---

References:

• (Pietroń et al., 11 Jul 2025) "A comprehensive study of LLM-based argument classification: from LLAMA through GPT-4o to Deepseek-R1"
• (Pietron et al., 2024) "Efficient argument classification with compact LLMs and ChatGPT-4 refinements"