ArgQuality Dataset: Benchmarking Argument Quality

• ArgQuality is a dataset class for computational argument quality assessment, featuring explicit scalar and categorical annotations on free-text arguments.
• It supports evaluation tasks such as pointwise ranking and pairwise preference, enabling robust benchmarking with transformer-based models.
• The dataset covers diverse controversial topics and employs rigorous annotation protocols to assess relevance, persuasiveness, and overall argumentative merit.

ArgQuality is a class of datasets for computational argumentation, designed to benchmark and advance methodologies for assessing the quality of free-text arguments. These resources are characterized by explicit annotation for a scalar or categorical “quality” dimension, a focus on controversial topics, and evaluation procedures that support both pointwise ranking and pairwise preference tasks. ArgQuality datasets have driven research in automatic argument quality assessment, neural argument ranking, and the robustness of LLM-based argument evaluation under adversarial perturbation (Dhole, 29 Dec 2025, Gretz et al., 2019, Toledo et al., 2019).

1. Dataset Composition and Coverage

ArgQuality datasets cover controversial argumentation domains, presenting user-generated arguments labeled for quality. A canonical instance, as referenced in adversarial evaluation experiments (Dhole, 29 Dec 2025), contains several thousand short, stand-alone arguments on topics such as “Ban Plastic Water Bottles” and “Is porn wrong,” with each argument associated with its topic and stance. Each argument is annotated along a single quality dimension, most frequently using a three-level categorical scale:

• Low quality
• Average quality
• High quality

Datasets such as the corpus by Habernal and Gurevych (2016) include these labels, while others (e.g., IBM-ArgQ datasets) use continuous quality scores in $[0,1]$ induced by aggregation of binary crowd judgments (Gretz et al., 2019, Toledo et al., 2019).

Argument lengths are typically constrained (e.g., 8–36 words in IBM-ArgQ; estimated 30–70 tokens per argument in Habernal & Gurevych-based resources), and domain coverage is intentionally diverse but biased towards web forums, student essays, and debate platforms. These arguments are predominantly short, single-turn, and self-contained. For the Adversarial Lens evaluation, test sets consist of 75 argument pairs, each structured as a tuple: (topic, stance, chosen_argument, rejected_argument), prioritizing cases where clear quality distinctions exist (Dhole, 29 Dec 2025).

2. Annotation Protocols and Label Aggregation

ArgQuality annotations combine standard best practices in subjective content labeling with measures for reliability and transitivity. Two primary schemes are used:

• Absolute (Single-Argument) Labeling: Annotators are prompted, disregarding their own opinions, whether they would recommend the use of an argument in a speech for a given stance. Labels are binary (yes/no) or mapped to categorical buckets (low/average/high). Quality scores are then aggregated, either as the proportion of “yes” votes or via latent variable models (MACE, weighted average) to yield a score in $[0,1]$.
• Relative (Pairwise) Labeling: Annotators choose which of two arguments better supports the same stance for a given topic. Pairs are constructed to ensure same-stance, length-matched, and sufficient gold quality difference (e.g., $|q_i - q_j| \geq 0.2$). Agreement thresholds (e.g., $>70\%$ consensus) filter noisy pairs (Toledo et al., 2019).

Inter-annotator agreement is monitored via task-average Cohen’s $\kappa$ (typical values: 0.10–0.12 for quality, moderate for stance), and high transitivity (e.g., 96.2% for triplets) supports label consistency. Annotation guidelines for quality stress the presence of clear, relevant, persuasive reasoning for high-quality, relevance and logical soundness for average, and off-topic, vague, or unsupported claims for low-quality (Dhole, 29 Dec 2025, Toledo et al., 2019).

3. Dataset Structure and Access

ArgQuality datasets are released in standard CSV formats for research use. Schemas typically provide argument identifiers, motion/topic, stance, text, quality score (absolute task), and for pairs: pair identifiers, two argument IDs, gold winner, agreement percentages, and valid annotation counts (Toledo et al., 2019).

Corpus	Size (arguments)	Topics	Labels
Habernal & Gurevych	~several thousand	>10	categorical (3-point)
IBM-ArgQ-6.3kArgs	6,257	22	$[0,1]$ (absolute)
IBM-Rank-30k	30,497	71	$[0,1]$ (absolute)

Argument-pair datasets (e.g., IBM-ArgQ-14kPairs) are similarly structured, facilitating both regression and binary classification.

4. Evaluation Tasks and Metrics

The primary ArgQuality assessment tasks are argument ranking and comparative quality selection:

• Pointwise Ranking: Given a single argument, predict its quality score or categorical label.
• Pairwise Preference: Given a pair (A, B), predict which is higher quality.

The main metric is pairwise accuracy:

$$\text{Accuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}(\hat{y}_i = y_i)$$

where $y_i$ is the gold preference for pair $i$, and $\hat{y}_i$ is the model prediction (Dhole, 29 Dec 2025).

For regression/ranking, Pearson’s $r$ and Spearman’s $\rho$ between predicted and gold scores are standard (Gretz et al., 2019, Toledo et al., 2019).

In LLM-based evaluation pipelines, baseline few-shot accuracy is reported as 0.42 (original test set), with fine-tuning yielding up to 0.60 accuracy. When adversarial perturbations are applied to arguments via attention-layer token substitutions, accuracies drop to 0.34 (few-shot) and 0.57 (fine-tuned), indicating concrete performance degradation (Dhole, 29 Dec 2025).

5. Neural Modeling and Benchmark Results

Recent ArgQuality benchmarks employ transformer-based models, primarily BERT architectures:

• Regression (Pointwise): BERT-final-layer [CLS] embeddings or concatenation of last $n$ layers feed into MLPs predicting a score in $[0,1]$. Loss: MSE.
• Pairwise Classification: [CLS] argA [SEP] argB, softmax over two classes; cross-entropy loss (Toledo et al., 2019, Gretz et al., 2019).

Score aggregation methods (MACE, weighted annotator reliability) determine gold labels for supervised learning (Gretz et al., 2019). Baselines (argument length, SVR+BOW, Bi-LSTM+GloVe) are consistently outperformed by fine-tuned BERT models.

Model	WA-score r	WA-score ρ	MACE-P r	MACE-P ρ
Arg-Length	0.21	0.22	0.22	0.23
SVR+BOW	0.32	0.31	0.33	0.33
Bi-LSTM+GloVe	0.44	0.41	0.43	0.42
BERT-FT	0.51	0.47	0.52	0.50
BERT-FTₜₒₚᵢc	0.52	0.48	0.53	0.52

All improvements over baselines are statistically significant (Williams’s test, $p \ll 0.01$) (Gretz et al., 2019). Pairwise argument classifiers achieve accuracy up to 0.83–0.86 (AUC 0.89) on high-agreement benchmarks (Toledo et al., 2019).

6. Limitations and Scope

ArgQuality datasets expose several structural limitations:

• The three-point quality scale is coarse, overlooking finer distinctions of argumentative merit (Dhole, 29 Dec 2025).
• Most resources are limited to short, single-turn arguments, leaving dialogic and multi-step reasoning unrepresented.
• Arguments are sampled from a restricted range of genres (online forums, essays), introducing potential domain shift when evaluating more formal or domain-specific argumentation.
• In LLM adversarial evaluation settings, small token-level perturbations can induce unexpected changes—sometimes clarifying, sometimes degrading arguments—revealing instability in the quality metrics and label boundaries (Dhole, 29 Dec 2025).

A plausible implication is that further robustness and granularity in both annotation schemes and evaluation procedures are required to advance the reliability of argument quality assessment in both human and machine judgment.

7. Applications and Impact

ArgQuality datasets serve as foundational resources for multiple lines of research:

• Training and evaluation of neural models for argument ranking and pairwise argument classification (Gretz et al., 2019, Toledo et al., 2019).
• Calibration and analysis of LLM-based evaluators, particularly for alignment and robustness to adversarially generated counterexamples (Dhole, 29 Dec 2025).
• Empirical and theoretical study of major argument quality dimensions, including global relevance and persuasiveness, via targeted re-annotation (Gretz et al., 2019).
• Benchmarking transitivity and inter-annotator agreement in subjectively labeled language data, and the design of fine-grained aggregation and reliability adjustment techniques.

The continued development of ArgQuality datasets is central to the construction of robust, interpretable benchmarks for computational argumentation and machine-in-the-loop evaluation of persuasive communication.