ArgCMV: Argument Summarization Benchmark

• ArgCMV is an argument summarization benchmark that extracts concise, salient key points from multi-turn, online debates from Reddit’s r/ChangeMyView.
• It comprises 12,262 arguments across 3,131 topics and uses a two-step LLM-based pipeline for both key point extraction and multi-label mapping.
• Benchmark results show that modern models like gemma-2-9b outperform traditional methods, highlighting challenges in coverage and context-aware summarization.

Searching arXiv for the ArgCMV paper and closely related argument summarization / KPA work to ground the article in current literature. ArgCMV is a benchmark for argument summarization via key point extraction introduced in “ArgCMV: An Argument Summarization Benchmark for the LLM-era” (Gurjar et al., 27 Aug 2025). It is centered on Key Point Analysis (KPA), where systems identify concise, salient, high-level summaries of arguments from stance-specific collections of argumentative text. ArgCMV was created to move beyond the long-dominant ArgKP21 setting toward a benchmark grounded in actual online human debate, specifically Reddit’s r/ChangeMyView (CMV), and to better reflect the long-context, conversational, and multi-key-point conditions that characterize contemporary LLM-oriented summarization research (Gurjar et al., 27 Aug 2025).

1. Definition and task scope

ArgCMV is an argument summarization benchmark centered on key point extraction, also described in the paper using the terms Argument Summarization (ArgSum), Key Point Analysis (KPA), and key point matching or mapping (Gurjar et al., 27 Aug 2025). In this formulation, the objective is not generic abstractive summarization, but the identification of short, salient, reusable key points (KPs) that capture the major lines of reasoning expressed in argumentative discourse.

The dataset supports both key point extraction or generation and key point mapping or matching. Extraction operates over a stance-specific pool of arguments for a topic and produces a set of concise KPs; mapping links individual arguments to the key points they express (Gurjar et al., 27 Aug 2025). The mapping setup is explicitly multi-label: an argument may map to multiple KPs, and the benchmark is designed around that reality rather than assuming one dominant key point per argument (Gurjar et al., 27 Aug 2025).

ArgCMV contains 12,262 arguments, drawn from 3,131 topics across 4,387 debates (Gurjar et al., 27 Aug 2025). Its split is performed randomly at the topic level in an 80/10/10 ratio, yielding 9,845 training arguments, 1,172 development arguments, and 1,245 test arguments (Gurjar et al., 27 Aug 2025). These scale characteristics place it substantially above ArgKP21 in topical breadth and overall dataset size, while also shifting the task toward long-context online discussion (Gurjar et al., 27 Aug 2025).

A central property of the benchmark is that it is built from actual CMV debate branches rather than curated one-shot arguments. The source discussions are therefore naturally occurring, interactive, and dialogic, which introduces co-reference, follow-up rebuttals, subjective language, and context dependence (Gurjar et al., 27 Aug 2025). This is the defining conceptual move of ArgCMV relative to prior KPA benchmarks.

2. Motivation and relationship to prior benchmarks

The benchmark is motivated by what the paper characterizes as major limitations of ArgKP21 (Gurjar et al., 27 Aug 2025). ArgKP21 contains around 7K crowd-sourced arguments on a small set of controversial topics, with expert-written KPs and argument–KP mappings, but it is described as too short, too non-conversational, and too weak a test of long-context understanding for the LLM era (Gurjar et al., 27 Aug 2025).

The most concrete contrast concerns argument length. ArgCMV arguments average 196.75 tokens, whereas ArgKP21 arguments average 19.61 tokens (Gurjar et al., 27 Aug 2025). The paper also reports mean numbers of Argumentative Discourse Units (ADUs), unique ADUs, and ADU entropy: ArgCMV has means of 4.27, 2.09, and 0.87, compared with 1.22, 1.17, and 0.24 for ArgKP21, with all differences reported as statistically significant at $p < 0.05$ (Gurjar et al., 27 Aug 2025). These figures support the claim that ArgCMV is structurally richer and more discourse-complex.

The paper further argues that ArgKP21 largely treats arguments as independent units and therefore misses sequential reasoning, reply structure, context carryover, and co-reference to earlier discussion (Gurjar et al., 27 Aug 2025). ArgCMV is explicitly intended to capture those phenomena by using debate branches from CMV where an original poster and a single interlocutor engage in back-and-forth exchange (Gurjar et al., 27 Aug 2025).

Another major distinction is key point multiplicity. ArgCMV has $2.80 \pm 1.76$ KPs per argument, whereas ArgKP21 has $0.76 \pm 0.52$ (Gurjar et al., 27 Aug 2025). The paper states that over 95% of ArgKP21 arguments have a single or no KP, while in ArgCMV about 75% of arguments have more than one KP (Gurjar et al., 27 Aug 2025). This directly changes the modeling assumptions that are viable: methods tailored to one-KP-per-argument settings are expected to fail mainly through poor recall and incomplete coverage on ArgCMV (Gurjar et al., 27 Aug 2025).

The benchmark is therefore positioned as a next-generation dataset for long-context, conversation-aware key point analysis. This suggests a broader shift in KPA evaluation from sentence-level, curated argumentative units to discourse-level, naturally occurring online debate. Related work on earlier KPA benchmarks provides the background against which this shift is defined (Gurjar et al., 27 Aug 2025).

3. Data source and construction pipeline

ArgCMV is built from Reddit’s r/ChangeMyView, using CMV data crawled from January 2020 to December 2020 (Gurjar et al., 27 Aug 2025). The authors restrict the corpus to dialogue-only threads, meaning branches with only two unique users, and include only branches where the original poster (OP) makes at least one comment (Gurjar et al., 27 Aug 2025). A branch is defined as a chain of replies from the root post to a leaf comment (Gurjar et al., 27 Aug 2025).

Stance is assigned structurally. All OP-authored messages, including the root post and later replies, are treated as pro with stance $+1$, while all messages by the other user are treated as con with stance $-1$ (Gurjar et al., 27 Aug 2025). This yields two stance-specific argument pools per thread.

The dataset is produced through a two-step LLM-based pipeline, followed by manual human validation (Gurjar et al., 27 Aug 2025). The first stage uses an ExtractionAgent to generate key points from a stance-specific pool of arguments; the second uses a MappingAgent to map each individual argument to the key points it expresses (Gurjar et al., 27 Aug 2025).

For extraction, all arguments from one stance in a thread are collected and passed to the ExtractionAgent, which returns a concise set of KPs for that stance (Gurjar et al., 27 Aug 2025). This is performed separately for the $+1$ and $-1$ pools. After model-selection experiments on ArgKP21, the authors choose gpt-4o-mini as the ExtractionAgent (Gurjar et al., 27 Aug 2025).

For mapping, the MappingAgent is given one argument at a time together with the KP list for the same stance and decides which KPs are expressed in that argument (Gurjar et al., 27 Aug 2025). The one-argument-at-a-time design is explicitly intended to reduce hallucination (Gurjar et al., 27 Aug 2025). The chosen mapping model is gpt-4o, selected because on ArgKP21 matching it achieved a higher average F1 than gpt-4o-mini: 0.8362 versus 0.7883 (Gurjar et al., 27 Aug 2025).

The prompts are specified in some detail. The system prompt for generation describes the model as a “professional debater and an expert at identifying concise, high-level reasoning patterns in extended argumentative discourse,” and asks for salient, non-overlapping key points, each no longer than {kp_token_length} tokens (Gurjar et al., 27 Aug 2025). The user prompt requests short, salient, non-overlapping stance-specific KPs for a topic, instructs the model to extract multiple KPs if needed, and requires each KP to appear on a new line preceded by - (Gurjar et al., 27 Aug 2025). The mapping prompt defines a KP as present when “the main idea is expressed clearly, even if reworded,” and asks for a Python-style list of matching KP indices such as [0, 2] (Gurjar et al., 27 Aug 2025).

4. Annotation validation and dataset characteristics

The first and second authors manually validate LLM outputs on a random sample of 100 arguments jointly, assessing KP Precision, KP Recall, and KP-Redundancy (Gurjar et al., 27 Aug 2025). They then each validate 100 additional arguments independently, bringing the manually validated total to 300 arguments, about 2.5% of the dataset (Gurjar et al., 27 Aug 2025).

The reported annotation statistics are as follows:

Metric	Mean annotator 1	Mean annotator 2
KP Precision	82.47	92.41
KP Recall	84.90	91.91

For agreement, the paper reports Krippendorff’s $\alpha$ of 0.66 for KP Precision and 0.65 for KP Recall, along with Pearson correlations of 0.445 and 0.530 respectively, and interprets $\alpha \approx 0.65$–0.66 as substantial agreement (Gurjar et al., 27 Aug 2025). It also reports almost no redundant KPs (Gurjar et al., 27 Aug 2025).

The paper introduces an additional metric, KP (Extraction) Coverage, intended to test exhaustiveness at the stance/topic level (Gurjar et al., 27 Aug 2025). The label is 0 if all human-identified KPs in the set of arguments were extracted by the LLM and 1 otherwise (Gurjar et al., 27 Aug 2025). Two annotators evaluate about 130 arguments from 70 topics and find that the LLM missed at least one human-identified KP in 15.7% and 17.1% of cases respectively; agreement is Cohen’s $\kappa = 0.33$, which the authors describe as minimal agreement due to the difficulty of exhaustiveness judgments on long arguments (Gurjar et al., 27 Aug 2025).

The benchmark’s structural properties are central to its design. From Table 3, ArgCMV has 12,262 arguments and 3,131 topics, compared with 6,549 arguments and 31 topics in ArgKP21 (Gurjar et al., 27 Aug 2025). It also reports $2.80 \pm 1.76$0 KPs per argument and $2.80 \pm 1.76$1 turns per debate (Gurjar et al., 27 Aug 2025). These quantities reflect not merely a larger dataset but a different discourse regime: longer, multi-turn, context-dependent, and multi-KP argumentation.

The paper analyzes discourse composition using ADU types from Morio et al.—Fact, Policy, Rhetorical, Testimony, and Value—labeled with the ADU mining model from Mirzakhmedova et al. (Gurjar et al., 27 Aug 2025). The main findings are that ArgCMV contains more Rhetorical and Testimony units, which are “almost absent” in ArgKP21, while ArgKP21 has a higher proportion of Policy units; the distributional difference is significant by $2.80 \pm 1.76$2 test with $2.80 \pm 1.76$3 (Gurjar et al., 27 Aug 2025). This provides a concrete explanation for why ArgCMV is harder: testimony and rhetorical discourse are less cleanly propositional and often more dependent on conversational context.

The paper also classifies topics into 7 high-level categories using few-shot prompting with Gemma2, with Politics and Sociology having the highest frequencies (Gurjar et al., 27 Aug 2025). This wide topical coverage reduces dependence on narrow issue templates and makes the benchmark more representative of open-domain online debate.

5. Benchmark formulation, baselines, and evaluation

The core benchmark task takes as input arguments associated with a topic and stance and requires a set of salient, concise, non-overlapping KPs summarizing the major reasoning (Gurjar et al., 27 Aug 2025). In the dataset-construction pipeline, extraction input is the full stance-specific pool for a thread and extraction output is the KP list for that stance; mapping input is one argument plus the corresponding stance-specific KP list, and output is the set of matching KP indices (Gurjar et al., 27 Aug 2025).

Although the benchmark supports a generation-based or abstractive view of key point extraction, the paper evaluates a mixture of generation, matching, clustering, and graph-partitioning approaches (Gurjar et al., 27 Aug 2025). One important baseline is the pairwise generation plus graph partitioning method of Li et al. (2024), originally framed as shared-KP detection for argument pairs (Gurjar et al., 27 Aug 2025). For an argument pair $2.80 \pm 1.76$4, the baseline uses the input

$2.80 \pm 1.76$5

and returns either a shared KP or “No.” (Gurjar et al., 27 Aug 2025). Because multiple shared KPs between a pair are common in ArgCMV, the authors adapt the method by choosing one output KP through a four-step procedure that tracks already-used KPs, removes them when possible, and otherwise samples from the available shared set (Gurjar et al., 27 Aug 2025).

They also modify the graph partitioning stage. Li et al.’s original method initializes K-Means using the number of ground-truth KPs for each topic-stance pair, but this fails on ArgCMV because the number of KPs can be higher than the number of arguments (Gurjar et al., 27 Aug 2025). The initialization is therefore changed to half the number of arguments in the graph, and topics on which the algorithm produces no KPs are excluded from metric computation for fairness (Gurjar et al., 27 Aug 2025).

The paper reports several baseline families: SKPM from Li et al. (2023), the Li et al. (2024) pairwise generation plus graph partitioning approach, and instruction-tuned open-source models including gemma-2-9b, llama-3-8b, and mistral-nemo-2407 (Gurjar et al., 27 Aug 2025). For the open models, prompts are adapted from Altemeyer et al. (2025), but with the important change that models may generate multiple KPs per cluster; instead of external clustering, the paper groups all arguments from the same user together (Gurjar et al., 27 Aug 2025).

Evaluation uses ROUGE-1, ROUGE-2, soft Precision (sP), soft Recall (sR), and soft F1 (sF1) (Gurjar et al., 27 Aug 2025). For ROUGE, all generated KPs are concatenated and compared with the concatenation of all reference KPs (Gurjar et al., 27 Aug 2025). The semantic soft metrics use BLEURT-20 similarity (Gurjar et al., 27 Aug 2025), with intended formulas: $2.80 \pm 1.76$6

$2.80 \pm 1.76$7

$2.80 \pm 1.76$8

(Gurjar et al., 27 Aug 2025).

These metrics are particularly important for ArgCMV because recall and coverage are central failure modes in a setting where multiple KPs per argument are common. A plausible implication is that lexical overlap alone is insufficient for this benchmark: semantic coverage is a necessary dimension of evaluation.

6. Empirical results and significance

The paper first uses ArgKP21 for model selection. On that dataset, gpt-4o-mini is reported as competitive in few-shot mode, with R-1 39.31, R-2 10.87, sP 64.15, sR 64.07, and sF1 64.08, outperforming the Li et al. baseline on the soft semantic metrics though not SKPM Flan-T5-xxl (Gurjar et al., 27 Aug 2025). This result is used to justify its selection as the ExtractionAgent in the ArgCMV construction pipeline (Gurjar et al., 27 Aug 2025).

The main benchmark results on ArgCMV are the paper’s central empirical contribution (Gurjar et al., 27 Aug 2025):

Model	R-1	R-2	sP	sR	sF1
Li et al. Flan-T5-base	21.96	6.62	47.74	42.0	44.48
Li et al. Flan-T5-large	17.50	6.39	60.61	41.13	48.69
gemma-2-9b	51.77	21.20	60.76	60.85	60.76
llama-3-8b	41.81	14.93	57.07	56.92	56.95
mistral-nemo-2407	47.67	17.91	54.92	54.47	54.62

The ranking by sF1 is: gemma-2-9b first with 60.76, followed by llama-3-8b with 56.95, mistral-nemo-2407 with 54.62, Li et al. Flan-T5-large with 48.69, and Li et al. Flan-T5-base with 44.48 (Gurjar et al., 27 Aug 2025).

The headline conclusion is that methods successful on ArgKP21 do not transfer well to ArgCMV (Gurjar et al., 27 Aug 2025). The paper quantifies this degradation for the Li et al. models: Flan-T5-large drops from 61.66 sF1 on ArgKP21 to 48.69 on ArgCMV, and Flan-T5-base drops from 60.37 to 44.48 (Gurjar et al., 27 Aug 2025). This is interpreted as evidence that progress measured on ArgKP21 can overstate real-world capability in long-context, conversational KPA.

Recall is identified as the major bottleneck. For example, Li et al. Flan-T5-large on ArgCMV achieves sP 60.61 but only sR 41.13 (Gurjar et al., 27 Aug 2025). The paper interprets this as indicating that the model can generate some KPs semantically close to the references but fails to cover many of the reference KPs (Gurjar et al., 27 Aug 2025). This is consistent with the dataset’s multi-KP structure and with its broader claim that insufficient coverage is the dominant failure mode in realistic KPA settings.

The strongest model in the paper is gemma-2-9b, with ROUGE-1 51.77, ROUGE-2 21.20, sP 60.76, sR 60.85, and sF1 60.76 (Gurjar et al., 27 Aug 2025). The fact that an instruction-tuned open model outperforms the older graph-partitioning baseline is presented as evidence that modern open models have substantial out-of-the-box ability on realistic KPA (Gurjar et al., 27 Aug 2025). This also suggests, in an inferential sense, that architectural and instruction-tuning advances matter at least as much as benchmark-specific pipeline engineering when the task shifts to long-context online discourse.

7. Research significance, limitations, and broader implications

The paper presents four primary contributions: a new ArgSum dataset for key point extraction built from actual multi-turn, long-context human conversations; evidence for ArgKP21’s limitations; a benchmark of existing and modern models showing failure to adapt; and public release of train/dev/test splits and mappings for future research (Gurjar et al., 27 Aug 2025). These contributions make ArgCMV significant as an evaluation resource rather than as a single model or algorithm.

Its main importance lies in reframing argument summarization around realistic online discussion. The benchmark is explicitly relevant to long-context KPA, LLM-based summarization research, and online discussion summarization, with potential use cases including moderation support, content curation, thread navigation, debate analytics, and recommendation systems (Gurjar et al., 27 Aug 2025). A plausible implication is that ArgCMV may serve as a stress test for systems claiming strong reasoning-aware summarization ability, because it forces simultaneous handling of discourse structure, stance, subjectivity, and coverage.

The benchmark also demonstrates a hybrid dataset-creation paradigm: LLM labeling + human validation (Gurjar et al., 27 Aug 2025). This is methodologically important because fully manual KPA annotation is expensive, while fully automatic pipelines can be noisy. ArgCMV offers an example of an intermediate route that preserves scale while retaining some direct human quality checks.

The paper identifies several limitations. Some topics contain only a few arguments per stance because some Reddit threads have low engagement (Gurjar et al., 27 Aug 2025). The data span only January–December 2020, which limits temporal breadth (Gurjar et al., 27 Aug 2025). Because CMV is public web data, some evaluated LLMs may have seen parts of it during pretraining, creating potential leakage (Gurjar et al., 27 Aug 2025). The annotation pipeline relies on OpenAI models, with an estimated cost of around $100, and only a subset of the dataset received manual validation, so annotation imperfections may remain (Gurjar et al., 27 Aug 2025).

The ethics protocol is also explicit. The dataset uses only public Reddit data via the official API, collects data passively and post hoc, excludes deleted or moderator-removed comments, removes embedded URLs, anonymizes usernames with random strings, and releases only comment/post IDs, extracted KPs, and anonymous metadata; researchers are expected to recollect text in compliance with Reddit API policy (Gurjar et al., 27 Aug 2025). These details matter because benchmark construction from social media data is inseparable from questions of data governance and responsible reuse.

In summary, ArgCMV is best understood as a benchmark that makes key point extraction realistic, long-context, and conversation-aware (Gurjar et al., 27 Aug 2025). Its defining properties are empirical rather than rhetorical: 12,262 arguments, 3,131 topics, arguments averaging 196.75 tokens, and a distribution in which about 75% of arguments express more than one KP (Gurjar et al., 27 Aug 2025). Its main scientific claim is that existing methods, particularly those tuned to ArgKP21, do not adapt well to this environment, chiefly because they fail on coverage (Gurjar et al., 27 Aug 2025). For research on argument summarization in the LLM era, ArgCMV therefore functions both as a harder benchmark and as a reframing of what the task is supposed to measure.