LLM-ARC Framework Overview

• LLM-ARC Framework is a method that employs LLMs to quantify argument preservation through multi-granular evaluation across legal and scientific texts.
• It formalizes argument coverage using full-set, role-level, and atomic fact metrics to assess the integrity of summary outputs.
• Empirical analyses reveal biases such as positional preferences and domain-specific performance variations, guiding targeted improvements in summarization.

The LLM-ARC Framework refers to a series of advanced methodologies and toolkits that leverage LLMs for tasks involving argument representation, reasoning, and structured coverage analysis, primarily in high-stakes domains such as legal and scientific summarization. The paradigm centers on the rigorous quantification of how well LLM-generated outputs capture salient argumentative content and causal structure. Central to this framework is the Argument Representation Coverage (ARC) methodology, which enables structured, multi-granular evaluation of information preservation in machinic summarization and structured discourse tasks (Elaraby et al., 29 May 2025).

1. Formalism and Definitions of Argument Representation Coverage

LLM-ARC is grounded in a principled mathematical formalization designed to measure the coverage of salient arguments in generated summaries. Let $S$ be the summary output and $\mathbb{A} = \{a_1,\dots,a_n\}$ the set of gold-standard "salient" arguments in the source document. The framework defines a general coverage function: $$\Phi(\upsilon,S)\in[0,1],\quad \upsilon\in\{\mathbb{A},\,a_i,\,m_{ij}\}$$ where $a_i$ is an argument unit and $m_{ij}$ denotes its constituent atomic facts.

Levels of granularity:

• Full-set coverage:

$$ARC_{\mathrm{fullset}}(S) = \Phi(\mathbb{A},S) = \frac{\ell(\mathbb{A},S) - 1}{3}, \ \ell\in\{1,2,3,4\}$$

where $\ell$ is a 1–4 Likert score by judge, human or LLM.

• Role-level coverage:

$$ARC_{\mathrm{role}}(S) = \frac{1}{n}\sum_{i=1}^n\Phi(a_i,S)$$

with

$$\Phi(a_i,S) = \begin{cases} 1, & a_i\ \text{fully preserved in } S \ 0, & \text{otherwise} \end{cases}$$

• Atomic fact (sub-argument) coverage:

$$ARC_{\mathrm{atomic}}(S) = \frac{1}{n}\sum_{i=1}^n \frac{1}{|M_i|}\sum_{m\in M_i} \Phi(m, S)$$

with

$$\Phi(m, S) = \begin{cases} 1, & m\ \text{supported in } S \ 0, & \text{missing or unfaithful} \end{cases}$$

Additionally, per-role precision, recall, and F₁ are defined as: $$\mathrm{Precision}_{r} = \frac{|A_{gen}^r \cap A_{ref}^r|}{|A_{gen}^r|},\quad \mathrm{Recall}_{r} = \frac{|A_{gen}^r \cap A_{ref}^r|}{|A_{ref}^r|},\quad F1_{r} = 2 \frac{\mathrm{Precision}_r\,\mathrm{Recall}_r}{\mathrm{Precision}_r + \mathrm{Recall}_r}$$

2. Argument Roles, Domains, and Datasets

The LLM-ARC framework is instantiated in two principal domains, each with domain-specific argument role schemas:

• Legal domain (CANLII, IRC annotation scheme):
  • Issue: the key legal question.
  • Reason: the justification supporting the court’s decision.
  • Conclusion: the decision or ruling.
• Scientific domain (SCI-ARG/DRI annotation):
  • Own Claim: central thesis or novel contribution.
  • Background Claim: foundational or prior domain knowledge.
  • Data: empirical or experimental findings.

Datasets:

• CANLII: 1,049 long-form legal opinions (average ∼4,382 tokens). Only 7.66% of tokens are argumentative, but these comprise 66.51% of reference summary content.
• DRI/SCI-ARG: 40 scientific articles (average ∼6,505 tokens), 74.14% of input sentences are annotated with an argument role; focus on those with maximum human relevance (relevance 5).

3. Evaluation Protocols and Scoring

The evaluation pipeline for LLM-ARC involves:

1. Gold argument extraction ($\mathbb{A}$), and decomposition into atomic facts ($M_i$), using LLMs (GPT-4o) and entailment models (DeBERTa).
2. Automated or LLM-augmented judging for each granularity:
   • Full-set Likert score mapped to coverage [0,1].
   • Binary preservation at the role level.
   • Fact-by-fact verification at the atomic layer.
3. Aggregation across roles or facts to derive overall and role-specific ARC scores.

Auxiliary analyses include:

• Position bias: Pearson correlation ($\rho$) between the average argument position in the document and $ARC_{\mathrm{atomic}}$.
• Role bias: Role-specific coverage adjusted for argument prevalence, via

$$\beta_r = ARC_{\mathrm{atomic},r} \times \frac{1}{\log\left(1+\frac{|r|_D}{|\mathrm{args}|_D}\right)}$$

4. Empirical Results and Behavioral Observations

4.1 Argument Coverage

• All tested LLMs—LlaMA-3.18B-instruct, Mistral-8B-instruct, Qwen2-7B(+YARN)—significantly under-cover salient argument information in sparsely argumentative contexts (e.g., CANLII), but achieve higher coverage in densely argumentative ones (e.g., DRI).
• Most errors are omissions ("missing") rather than factual inaccuracies.
• Mistral-8B exhibits the lowest role/fact-level coverage, with LlaMA-3.18B and Qwen2-7B performing comparably better.

4.2 Positional and Role Biases

• All models show a pronounced U-shaped context-window selection bias: head/tail sentences are favored over medial ones.
• In the legal domain (CANLII), argument/fact position has a negative correlation with ARCatomic ($\rho$ from –0.23 to –0.37, $p<0.05$); in scientific texts (DRI), position bias is markedly weaker or non-significant.
• For legal summarization, "Conclusions" receive the highest coverage, followed by "Issues," with "Reasons" least preserved—consistent even after controlling for argument length and position.
• In the scientific domain, after normalization, role bias across "Own Claim," "Background Claim," and "Data" is minor, though "Own Claim" sentences dominate the input.

5. Implications, Recommendations, and Future Directions

The LLM-ARC framework highlights several systematic gaps and behavioral patterns in zero-shot LLM summarization:

• LLMs systematically omit sparsely distributed arguments and exhibit both positional and role-type preferences in the summarization process.
• To address these deficiencies, recommended strategies include:
  • Argument-aware prompting (e.g., explicitly instructing the model to preserve "issues, reasons, conclusions").
  • Multi-stage summarization pipelines (argument extraction → planning → generation).
  • Post-hoc re-ranking of summary candidates by fine-grained ARCatomic.
  • Incorporation of structured argument signals in instruction-tuning or fine-tuning, especially on legal or scientific corpora enriched with role/fact annotations.

6. Theoretical and Practical Significance

The LLM-ARC framework represents a departure from traditional n-gram overlap and unstructured scoring metrics by introducing a multi-granular, argument-aware, and annotation-driven approach to evaluation. It provides a robust substrate for future research into argument-preserving text summarization and sets a rigorous foundation for structured discourse evaluation, particularly in zero-shot or instruction-following summarization scenarios. The formalization and empirical analyses in LLM-ARC underscore the need for further research into argument-aware model design, annotation standards, and evaluation paradigms for high-stakes, content-sensitive applications in NLP (Elaraby et al., 29 May 2025).