LLM-ReSum: Iterative Self-Reflective Summaries

• LLM-ReSum is a self-reflective summarization framework that iteratively refines generated summaries using evaluation-driven feedback without model finetuning.
• It employs an iterative refinement process using quality scores based on clarity, accuracy, coverage, and overall quality, leading to substantial improvements on weak summaries.
• The framework’s meta-evaluation across news, scientific, and legal domains demonstrates that LLM-based evaluators align more closely with human judgments than traditional metrics.

Searching arXiv for the specific paper and closely related work to ground the article with current citations. arXiv search query: (Nguyen et al., 28 Apr 2026) LLM-ReSum self-reflective summarization metric evaluation LLM-ReSum is a self-reflective summarization framework in which LLMs are used not only to generate summaries but also to evaluate and revise them in a closed feedback loop without model finetuning. It is grounded in a broad meta-evaluation of summarization metrics across seven datasets spanning five domains, covering documents from short news articles to long scientific, governmental, and legal texts, and it is paired with a new legal-domain benchmark, PatentSumEval. Within this formulation, summarization is not a one-shot generation task but an iterative process driven by reference-free quality assessment along dimensions such as clarity, accuracy, coverage, and overall quality (Nguyen et al., 28 Apr 2026).

1. Definition and problem setting

LLM-ReSum addresses two linked problems. The first is an evaluation problem: existing automatic metrics often do not align with human judgments for abstractive LLM summaries, especially on long documents and specialized domains. The second is an improvement problem: even when a summary is recognized as weak, there is often no generic, scalable mechanism to repair it without training a new model, applying reinforcement learning from human feedback, or building domain-specific post-processing systems. LLM-ReSum joins these two problems by treating evaluation as a source of actionable feedback for iterative refinement (Nguyen et al., 28 Apr 2026).

The framework is explicitly reference-free at inference time. Given a source document $D$ and a current summary $S$, an evaluator produces a vector of quality scores over multiple dimensions,

$\mathbf{q}(S, D) = [q_1(S,D), \dots, q_K(S,D)],$

where each $q_i(S,D)$ is a Likert score for a quality dimension such as clarity, accuracy, coverage, or overall quality. The system then uses low-scoring dimensions and their rationales to construct feedback for the next refinement round.

The motivating empirical setting is unusually broad. The underlying meta-evaluation spans more than 1,500 human-annotated summaries across news, social media, scientific papers, government reports, and legal or patent texts. Document lengths range from roughly 2K to 27K words. This breadth is central to the framework’s argument: summarization quality cannot be reduced to a single metric or a single domain-specific heuristic.

2. Meta-evaluation of summarization metrics

The meta-evaluation underpinning LLM-ReSum compares 14 automatic metrics and several LLM-based evaluators against human judgments. The metric families include readability measures, lexical overlap metrics, embedding-based metrics, and task-specific factuality or QA-style metrics. The principal result is that no single traditional metric is reliable across domains and dimensions, while LLM-based evaluators achieve substantially higher alignment with human judgments, especially for linguistic quality; at the same time, LLM evaluators remain weak for exhaustive coverage checking on very long documents such as Arxiv and GovReport (Nguyen et al., 28 Apr 2026).

Before the table, three empirical patterns are especially important. First, classical lexical overlap metrics can correlate weakly or even negatively with human judgments for LLM-generated summaries; the paper reports, for example, BLEU on Arxiv accuracy at $\tau=-0.61$. Second, task-specific neural metrics such as SummaC-CV and BLANC can perform strongly for factual accuracy in some settings, but their behavior is unstable across domains. Third, LLM evaluators, including multi-agent aggregation, achieve the strongest reported agreement on several dimensions; majority voting reaches SummEval accuracy $\tau=0.84$.

Metric family	Examples	Reported behavior
Readability and lexical overlap	FRE, Dale-Chall, ROUGE, BLEU, METEOR, CHRF	Often weak or negative on abstractive, long, or technical summaries
Embedding-based and task-specific neural	BERTScore, BARTScore, SummaC, BLANC, QAEval, QuestEval	Often moderate to strong in some domains, but not robust across all settings
LLM-based evaluators	Qwen2-7B, Linkbricks-V6-32B, multi-agent averaging, majority voting, leader-based aggregation	Highest alignment with human judgments for several dimensions, especially linguistic quality

A common misconception follows directly from these results: high ROUGE or BLEU does not imply high human-perceived summary quality. Another misconception is the converse claim that LLM-as-judge universally solves evaluation. The paper shows that LLM evaluators can approach perfect alignment on PatentSumEval, with coverage and accuracy around $\tau \approx 0.95$–1.0, yet still perform poorly on long-document coverage and accuracy for Arxiv and GovReport.

3. Reflective summarization loop and algorithmic structure

LLM-ReSum is organized around four components: a base summarization model, one or more evaluation models, a refinement model, and a controller that manages the loop. The initial summary is produced by a standard instruction-tuned LLM. A separate evaluation prompt then scores the current summary on clarity, accuracy, coverage, and overall quality, returning both scores and short explanations in a structured format. These rationales are converted into explicit feedback, which conditions the refinement prompt for the next iteration (Nguyen et al., 28 Apr 2026).

The refinement process is iterative:

$$S^{(t+1)} = f_{\text{refine}}(S^{(t)}, D, F^{(t)}),$$

where $F^{(t)}$ is structured feedback derived from the scores and rationales at step $t$. The controller stops when all dimensions exceed a threshold or when a maximum number of iterations is reached. The paper sets the threshold at $S$0 on a 1–5 Likert scale and uses $S$1 refinement rounds.

The stopping criterion is defined by the minimum score across dimensions:

$S$2

If this condition is satisfied, the current summary is returned. Otherwise, the deficient dimensions are identified as

$S$3

and only those low-scoring aspects are fed into the next refinement prompt. If the loop reaches $S$4 without convergence, the framework returns the iteration whose worst dimension is maximized:

$S$5

The prompt design is central. The evaluation prompt is reasoning-reinforced: it requires the evaluator to justify each score with short evidence-based explanations. The refinement prompt then embeds those explanations as targeted corrective instructions. This makes the loop qualitatively different from generic “improve this summary” prompting. The model is not merely asked to rewrite; it is asked to repair specific defects identified under explicit quality dimensions.

4. Empirical performance and PatentSumEval

The framework is evaluated on news, scientific, and legal data, with 30 documents per dataset and pairwise human comparison between an Initial Summary and an Enhanced Summary. Three graduate student annotators perform the comparison, justifications are required, attention checks are applied, and inter-annotator agreement is reported as Krippendorff’s $S$6. The main outcome is asymmetric: when the initial summary is already strong, average gains are small, but when the initial summary is weak, improvements are substantial (Nguyen et al., 28 Apr 2026).

For all summaries, gains are modest because many initial outputs already score around 4.3–4.7 out of 5. On CNN, for example, clarity moves from 4.32 to 4.32, accuracy from 4.49 to 4.51, coverage from 4.14 to 4.16, and overall quality from 4.30 to 4.31. This behavior supports the interpretation that the framework does not aggressively rewrite already-good outputs.

For low-quality summaries, defined as cases with initial score below 4.0 on a dimension, the gains are much larger. On CNN, accuracy improves from 3.20 to 4.25 and coverage from 3.30 to 4.20. On Daily Mail, coverage improves from 3.00 to 4.17. On PubMed, accuracy moves from 3.00 to 4.00 and overall quality from 3.67 to 4.33. On PatentSumEval, accuracy moves from 3.00 to 4.00. Across these settings, the paper reports improvements of up to 33% in factual accuracy and 39% in coverage, with refined summaries preferred in 89% of pairwise judgments for overall quality.

PatentSumEval is integral rather than auxiliary. It is a human-annotated legal-domain benchmark built from 30 patent documents in communication and streaming technologies, with 6 model-generated summaries per patent, yielding 180 summaries total. The summarization inputs are the abstract and claims rather than full patents. Mean source length is about 9,754 words and mean summary length about 647 words. Three annotators with Master’s-level CS or engineering background evaluate each summary on a 5-point Likert scale for clarity, accuracy, coverage, and overall quality. The benchmark exposes a domain where exact terminology matters and hallucinations are especially dangerous.

5. Position within adjacent summarization paradigms

LLM-ReSum belongs to a broader family of systems that couple generation with some form of reflective control, but its mechanism is distinct. “Multi-LLM Text Summarization” organizes summarization as parallel generation plus centralized or decentralized evaluation across several models and reports improvements over single-LLM baselines by up to 3x, whereas LLM-ReSum concentrates on evaluator-guided self-refinement of a summary candidate rather than cross-model winner selection (Fang et al., 2024). “LLM Based Multi-Document Summarization Exploiting Main-Event Biased Monotone Submodular Content Extraction” constrains content selection through main-event-biased monotone submodular extraction before an LLM rewriting step, emphasizing controllable extraction over reflective revision (Kurisinkel et al., 2023). Two later systems named “ReSum” operate in different regimes: one uses self-summarization inside reinforcement learning with verifiable rewards for long-horizon reasoning and reports an average 4% improvement with 18.6% shorter rollouts (Wang et al., 11 Jun 2026), while another uses periodic context summarization for web agents and reports an average absolute improvement of 4.5% over ReAct, with additional gains after ReSum-GRPO training (Wu et al., 16 Sep 2025). This suggests that the shared terminology spans at least three distinct ideas: reflective summary improvement, reasoning-trajectory compression, and context-window management.

That distinction matters because “LLM-ReSum” is sometimes informally conflated with any LLM system that summarizes and then reuses the summary. In the 2026 framework, however, the summary is not merely compressed state. It is an object of explicit judgment under human-aligned dimensions, and revision is driven by evaluator rationales rather than by memory pressure, RL reward shaping, or multi-agent voting alone.

6. Limitations, misconceptions, and significance

Several limitations are explicit. LLM-based evaluators still struggle with long-document coverage and accuracy, particularly on Arxiv and GovReport. Evaluator bias and self-preference remain unresolved; using LLMs to judge LLM outputs risks systematic stylistic preference. The loop adds computational cost because each iteration requires evaluation and refinement, possibly with multiple judges. Empirical coverage is limited to English and a bounded set of domains. Over-refinement is a real concern, especially when already-good summaries are revised for little gain (Nguyen et al., 28 Apr 2026).

Three misconceptions are therefore corrected by the framework’s own evidence. First, lexical overlap is not a reliable proxy for summary quality across heterogeneous domains. Second, LLM-as-judge is not a universal replacement for specialized factuality metrics, especially on very long inputs. Third, reflective refinement does not uniformly improve all summaries; its largest benefits appear on low-quality initial outputs, while strong summaries often change only marginally.

The broader significance of LLM-ReSum lies in the shift from single-pass summarization to evaluation-conditioned revision. This reorients summarization around explicit quality dimensions and turns automatic evaluation into an active component of generation rather than a purely post hoc measurement. A plausible implication is that future summarization systems will be assessed less by static overlap against references and more by their ability to diagnose and repair their own failures under domain-specific quality criteria.