Summarization is Not Dead Yet

Abstract: The progress of LLMs has fueled claims that model-generated summaries rival or even surpass human-written references, raising questions about whether summarization remains an open research problem. We re-examine this narrative through a multi-track evaluation covering five diverse datasets and five state-of-the-art LLMs, combining controlled human assessment, bias-mitigated LLM-as-Judge protocols, factuality verification against external knowledge, and corpus-level linguistic analysis. Our findings reveal a more nuanced landscape in which human reference summaries continue to demonstrate advantages in informativeness and faithfulness, whereas LLM outputs are preferred mainly for surface-level coherence and fluency. Factuality verification indicates that human references remain more reliable, particularly for claims involving reasoning or synthesis, and linguistic analysis uncovers a pattern of stylistic homogeneity across different models. These observations suggest that current LLMs have raised the floor of summarization quality, but the ceiling of their performance remains below human capabilities.

• The paper demonstrates that despite surface improvements, human summaries outperform LLMs in informativeness, factuality, and content quality with win rates as high as 0.90.
• The methodology employs multi-track evaluations including controlled human annotation, LLM-as-Judge protocols, and factuality verification across five diverse datasets.
• The findings highlight crucial challenges like evaluation robustness, benchmark contamination, and the need for improved faithfulness verification for future summarization research.

Summarization is Not Dead Yet: Revisiting the Human Ceiling in LLM Summarization

Context and Motivation

As LLMs have advanced, assertions proliferated that model-generated summaries rival or supersede human-written references, casting doubt on whether summarization is an open research problem. This paper critically reassesses that narrative by conducting comprehensive, multi-track evaluations of five state-of-the-art LLMs (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro Preview, Qwen3.5-397B-A17B, and Kimi-K2.5) across five challenging datasets covering multiple domains, modalities, and languages. The overarching aim is to precisely delineate the gap between human and model summarization along content fidelity, factuality, linguistic diversity, and dimensional evaluation with robust methodological controls.

Summarization remains a vibrant research area, with continued growth in publication volume at major NLP conferences.

Figure 1: Number of summarization papers at major NLP venues (2020–2025), documenting persistent and growing scholarly activity.

Evaluation Protocols and Methodological Controls

Multi-Track Evaluations

The study employs a rigorous evaluation schema including:

• Controlled human annotation with dimension-specific Likert ratings (informativeness, faithfulness, coherence, conciseness) and listwise rankings;
• LLM-as-Judge protocols using cross-model, bias-mitigated multi-judge scoring (with self-judgment exclusion);
• Factuality verification using four diverse claim decomposition and verification tools (FaStFact, SAFE, FActScore, VeriScore), cross-referenced with both source and external evidence;
• Large-scale corpus linguistic analysis at the lexical, syntactic, and discourse levels.

Datasets span Chinese long-form fiction (CNNSum), English lay scientific reporting (SciNews), multi-document news (DiverseSumm), multimodal scholarly video (VISTA), and EU legal documents in 24 languages (EurLexSum), ensuring broad coverage. To mitigate contamination, a second set of controlled human references was newly created for clean comparison.

Key Findings

Human vs. LLM: Content Quality and Fidelity

Human summaries exhibit clear, statistically significant advantages in informativeness and faithfulness relative to all five evaluated LLMs, with pairwise human win rates against models ranging from 0.66 to 0.90 in crowd annotation setups.

Figure 2: Pairwise win-rate matrices from human evaluation; blue dominance by the human row in all cases.

Dimension-specific analysis reveals that models slightly edge out humans only in surface-level coherence and conciseness. However, when prompt fidelity is varied, or chain-of-thought/self-correction strategies are applied, the human edge on informativeness and faithfulness persists; model superiority on coherence shrinks as document length increases.

Figure 3: Score gap matrices from human evaluation across dimensions; human summaries consistently score higher in content-oriented dimensions.

Bias-mitigated LLM-as-Judge protocols corroborate the human annotator results, with human references achieving win rates of 0.70 to 0.95 against models under the self-exclusion paradigm.

Figure 4: Pairwise win rates from LLM-as-Judge rankings; humans consistently outperform models.

Figure 5: Score gap matrices from LLM-as-Judge; content dimensions reliably favor humans.

Factuality Assessment

Contrary to claims in the literature that LLMs hallucinate less, this study's multifaceted factuality verification finds that human references remain more reliable than LLM outputs on all datasets and metrics. The measured margins range from 0.04 to 0.13 in favor of humans, with the highest disparities occurring on reasoning- and synthesis-heavy claims. Notably, prior results favoring LLM factuality are shown to be artifacts of source-only verification, which penalizes human contextualization but often overlooks model fabrications.

Figure 6: Factuality score distributions; human summaries demonstrate higher factual consistency on average.

Figure 7: Human summaries achieve the highest factuality scores across all datasets.

The cross-lingual factuality analysis on EurLexSum further demonstrates performance heterogeneity tied to language resource availability but maintains the human advantage globally.

Linguistic Properties and Stylistic Divergence

Linguistic analyses demonstrate that LLM-generated summaries display lower lexical diversity, shallower syntactic structure, and reduced discourse-level reorganization compared to human references. Model outputs are stylistically homogeneous, largely adhering to source document order and performing less aggressive abstraction/compression.

Figure 8: Divergence between human and model summaries, $\Delta = \text{Human} - \text{Model}$: humans exceed models in lexical/syntactic richness and topic progression.

Per-model comparisons replicate these findings, with all families converging to similar stylistic means. Multilingual analysis further supports the generality of this pattern.

Implications for Downstream NLP and Theoretical Understanding

Summarization remains a bottleneck for numerous downstream applications, including retrieval-augmented generation (RAG), AI-powered search, professional knowledge management, and decision support. Any deficit in information density, factual reliability, or linguistic abstraction propagates to downstream errors—impaired index construction, retrieval, and multi-hop reasoning.

Theoretical implications include a pressing need for continued advances on:

• Faithfulness verification protocols that robustly separate hallucination from legitimate enrichment;
• Multidimensional, domain- and language-sensitive evaluation metrics (beyond surface lexical overlap);
• Data benchmark management to manage contamination and ensure reliable progression measurement.

Open Challenges and Future Directions

The study identifies three acute challenges:

• Faithfulness Verification: LLMs struggle with subtle inference and synthesis; errors include entity substitution, quantity and causal distortion, and are not reliably caught by current metrics.
• Evaluation Robustness: Existing human and LLM-based evaluators are prone to various biases (position, self-preference, fluency anchoring), and standard reference-based metrics (ROUGE, BERTScore) fail to capture many critical dimensions.
• Benchmark Contamination: Data leakage is endemic and inflates model performance estimates; contamination-aware evaluation and dynamic, arena-style, non-public benchmarks are highlighted as vital.

Conclusion

Summarization is demonstrably not a solved problem. While modern LLMs have raised the minimum performance floor, the human upper bound for informativeness, faithfulness, and linguistic richness remains unimpeached. Superficial improvements in fluency and coherence do not entail substantive progress on trustworthiness or information compression, with key quality gaps exposed under meticulous, multi-dimensional scrutiny. Summarization thus persists as an essential proving ground for research on faithful reasoning, information abstraction, and high-variance, controlled generation.

Outlook

Long-term progress will require (1) models with deeper explicit reasoning and world knowledge integration, (2) robust, modular mechanisms for factual and stylistic control, and (3) ongoing, contamination-resilient evaluation infrastructures. Moreover, as LLMs are incorporated into critical decision pipelines requiring interpretability and precise coverage, the field’s attention should remain focused on surpassing—not merely matching—human summarization capabilities.