SummQ: Multi-Agent Long-Document Summarization

• SummQ is a multi-agent long-document summarization framework that couples summary generation with adversarial quizzing to verify answerability.
• It employs five types of agents in an iterative loop to generate, review, and refine summaries and quizzes, targeting information loss, factual inconsistencies, and coherence issues.
• Empirical evaluations on benchmarks like MENSA and GovReport demonstrate state-of-the-art performance with improved ROUGE and BERTScore metrics.

Searching arXiv for SummQ and closely related summarization work. SummQ is a multi-agent framework for long document summarization that couples summary generation with adversarial quizzing. It is designed for a setting in which LLMs tend to drop important information, invent unsupported details, and lose coherence on excessively long inputs. Its defining premise is that a good summary should contain enough information to answer carefully designed questions about the source document, so summarization quality is continuously pressure-tested through quiz generation, quiz review, and an examinee that answers using only the summary (Wang et al., 25 Sep 2025).

1. Definition, scope, and terminological context

SummQ, as introduced in 2025, is a framework for long document summarization built around collaborative intelligence between agents operating in two complementary domains: summarization and quizzing. The system uses summary generators and reviewers to create and evaluate summaries, and quiz generators and reviewers to create comprehension questions that serve as quality checks; an examinee then tests whether the summary contains the information needed to answer those questions (Wang et al., 25 Sep 2025).

The framework targets failure modes that are prominent in long-context summarization: information loss, factual inconsistencies, and coherence issues. Rather than treating summarization as a single-pass generation problem, it organizes the task as an iterative adversarial process in which summaries are refined in response to critiques and answerability failures (Wang et al., 25 Sep 2025).

In adjacent literature, the label “SummQ” also appears as shorthand for query-focused summarization, particularly in QMSum-style meeting summarization, where a system must summarize only the parts of a meeting transcript that answer a user query (Zhong et al., 2021, Liu et al., 2023). In that broader usage, the emphasis is query-conditioned evidence selection. In the 2025 framework, by contrast, SummQ denotes a specific long-document summarization method whose central mechanism is quiz-based verification (Wang et al., 25 Sep 2025).

2. Architectural design and optimization loop

SummQ contains five classes of agents: Summary Generators $\mathcal{G}_s$, Summary Reviewers $\mathcal{R}_s$, Quiz Generators $\mathcal{G}_q$, Quiz Reviewers $\mathcal{R}_q$, and an Examinee $\mathcal{E}$ (Wang et al., 25 Sep 2025). These agents operate in an iterative loop over a document $D$, beginning from empty initial states,

$$S^{(0)} \leftarrow \emptyset,\quad Q^{(0)} \leftarrow \emptyset.$$

At iteration $t=1,\ldots,T_{\text{iter}}$, the framework generates a candidate summary and a candidate quiz,

$$S^{(t)} \leftarrow Generate(\mathcal{G}_s, D, S^{(t-1)}),$$

$$Q^{(t)} \leftarrow Generate(\mathcal{G}_q, D, Q^{(t-1)}),$$

reviews both artifacts,

$\mathcal{R}_s$0

$\mathcal{R}_s$1

and then asks the examinee to answer the quiz from the summary alone,

$\mathcal{R}_s$2

Examinee feedback is merged into the summary-side and quiz-side feedback streams; if both feedback sets are empty, the pair is accepted and returned (Wang et al., 25 Sep 2025).

The paper does not define a gradient-based supervised loss for the overall framework. Instead, the optimization is agentic and prompt-driven: generators receive the original document, prior outputs, reviewer feedback, and examinee feedback, and are instructed to refine the summary or quiz accordingly (Wang et al., 25 Sep 2025). This suggests that SummQ treats summarization as an iterative constrained search over coverage, faithfulness, brevity, clarity, and answerability rather than as a single decoding pass.

3. Summary generation, review, and adjudication

The Generate() procedure used by SummQ is collaborative and multi-stage. In Phase 1, each generator produces an independent draft,

$\mathcal{R}_s$3

The resulting set of drafts is

$\mathcal{R}_s$4

In Phase 2, an aggregator agent merges them,

$\mathcal{R}_s$5

and in Phase 3 a ranker selects the strongest individual draft,

$\mathcal{R}_s$6

In Phase 4, the candidates

$\mathcal{R}_s$7

are compared by collective voting: each generator votes for the better candidate,

$\mathcal{R}_s$8

and the final output is

$\mathcal{R}_s$9

The same collaboration protocol is used for both summary generation and quiz generation (Wang et al., 25 Sep 2025).

Summary reviewers then inspect a candidate summary against the original document and the current quiz. Their prompt asks them to verify four criteria: Coverage, Faithful, Brevity, and Clarity. Coverage requires that at least 90% of key facts needed to answer every quiz question appear in the summary; Faithful requires no hallucinations or contradictions; Brevity requires $\mathcal{G}_q$0 of source tokens or $\mathcal{G}_q$1 words; Clarity requires precise, coherent language (Wang et al., 25 Sep 2025).

The Review() mechanism is also structured. Each reviewer produces annotations,

$\mathcal{G}_q$2

which are partitioned into Agreed issues and Contested issues. Issues flagged by at least two reviewers form

$\mathcal{G}_q$3

while those flagged by fewer than two reviewers form

$\mathcal{G}_q$4

Each contested issue is debated for $\mathcal{G}_q$5 rounds via

$\mathcal{G}_q$6

followed by majority vote,

$\mathcal{G}_q$7

Accepted contested issues are added to $\mathcal{G}_q$8, and the final issue list is

$\mathcal{G}_q$9

This design makes disagreement itself an explicit part of refinement rather than a discarded by-product (Wang et al., 25 Sep 2025).

4. Quizzing as verification and coverage control

Quiz generation is not ancillary in SummQ; it is the main diagnostic mechanism. Quiz generators collaboratively produce three question types—multiple-choice, true/false, and short-answer—generating 10 questions of each type together with corresponding answers, for a total of 30 QA pairs per quiz (Wang et al., 25 Sep 2025). Their prompts ask for questions that cover key points, avoid trivia, are balanced across document regions, and vary in difficulty.

Quiz reviewers then verify the quiz along five dimensions: Coverage Distribution, Cognitive Depth, Format Balance, Difficulty Gradient, and Clarity / quality. The prompt requires that every major section, scene, or argument be addressed; questions be spread across beginning, middle, and end; at least 40% target remember/understand; at most 20% target evaluate/create; difficulty be distributed as 30% easy, 50% medium, 20% hard; and no repeated facts or ambiguous wording appear (Wang et al., 25 Sep 2025).

The examinee is central because it answers the quiz using only the summary, not the original document. For multiple-choice items it returns only A/B/C/D; for true/false it returns only True/False; for short-answer it returns a short phrase or sentence taken verbatim from the text (Wang et al., 25 Sep 2025). If the summary cannot support accurate quiz answering, the failure is fed back into subsequent refinement.

This mechanism reframes summary quality as answerability. A plausible implication is that SummQ operationalizes coverage in a more discriminative way than direct summary scoring alone: instead of merely asking whether the summary “looks complete,” it tests whether the summary supports retrieval of document content across multiple question formats and difficulty levels (Wang et al., 25 Sep 2025).

5. Empirical evaluation, benchmarks, and ablations

SummQ is evaluated on three long-document summarization benchmarks: MENSA, a movie script summarization benchmark; BookSum, a long-form narrative summarization dataset; and GovReport, a government report summarization benchmark (Wang et al., 25 Sep 2025). Evaluation uses ROUGE-1, ROUGE-2, ROUGE-L, BERTScore-$\mathcal{R}_q$0, LLM-as-a-Judge, and human evaluation. The BERTScore model is bert-base-uncased (Wang et al., 25 Sep 2025).

The paper reports two configurations, a single-agent variant and a multi-agent ensemble variant, with the multi-agent version performing best. On MENSA, SummQ reaches ROUGE-1 41.58, ROUGE-2 11.08, ROUGE-L 18.24, and BERTScore $\mathcal{R}_q$1 62.76. On BookSum, it reaches 44.62 / 11.14 / 20.38 / 61.49. On GovReport, it reaches 52.79 / 18.47 / 21.76 / 65.46 (Wang et al., 25 Sep 2025). The paper states that SummQ is state-of-the-art among prompting and multi-agent approaches on all three benchmarks, surpassing GPT-4o, GPT-4.1, GPT-5, o3, DeepSeek-R1, Qwen3-32B, and earlier multi-agent baselines such as HM-SR; on GovReport, supervised long-document models such as U.former, SLED, and CachED remain competitive on certain metrics, but SummQ still outperforms all prompting baselines and remains strong overall (Wang et al., 25 Sep 2025).

The LLM-as-a-Judge evaluation uses GPT-5 and GPT-4.1 in pairwise comparison with order reversal to reduce position bias. The reported result is that SummQ consistently wins more often than compared baselines, including GPT-4o and o3 (Wang et al., 25 Sep 2025). Human evaluation is conducted on 20 randomly selected NLP papers published after June 2024, with 5 Ph.D. students comparing summaries for Informativeness, Coherence, and Factuality. SummQ defeats GPT-4o with an 88% winning rate and o3 with an 82% winning rate; after shortening and rephrasing SummQ outputs to reduce length bias, it still wins 65% against GPT-4o and 60% against o3 (Wang et al., 25 Sep 2025).

The ablations isolate the importance of quizzing, multiplicity of agents, and iteration count. On MENSA, removing quizzing reduces ROUGE-1 by 5.00 and BERTScore $\mathcal{R}_q$2 by 6.40 in one setting; in the multi-agent setting, removing quizzing reduces ROUGE-1 by 2.09 and $\mathcal{R}_q$3 by 2.17 (Wang et al., 25 Sep 2025). Increasing agents per component from 1 to 5 improves performance from ROUGE-1 39.30 and $\mathcal{R}_q$4 61.84, to 41.58 and 62.76 at 3 agents, and to 42.52 and 62.96 at 5 agents, indicating diminishing returns (Wang et al., 25 Sep 2025). Increasing the number of iterations improves performance from 1 to 3 iterations and then plateaus or slightly degrades, with the best reported setting around 3 iterations (Wang et al., 25 Sep 2025).

The paper also reports that quiz coverage becomes more balanced over time. Early quizzes concentrate on the beginning and end of the document, mirroring the “lost in the middle” effect; by iteration 3, coverage is more evenly distributed, with greater attention to middle segments (Wang et al., 25 Sep 2025). Quiz answer accuracy improves across multiple-choice, true/false, and short-answer items, with multiple-choice and true/false improving faster and short-answer improving more gradually (Wang et al., 25 Sep 2025).

6. Relation to adjacent summarization research, limitations, and significance

SummQ belongs to a broader line of work that treats summarization as a source-conditioned, evidence-sensitive task rather than a purely stylistic generation problem. In meeting summarization, QMSum formalized query-based multi-domain meeting summarization as

$\mathcal{R}_q$5

with a locate-then-summarize pipeline because long meetings exceed model input limits (Zhong et al., 2021). A later Ranker-Generator framework argued that learning to rank utterances by pairwise and listwise objectives is better aligned with query-focused summarization than binary relevance classification or ROUGE regression, and reached 35.51 ROUGE-1, 12.23 ROUGE-2, and 31.28 ROUGE-L on QMSum (Liu et al., 2023). Compared with that line of work, SummQ does not primarily solve evidence selection under a user query; it instead introduces quiz-based adversarial verification for global long-document summarization (Wang et al., 25 Sep 2025).

Its evaluation philosophy also sits near source-grounded quality estimation. SummScore proposed a Cross-Encoder metric that evaluates summaries against the original document rather than a reference, decomposing quality into Coherence, Consistency, Fluency, and Relevance (Lin et al., 2022). PreSumm later introduced pre-summarization quality estimation from the source document alone and linked low predicted quality to coherence issues, complex content, or the absence of a clear main theme (Koniaev et al., 7 Apr 2025). This suggests that SummQ can be read as a generative analogue of source-conditioned evaluation: instead of only predicting or scoring difficulty, it actively probes whether a summary preserves answerable information (Wang et al., 25 Sep 2025).

The paper notes several constraints. More agents improve quality but increase cost; more iterations help only up to a point; and backbone choice matters, although SummQ helps across GPT-4o-mini, GPT-4.1, o3, DeepSeek-R1, Qwen3-32B, GPT-5, and mixed-model ensembles (Wang et al., 25 Sep 2025). The best setting is not monotone in agent count or iteration count, which indicates a quality-versus-cost tradeoff rather than an unbounded scaling law (Wang et al., 25 Sep 2025).

The significance of SummQ lies in its reformulation of long-document summarization as a closed-loop adversarial collaboration problem. Summary generators produce candidate compressions, quiz generators expose gaps, reviewers convert quality criteria into explicit issue lists, and the examinee tests whether the summary can support downstream question answering (Wang et al., 25 Sep 2025). A plausible implication is that the framework shifts the criterion for summary adequacy from resemblance to a reference toward verifiable informational sufficiency, especially under long-context conditions where omission and unsupported detail are persistent failure modes.