---
title: 'SummQ: Multi-Agent Long-Document Summarization'
url: https://www.emergentmind.com/topics/summq
type: topic
---

# SummQ: Multi-Agent Long-Document Summarization

Searching arXiv for SummQ and closely related summarization work.
SummQ is a multi-agent framework for long document summarization that couples summary generation with adversarial quizzing. It is designed for a setting in which large language models tend to drop important information, invent unsupported details, and lose coherence on excessively long inputs. Its defining premise is that a good summary should contain enough information to answer carefully designed questions about the source document, so summarization quality is continuously pressure-tested through quiz generation, quiz review, and an examinee that answers using only the summary [2509.20900].

## 1. Definition, scope, and terminological context

SummQ, as introduced in 2025, is a framework for long document summarization built around collaborative intelligence between agents operating in two complementary domains: summarization and quizzing. The system uses summary generators and reviewers to create and evaluate summaries, and quiz generators and reviewers to create comprehension questions that serve as quality checks; an examinee then tests whether the summary contains the information needed to answer those questions [2509.20900].

The framework targets failure modes that are prominent in long-context summarization: information loss, factual inconsistencies, and coherence issues. Rather than treating summarization as a single-pass generation problem, it organizes the task as an iterative adversarial process in which summaries are refined in response to critiques and answerability failures [2509.20900].

In adjacent literature, the label “SummQ” also appears as shorthand for query-focused summarization, particularly in QMSum-style meeting summarization, where a system must summarize only the parts of a meeting transcript that answer a user query [2104.05938, 2305.12753]. In that broader usage, the emphasis is query-conditioned evidence selection. In the 2025 framework, by contrast, SummQ denotes a specific long-document summarization method whose central mechanism is quiz-based verification [2509.20900].

## 2. Architectural design and optimization loop

SummQ contains five classes of agents: Summary Generators $\mathcal{G}_s$, Summary Reviewers $\mathcal{R}_s$, Quiz Generators $\mathcal{G}_q$, Quiz Reviewers $\mathcal{R}_q$, and an Examinee $\mathcal{E}$ [2509.20900]. These agents operate in an iterative loop over a document $D$, beginning from empty initial states,
$$
S^{(0)} \leftarrow \emptyset,\quad Q^{(0)} \leftarrow \emptyset.
$$
At iteration $t=1,\ldots,T_{\text{iter}}$, the framework generates a candidate summary and a candidate quiz,
$$
S^{(t)} \leftarrow Generate(\mathcal{G}_s, D, S^{(t-1)}),
$$
$$
Q^{(t)} \leftarrow Generate(\mathcal{G}_q, D, Q^{(t-1)}),
$$
reviews both artifacts,
$$
F_s^{(t)} \leftarrow Review(\mathcal{R}_s, S^{(t)}, Q^{(t)}, D),
$$
$$
F_q^{(t)} \leftarrow Review(\mathcal{R}_q, Q^{(t)}, S^{(t)}, D),
$$
and then asks the examinee to answer the quiz from the summary alone,
$$
F_e^{(t)} \leftarrow TakeQuiz(\mathcal{E}, Q^{(t)}, S^{(t)}).
$$
Examinee feedback is merged into the summary-side and quiz-side feedback streams; if both feedback sets are empty, the pair is accepted and returned [2509.20900].

The paper does not define a gradient-based supervised loss for the overall framework. Instead, the optimization is agentic and prompt-driven: generators receive the original document, prior outputs, reviewer feedback, and examinee feedback, and are instructed to refine the summary or quiz accordingly [2509.20900]. This suggests that SummQ treats summarization as an iterative constrained search over coverage, faithfulness, brevity, clarity, and answerability rather than as a single decoding pass.

## 3. Summary generation, review, and adjudication

The `Generate()` procedure used by SummQ is collaborative and multi-stage. In Phase 1, each generator produces an independent draft,
$$
z_i \leftarrow Draft(g_i, D, z').
$$
The resulting set of drafts is
$$
\mathcal{Z} = \{z_1, z_2, \ldots, z_n\}.
$$
In Phase 2, an aggregator agent merges them,
$$
z_{\text{agg}} \leftarrow Aggregate(A_{\text{Agg}}, \mathcal{Z}),
$$
and in Phase 3 a ranker selects the strongest individual draft,
$$
z_{\text{best}} \leftarrow BestSelect(A_{\text{Ranker}}, \mathcal{Z}).
$$
In Phase 4, the candidates
$$
\mathcal{C} = \{z_{\text{agg}}, z_{\text{best}}\}
$$
are compared by collective voting: each generator votes for the better candidate,
$$
\text{vote}_j \leftarrow Prefer(g_j, \mathcal{C}, D),
$$
and the final output is
$$
z^* \leftarrow \arg\max_{z \in \mathcal{C}} \left|\{j : \text{vote}_j = z\}\right|.
$$
The same collaboration protocol is used for both summary generation and quiz generation [2509.20900].

Summary reviewers then inspect a candidate summary against the original document and the current quiz. Their prompt asks them to verify four criteria: Coverage, Faithful, Brevity, and Clarity. Coverage requires that at least 90% of key facts needed to answer every quiz question appear in the summary; Faithful requires no hallucinations or contradictions; Brevity requires $\leq 25\%$ of source tokens or $\leq 500$ words; Clarity requires precise, coherent language [2509.20900].

The `Review()` mechanism is also structured. Each reviewer produces annotations,
$$
\mathcal{A}_i \leftarrow Annotate(r_i, z, D),
$$
which are partitioned into Agreed issues and Contested issues. Issues flagged by at least two reviewers form
$$
\mathcal{M} \leftarrow \left\{ a \mid a \in \bigcup_{i=1}^n \mathcal{A}_i \text{ and } \left| \{ i : a \in \mathcal{A}_i \} \right| \geq 2 \right\},
$$
while those flagged by fewer than two reviewers form
$$
\mathcal{C} \leftarrow \left\{ a \mid a \in \bigcup_{i=1}^n \mathcal{A}_i \text{ and } \left| \{ i : a \in \mathcal{A}_i \} \right| < 2 \right\}.
$$
Each contested issue is debated for $T_{\text{debate}}$ rounds via
$$
Argue(r_i, c, D, z),
$$
followed by majority vote,
$$
\text{vote}_c \leftarrow MajorityVote(\mathcal{R}, c).
$$
Accepted contested issues are added to $\mathcal{K}$, and the final issue list is
$$
\mathcal{I} \leftarrow \mathcal{M} \cup \mathcal{K}.
$$
This design makes disagreement itself an explicit part of refinement rather than a discarded by-product [2509.20900].

## 4. Quizzing as verification and coverage control

Quiz generation is not ancillary in SummQ; it is the main diagnostic mechanism. Quiz generators collaboratively produce three question types—multiple-choice, true/false, and short-answer—generating 10 questions of each type together with corresponding answers, for a total of 30 QA pairs per quiz [2509.20900]. Their prompts ask for questions that cover key points, avoid trivia, are balanced across document regions, and vary in difficulty.

Quiz reviewers then verify the quiz along five dimensions: Coverage Distribution, Cognitive Depth, Format Balance, Difficulty Gradient, and Clarity / quality. The prompt requires that every major section, scene, or argument be addressed; questions be spread across beginning, middle, and end; at least 40% target remember/understand; at most 20% target evaluate/create; difficulty be distributed as 30% easy, 50% medium, 20% hard; and no repeated facts or ambiguous wording appear [2509.20900].

The examinee is central because it answers the quiz using only the summary, not the original document. For multiple-choice items it returns only A/B/C/D; for true/false it returns only True/False; for short-answer it returns a short phrase or sentence taken verbatim from the text [2509.20900]. If the summary cannot support accurate quiz answering, the failure is fed back into subsequent refinement.

This mechanism reframes summary quality as answerability. A plausible implication is that SummQ operationalizes coverage in a more discriminative way than direct summary scoring alone: instead of merely asking whether the summary “looks complete,” it tests whether the summary supports retrieval of document content across multiple question formats and difficulty levels [2509.20900].

## 5. Empirical evaluation, benchmarks, and ablations

SummQ is evaluated on three long-document summarization benchmarks: MENSA, a movie script summarization benchmark; BookSum, a long-form narrative summarization dataset; and GovReport, a government report summarization benchmark [2509.20900]. Evaluation uses ROUGE-1, ROUGE-2, ROUGE-L, BERTScore-$F_1$, LLM-as-a-Judge, and human evaluation. The BERTScore model is `bert-base-uncased` [2509.20900].

The paper reports two configurations, a single-agent variant and a multi-agent ensemble variant, with the multi-agent version performing best. On MENSA, SummQ reaches ROUGE-1 41.58, ROUGE-2 11.08, ROUGE-L 18.24, and BERTScore $F_1$ 62.76. On BookSum, it reaches 44.62 / 11.14 / 20.38 / 61.49. On GovReport, it reaches 52.79 / 18.47 / 21.76 / 65.46 [2509.20900]. The paper states that SummQ is state-of-the-art among prompting and multi-agent approaches on all three benchmarks, surpassing GPT-4o, GPT-4.1, GPT-5, o3, DeepSeek-R1, Qwen3-32B, and earlier multi-agent baselines such as HM-SR; on GovReport, supervised long-document models such as U.former, SLED, and CachED remain competitive on certain metrics, but SummQ still outperforms all prompting baselines and remains strong overall [2509.20900].

The LLM-as-a-Judge evaluation uses GPT-5 and GPT-4.1 in pairwise comparison with order reversal to reduce position bias. The reported result is that SummQ consistently wins more often than compared baselines, including GPT-4o and o3 [2509.20900]. Human evaluation is conducted on 20 randomly selected NLP papers published after June 2024, with 5 Ph.D. students comparing summaries for Informativeness, Coherence, and Factuality. SummQ defeats GPT-4o with an 88% winning rate and o3 with an 82% winning rate; after shortening and rephrasing SummQ outputs to reduce length bias, it still wins 65% against GPT-4o and 60% against o3 [2509.20900].

The ablations isolate the importance of quizzing, multiplicity of agents, and iteration count. On MENSA, removing quizzing reduces ROUGE-1 by 5.00 and BERTScore $F_1$ by 6.40 in one setting; in the multi-agent setting, removing quizzing reduces ROUGE-1 by 2.09 and $F_1$ by 2.17 [2509.20900]. Increasing agents per component from 1 to 5 improves performance from ROUGE-1 39.30 and $F_1$ 61.84, to 41.58 and 62.76 at 3 agents, and to 42.52 and 62.96 at 5 agents, indicating diminishing returns [2509.20900]. Increasing the number of iterations improves performance from 1 to 3 iterations and then plateaus or slightly degrades, with the best reported setting around 3 iterations [2509.20900].

The paper also reports that quiz coverage becomes more balanced over time. Early quizzes concentrate on the beginning and end of the document, mirroring the “lost in the middle” effect; by iteration 3, coverage is more evenly distributed, with greater attention to middle segments [2509.20900]. Quiz answer accuracy improves across multiple-choice, true/false, and short-answer items, with multiple-choice and true/false improving faster and short-answer improving more gradually [2509.20900].

## 6. Relation to adjacent summarization research, limitations, and significance

SummQ belongs to a broader line of work that treats summarization as a source-conditioned, evidence-sensitive task rather than a purely stylistic generation problem. In meeting summarization, QMSum formalized query-based multi-domain meeting summarization as
$$
p(y_{1},y_{2},\cdots,y_m \mid Q, (u_1, s_1),\cdots,(u_n, s_n)),
$$
with a locate-then-summarize pipeline because long meetings exceed model input limits [2104.05938]. A later Ranker-Generator framework argued that learning to rank utterances by pairwise and listwise objectives is better aligned with query-focused summarization than binary relevance classification or ROUGE regression, and reached 35.51 ROUGE-1, 12.23 ROUGE-2, and 31.28 ROUGE-L on QMSum [2305.12753]. Compared with that line of work, SummQ does not primarily solve evidence selection under a user query; it instead introduces quiz-based adversarial verification for global long-document summarization [2509.20900].

Its evaluation philosophy also sits near source-grounded quality estimation. SummScore proposed a Cross-Encoder metric that evaluates summaries against the original document rather than a reference, decomposing quality into Coherence, Consistency, Fluency, and Relevance [2207.04660]. PreSumm later introduced pre-summarization quality estimation from the source document alone and linked low predicted quality to coherence issues, complex content, or the absence of a clear main theme [2504.05420]. This suggests that SummQ can be read as a generative analogue of source-conditioned evaluation: instead of only predicting or scoring difficulty, it actively probes whether a summary preserves answerable information [2509.20900].

The paper notes several constraints. More agents improve quality but increase cost; more iterations help only up to a point; and backbone choice matters, although SummQ helps across GPT-4o-mini, GPT-4.1, o3, DeepSeek-R1, Qwen3-32B, GPT-5, and mixed-model ensembles [2509.20900]. The best setting is not monotone in agent count or iteration count, which indicates a quality-versus-cost tradeoff rather than an unbounded scaling law [2509.20900].

The significance of SummQ lies in its reformulation of long-document summarization as a closed-loop adversarial collaboration problem. Summary generators produce candidate compressions, quiz generators expose gaps, reviewers convert quality criteria into explicit issue lists, and the examinee tests whether the summary can support downstream question answering [2509.20900]. A plausible implication is that the framework shifts the criterion for summary adequacy from resemblance to a reference toward verifiable informational sufficiency, especially under long-context conditions where omission and unsupported detail are persistent failure modes.

Source: https://www.emergentmind.com/topics/summq