AQEval: QA-Based Evaluation Metric

• AQEval is a reference-based evaluation metric that converts summaries into question–answer pairs to assess factual content overlap.
• It employs NP-chunk selection and models like BART for question generation, ensuring comprehensive coverage of summary details.
• Empirical results show that AQEval’s system-level performance rivals Pyramid methods and exceeds traditional metrics like ROUGE.

Searching arXiv for “AQEval” and closely related usages to ground the article in the relevant literature. Search query: AQEval OR QAEval OR "Question-Answering as an Automatic Metric" AQEval, in the summarization literature, refers to the same metric as QAEval: a reference-based evaluation method that measures summary content quality by converting reference information into question–answer pairs and then testing whether a candidate summary can answer those questions correctly (Deutsch et al., 2020). Its central premise is that content quality is better estimated by direct information overlap than by lexical or embedding overlap alone. The acronym has subsequently been reused in other research areas, including label quality assessment, multiple-choice question usability, aesthetics-guided low-light image enhancement, and Audio Question Answering evaluation, but the best-specified and earliest usage in the provided literature is the summarization metric introduced in "Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary" (Deutsch et al., 2020).

1. Conceptual basis and naming

QAEval was proposed as a response to the limitations of traditional reference-based metrics such as ROUGE, as well as embedding-based metrics such as BERTScore and MoverScore. These methods primarily match tokens, either lexically or via embeddings, and therefore can reward topic similarity without verifying whether the same facts are conveyed. QAEval instead treats a reference summary as a set of answerable information probes. A candidate summary is scored by the proportion of those probes it can answer correctly, making the metric explicitly information-centric rather than surface-centric (Deutsch et al., 2020).

Within the supplied literature, the query term “AQEval” is explicitly identified with this QAEval metric. The same details also place it in relation to adjacent QA-based metrics. APES targets named entities with fill-in-the-blank questions, whereas QAEval targets noun phrases more broadly. FEQA and QAGS are distinguished as faithfulness or consistency metrics between a summary and its source document, rather than reference-based quality metrics. This suggests that AQEval/QAEval occupies a specific niche: automatic evaluation of content quality relative to human-written references, with question answering as the operational mechanism (Deutsch et al., 2020).

2. Formal definition

Let the reference summary induce a set of $N$ question–answer pairs, $Q=\{q_i\}_{i=1}^N$, each associated with a reference answer $r_i$. Given a candidate summary $C$, a machine reading comprehension model answers each question using $C$, producing a predicted answer span $s_i$ or a null answer if the question is judged unanswerable. Per-question verification follows standard SQuAD conventions: exact match or token-level $F_1$. If $s_i$ is null, the score is $0$ (Deutsch et al., 2020).

For token-level verification, with token sets $T(s_i)$ and $Q=\{q_i\}_{i=1}^N$0, the metric uses

$Q=\{q_i\}_{i=1}^N$1

and

$Q=\{q_i\}_{i=1}^N$2

The aggregate scores are

$Q=\{q_i\}_{i=1}^N$3

When multiple reference summaries are available, scores are macro-averaged across references. The paper therefore reports two variants, QAEval-EM and QAEval-F1 (Deutsch et al., 2020).

The evaluation protocol follows standard summarization correlation analysis. Summary-level correlation is computed per input across systems and then averaged over inputs,

$Q=\{q_i\}_{i=1}^N$4

whereas system-level correlation compares systems’ average metric scores with their average human scores across inputs,

$Q=\{q_i\}_{i=1}^N$5

This makes AQEval a metric designed not merely for per-summary scoring, but also for ranking systems by their agreement with expert evaluation (Deutsch et al., 2020).

3. Operational pipeline

The QAEval pipeline consists of answer selection, question generation, machine reading comprehension, answer normalization, and score aggregation. Answer selection anchors the reference content. The paper compares three strategies—named entities, noun phrase chunks, and maximally-sized noun phrases—and reports a manual mapping study over 20 references and 801 questions. NP chunks yielded an average of 28.8 QA pairs per reference, 79% QA precision, and 91% SCU coverage, outperforming the alternatives in coverage of Pyramid summary content units. QAEval therefore uses NP chunks in the rest of the experiments (Deutsch et al., 2020).

Question generation is answer-aware. Given a sentence containing a selected answer span marked with special tokens, a learned question generation model produces a question whose answer is that span. The model is BART, fine-tuned on approximately 55k human-written QA pairs. The paper notes that model-generated questions are often more verbose than expert-written questions and frequently copy large portions of the source sentence, but they are generally clear and do not leak the answer. On a TAC’08 subset, expert-written and model-generated questions produce near-identical summary-level correlations, while model-generated questions yield slightly higher system-level correlations (Deutsch et al., 2020).

For answer extraction from the candidate summary, QAEval uses ELECTRA-Large fine-tuned on SQuAD 2.0, which includes unanswerable questions. Input consists of the candidate summary and a generated question; output is either an answer span or null, whichever is more probable. Verification uses standard SQuAD EM and token-level $Q=\{q_i\}_{i=1}^N$6, including lowercasing, stripping punctuation and articles, and whitespace canonicalization. No importance weighting analogous to Pyramid tiers is applied. In practical terms, NP-chunk answer selection produces roughly 20–30 questions per reference summary, and with multiple references this reaches approximately 100 or more questions per candidate summary (Deutsch et al., 2020).

4. Empirical performance and relation to Pyramid

QAEval was evaluated on TAC’08, TAC’09, and the SummEval subset of CNN/DailyMail. On TAC’08 system-level correlation to responsiveness, QAEval-EM achieved $Q=\{q_i\}_{i=1}^N$7, $Q=\{q_i\}_{i=1}^N$8, and $Q=\{q_i\}_{i=1}^N$9, while QAEval-F1 achieved $r_i$0, $r_i$1, and $r_i$2. The paper explicitly notes that QAEval-EM’s $r_i$3 is higher than Pyramid’s $r_i$4 and ROUGE-2’s $r_i$5. On TAC’09 system-level evaluation, QAEval-F1 achieved $r_i$6, $r_i$7, and $r_i$8, while QAEval-EM achieved $r_i$9 and $C$0; the paper further notes that a single outlier depresses Pearson $C$1, and removing it raises both EM and F1 variants to approximately $C$2–$C$3 (Deutsch et al., 2020).

At the summary level, performance is more mixed. On TAC’08, QAEval-F1 reached $C$4, $C$5, and $C$6, competitive with ROUGE-2, while QAEval-EM was lower at $C$7. On TAC’09, QAEval-F1 obtained $C$8, $C$9, and $C$0, again competitive with ROUGE-2 and ROUGE-SU4. On CNN/DailyMail system-level correlation to relevance, QAEval-EM achieved $C$1, $C$2, and $C$3, while QAEval-F1 achieved $C$4, $C$5, and $C$6, substantially above APES at $C$7 and ROUGE-1 at $C$8. On CNN/DailyMail summary-level evaluation, QAEval-F1 obtained $C$9, $s_i$0, and $s_i$1, top or tied with ROUGE-1 (Deutsch et al., 2020).

The metric’s relation to the Pyramid Method is central. QAEval approximates Pyramid’s content-unit logic by probing reference facts with QA pairs. NP-chunk selection covers 91% of SCUs and also captures additional specific details that SCUs sometimes abstract away. Empirically, the paper argues that QAEval’s system-level performance rivals or exceeds Pyramid’s correlations to responsiveness. A plausible implication is that QA-based probing can function as an automatic surrogate for a substantial part of Pyramid-style content assessment, especially when many QA probes are aggregated (Deutsch et al., 2020).

5. Error sources, bottlenecks, and upper bounds

The paper’s component analysis identifies answer selection as relatively strong and QA plus answer verification as the main bottlenecks. NP chunks achieve 91% SCU coverage, and approximately 21% of QA pairs probe information not captured by SCUs, such as specific locations. By contrast, the QA model is trained on Wikipedia-style SQuAD 2.0 and therefore encounters a substantial domain shift on summarization data. On a human-annotated evaluation set of 2.9k TAC’08 QA pairs and 2.3k CNN/DailyMail QA pairs, unanswerability classification measured by IsAns-F1 drops from 92.0% on SQuAD to 52.4% on TAC’08 and 75.3% on CNN/DailyMail (Deutsch et al., 2020).

Span verification is also noisy when the reference answer and predicted answer come from different texts. On examples where both model and human agree that a question is answerable, TAC’08 yields EM 30.5 and $s_i$2 47.2 under standard answer matching, even though human-judged accuracy is 84.3. CNN/DailyMail shows the same pattern: EM 33.2 and $s_i$3 52.0 versus human accuracy 86.3. The paper interprets this as evidence that SQuAD-style EM and $s_i$4 undervalue correct answers under paraphrastic variation (Deutsch et al., 2020).

An upper-bound analysis reinforces that conclusion. On TAC’08 summary-level evaluation over 10 inputs and 4 systems, model QA plus human verification yields $s_i$5, $s_i$6, and $s_i$7, while human QA plus human verification yields $s_i$8, $s_i$9, and $F_1$0, approaching Pyramid at $F_1$1, $F_1$2, and $F_1$3. This suggests that AQEval’s conceptual design is not the principal limitation; rather, current QA and verification components constrain its realized performance, especially in multi-document summarization (Deutsch et al., 2020).

6. Subsequent reuse of the acronym

Later literature reuses the string “AQEval” or closely related variants for distinct evaluation problems. In label-noise research, AQuA introduces a benchmarking environment for label quality assessment under class-conditional, instance-dependent, synthetic, and multi-annotator noise, spanning 17 datasets across vision, NLP, tabular, and time-series modalities. It evaluates methods such as Confident Learning, AUM, SimiFeat, and CINCER with detection metrics including precision, recall, F1, AUROC, Average Precision, and downstream weighted $F_1$4 after cleaning (Goswami et al., 2023).

In educational assessment, SAQUET frames automatic question usability evaluation around the 19 Item-Writing Flaws rubric for multiple-choice questions. It combines text-matching detectors, NLP-based detectors, and GPT-4–assisted verification, and reports 94.13% criterion-level accuracy, 38% exact match ratio, and 5.9% Hamming Loss over 271 MCQs across Chemistry, Statistics, Computer Science, Humanities, and Healthcare (Moore et al., 2024). In low-light image enhancement, ALL-E introduces an aesthetics-guided reinforcement learning framework in which NIMA serves as an aesthetic assessor and the reward includes an aesthetic term, a feature preservation term, and an exposure control term; in that context, “AQEval” denotes aesthetic quality evaluation rather than text or summary evaluation (Li et al., 2023).

The most explicit later reuse appears in Audio Question Answering. The benchmark named AQEval contains 9,974 triplets of audio, question, reference answer, and model response, with five MTurk annotators per item and aggregate labels of 1.0, 0.5, or 0.0 to encode correctness and partial correctness. It was introduced alongside the AURA score, which combines an LLM-based contextual score with a CLAP-based audio entailment component and reaches an overall correlation of 61.80 with human judgments, compared with 56.64 for an LLM-only baseline and 27.86 for METEOR (Dixit et al., 6 Oct 2025). Taken together, these later usages show that “AQEval” has evolved from a specific QA-based summarization metric into a broader acronymic label for evaluation frameworks centered on quality, usability, or correctness across modalities.