---
title: 'AQEval: QA-Based Evaluation Metric'
url: https://www.emergentmind.com/topics/aqeval
type: topic
---

# AQEval: QA-Based Evaluation Metric

Searching arXiv for “AQEval” and closely related usages to ground the article in the relevant literature.
Search query: AQEval OR QAEval OR "Question-Answering as an Automatic Metric"
AQEval, in the summarization literature, refers to the same metric as QAEval: a reference-based evaluation method that measures summary content quality by converting reference information into question–answer pairs and then testing whether a candidate summary can answer those questions correctly [2010.00490]. Its central premise is that content quality is better estimated by direct information overlap than by lexical or embedding overlap alone. The acronym has subsequently been reused in other research areas, including label quality assessment, multiple-choice question usability, aesthetics-guided low-light image enhancement, and Audio Question Answering evaluation, but the best-specified and earliest usage in the provided literature is the summarization metric introduced in "Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary" [2010.00490].

## 1. Conceptual basis and naming

QAEval was proposed as a response to the limitations of traditional reference-based metrics such as ROUGE, as well as embedding-based metrics such as BERTScore and MoverScore. These methods primarily match tokens, either lexically or via embeddings, and therefore can reward topic similarity without verifying whether the same facts are conveyed. QAEval instead treats a reference summary as a set of answerable information probes. A candidate summary is scored by the proportion of those probes it can answer correctly, making the metric explicitly information-centric rather than surface-centric [2010.00490].

Within the supplied literature, the query term “AQEval” is explicitly identified with this QAEval metric. The same details also place it in relation to adjacent QA-based metrics. APES targets named entities with fill-in-the-blank questions, whereas QAEval targets noun phrases more broadly. FEQA and QAGS are distinguished as faithfulness or consistency metrics between a summary and its source document, rather than reference-based quality metrics. This suggests that AQEval/QAEval occupies a specific niche: automatic evaluation of content quality relative to human-written references, with question answering as the operational mechanism [2010.00490].

## 2. Formal definition

Let the reference summary induce a set of $N$ question–answer pairs, $Q=\{q_i\}_{i=1}^N$, each associated with a reference answer $r_i$. Given a candidate summary $C$, a machine reading comprehension model answers each question using $C$, producing a predicted answer span $s_i$ or a null answer if the question is judged unanswerable. Per-question verification follows standard SQuAD conventions: exact match or token-level $F_1$. If $s_i$ is null, the score is $0$ [2010.00490].

For token-level verification, with token sets $T(s_i)$ and $T(r_i)$, the metric uses
$$
\mathrm{Prec}_i=\frac{|T(s_i)\cap T(r_i)|}{|T(s_i)|}, \qquad
\mathrm{Rec}_i=\frac{|T(s_i)\cap T(r_i)|}{|T(r_i)|},
$$
and
$$
F1_i=\frac{2\cdot \mathrm{Prec}_i \cdot \mathrm{Rec}_i}{\mathrm{Prec}_i+\mathrm{Rec}_i}.
$$
The aggregate scores are
$$
S_{EM}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}[s_i=r_i], \qquad
S_{F1}=\frac{1}{N}\sum_{i=1}^N F1_i.
$$
When multiple reference summaries are available, scores are macro-averaged across references. The paper therefore reports two variants, QAEval-EM and QAEval-F1 [2010.00490].

The evaluation protocol follows standard summarization correlation analysis. Summary-level correlation is computed per input across systems and then averaged over inputs,
$$
\rho_{\mathrm{Sum}}=\frac{1}{M}\sum_j \mathrm{Corr}(\{(x_i^j,y_i^j)\}_{i=1}^N),
$$
whereas system-level correlation compares systems’ average metric scores with their average human scores across inputs,
$$
\rho_{\mathrm{Sys}}=\mathrm{Corr}\left(\left\{\left(\frac{1}{M}\sum_j x_i^j,\frac{1}{M}\sum_j y_i^j\right)\right\}_{i=1}^N\right).
$$
This makes AQEval a metric designed not merely for per-summary scoring, but also for ranking systems by their agreement with expert evaluation [2010.00490].

## 3. Operational pipeline

The QAEval pipeline consists of answer selection, question generation, machine reading comprehension, answer normalization, and score aggregation. Answer selection anchors the reference content. The paper compares three strategies—named entities, noun phrase chunks, and maximally-sized noun phrases—and reports a manual mapping study over 20 references and 801 questions. NP chunks yielded an average of 28.8 QA pairs per reference, 79% QA precision, and 91% SCU coverage, outperforming the alternatives in coverage of Pyramid summary content units. QAEval therefore uses NP chunks in the rest of the experiments [2010.00490].

Question generation is answer-aware. Given a sentence containing a selected answer span marked with special tokens, a learned question generation model produces a question whose answer is that span. The model is BART, fine-tuned on approximately 55k human-written QA pairs. The paper notes that model-generated questions are often more verbose than expert-written questions and frequently copy large portions of the source sentence, but they are generally clear and do not leak the answer. On a TAC’08 subset, expert-written and model-generated questions produce near-identical summary-level correlations, while model-generated questions yield slightly higher system-level correlations [2010.00490].

For answer extraction from the candidate summary, QAEval uses ELECTRA-Large fine-tuned on SQuAD 2.0, which includes unanswerable questions. Input consists of the candidate summary and a generated question; output is either an answer span or null, whichever is more probable. Verification uses standard SQuAD EM and token-level $F_1$, including lowercasing, stripping punctuation and articles, and whitespace canonicalization. No importance weighting analogous to Pyramid tiers is applied. In practical terms, NP-chunk answer selection produces roughly 20–30 questions per reference summary, and with multiple references this reaches approximately 100 or more questions per candidate summary [2010.00490].

## 4. Empirical performance and relation to Pyramid

QAEval was evaluated on TAC’08, TAC’09, and the SummEval subset of CNN/DailyMail. On TAC’08 system-level correlation to responsiveness, QAEval-EM achieved $r=.93$, $\rho=.91$, and $\tau=.76$, while QAEval-F1 achieved $r=.90$, $\rho=.88$, and $\tau=.71$. The paper explicitly notes that QAEval-EM’s $\tau=.76$ is higher than Pyramid’s $\tau=.70$ and ROUGE-2’s $\tau=.67$. On TAC’09 system-level evaluation, QAEval-F1 achieved $r=.81$, $\rho=.89$, and $\tau=.72$, while QAEval-EM achieved $\rho=.87$ and $\tau=.69$; the paper further notes that a single outlier depresses Pearson $r$, and removing it raises both EM and F1 variants to approximately $.92$–$.93$ [2010.00490].

At the summary level, performance is more mixed. On TAC’08, QAEval-F1 reached $r=.46$, $\rho=.45$, and $\tau=.36$, competitive with ROUGE-2, while QAEval-EM was lower at $r=.33$. On TAC’09, QAEval-F1 obtained $r=.50$, $\rho=.45$, and $\tau=.36$, again competitive with ROUGE-2 and ROUGE-SU4. On CNN/DailyMail system-level correlation to relevance, QAEval-EM achieved $r=.80$, $\rho=.91$, and $\tau=.77$, while QAEval-F1 achieved $r=.82$, $\rho=.91$, and $\tau=.77$, substantially above APES at $\rho=.73$ and ROUGE-1 at $\rho=.62$. On CNN/DailyMail summary-level evaluation, QAEval-F1 obtained $r=.30$, $\rho=.29$, and $\tau=.22$, top or tied with ROUGE-1 [2010.00490].

The metric’s relation to the Pyramid Method is central. QAEval approximates Pyramid’s content-unit logic by probing reference facts with QA pairs. NP-chunk selection covers 91% of SCUs and also captures additional specific details that SCUs sometimes abstract away. Empirically, the paper argues that QAEval’s system-level performance rivals or exceeds Pyramid’s correlations to responsiveness. A plausible implication is that QA-based probing can function as an automatic surrogate for a substantial part of Pyramid-style content assessment, especially when many QA probes are aggregated [2010.00490].

## 5. Error sources, bottlenecks, and upper bounds

The paper’s component analysis identifies answer selection as relatively strong and QA plus answer verification as the main bottlenecks. NP chunks achieve 91% SCU coverage, and approximately 21% of QA pairs probe information not captured by SCUs, such as specific locations. By contrast, the QA model is trained on Wikipedia-style SQuAD 2.0 and therefore encounters a substantial domain shift on summarization data. On a human-annotated evaluation set of 2.9k TAC’08 QA pairs and 2.3k CNN/DailyMail QA pairs, unanswerability classification measured by IsAns-F1 drops from 92.0% on SQuAD to 52.4% on TAC’08 and 75.3% on CNN/DailyMail [2010.00490].

Span verification is also noisy when the reference answer and predicted answer come from different texts. On examples where both model and human agree that a question is answerable, TAC’08 yields EM 30.5 and $F_1$ 47.2 under standard answer matching, even though human-judged accuracy is 84.3. CNN/DailyMail shows the same pattern: EM 33.2 and $F_1$ 52.0 versus human accuracy 86.3. The paper interprets this as evidence that SQuAD-style EM and $F_1$ undervalue correct answers under paraphrastic variation [2010.00490].

An upper-bound analysis reinforces that conclusion. On TAC’08 summary-level evaluation over 10 inputs and 4 systems, model QA plus human verification yields $r=.44$, $\rho=.45$, and $\tau=.42$, while human QA plus human verification yields $r=.58$, $\rho=.62$, and $\tau=.59$, approaching Pyramid at $r=.63$, $\rho=.69$, and $\tau=.65$. This suggests that AQEval’s conceptual design is not the principal limitation; rather, current QA and verification components constrain its realized performance, especially in multi-document summarization [2010.00490].

## 6. Subsequent reuse of the acronym

Later literature reuses the string “AQEval” or closely related variants for distinct evaluation problems. In label-noise research, AQuA introduces a benchmarking environment for label quality assessment under class-conditional, instance-dependent, synthetic, and multi-annotator noise, spanning 17 datasets across vision, NLP, tabular, and time-series modalities. It evaluates methods such as Confident Learning, AUM, SimiFeat, and CINCER with detection metrics including precision, recall, F1, AUROC, Average Precision, and downstream weighted $F_1$ after cleaning [2306.09467].

In educational assessment, SAQUET frames automatic question usability evaluation around the 19 Item-Writing Flaws rubric for multiple-choice questions. It combines text-matching detectors, NLP-based detectors, and GPT-4–assisted verification, and reports 94.13% criterion-level accuracy, 38% exact match ratio, and 5.9% Hamming Loss over 271 MCQs across Chemistry, Statistics, Computer Science, Humanities, and Healthcare [2405.20529]. In low-light image enhancement, ALL-E introduces an aesthetics-guided reinforcement learning framework in which NIMA serves as an aesthetic assessor and the reward includes an aesthetic term, a feature preservation term, and an exposure control term; in that context, “AQEval” denotes aesthetic quality evaluation rather than text or summary evaluation [2304.14610].

The most explicit later reuse appears in Audio Question Answering. The benchmark named AQEval contains 9,974 triplets of audio, question, reference answer, and model response, with five MTurk annotators per item and aggregate labels of 1.0, 0.5, or 0.0 to encode correctness and partial correctness. It was introduced alongside the AURA score, which combines an LLM-based contextual score with a CLAP-based audio entailment component and reaches an overall correlation of 61.80 with human judgments, compared with 56.64 for an LLM-only baseline and 27.86 for METEOR [2510.04934]. Taken together, these later usages show that “AQEval” has evolved from a specific QA-based summarization metric into a broader acronymic label for evaluation frameworks centered on quality, usability, or correctness across modalities.

Source: https://www.emergentmind.com/topics/aqeval