ASTRA-QA: A Benchmark for Abstract Question Answering over Documents

Abstract: Document-based question answering (QA) increasingly includes abstract questions that require synthesizing scattered information from long documents or across multiple documents into coherent answers. However, this setting is still poorly supported by existing benchmarks and evaluation methods, which often lack stable abstract references or rely on coarse similarity metrics and unstable head-to-head comparisons. To alleviate this issue, we introduce ASTRA-QA, a benchmark for AbSTRAct Question Answering over documents. ASTRA-QA contains 869 QA instances over academic papers and news documents, covering five abstract question types and three controlled retrieval scopes. Each instance is equipped with explicit evaluation annotations, including answer topic sets, curated unsupported topics, and aligned evidence. Building on these annotations, ASTRA-QA assesses whether answers cover required key points and avoid unsupported content by directly scoring topic coverage and curated unsupported content, enabling scalable evaluation without exhaustive head-to-head comparisons. Experiments with representative Retrieval-Augmented Generation (RAG) methods spanning vanilla, graph-based, and hierarchical retrieval settings show that ASTRA-QA provides reference-grounded diagnostics for coverage, hallucination, and retrieval-scope robustness. Our dataset and code are available at https://xinyangsally.github.io/astra-benchmark.

• The paper introduces ASTRA-QA, a benchmark that fills the gap in abstract question answering by using reference-grounded, topic-set evaluation over academic and news documents.
• It details a systematic three-stage methodology—data collection, QA instance generation with GPT-4, and iterative refinement—to produce 869 richly annotated instances with clear evidence and hallucination sets.
• Empirical evaluations reveal critical tradeoffs between coverage and hallucination across RAG methods and demonstrate a >21× reduction in evaluation costs with interpretable, scalable metrics.

ASTRA-QA: A Benchmark for Abstract Question Answering over Documents

Motivation and Benchmark Design

The ASTRA-QA benchmark addresses a fundamental gap in Retrieval-Augmented Generation (RAG) evaluation: the absence of high-fidelity, reference-grounded resources for abstract question answering over documents. Existing datasets either target factoid questions, multi-hop reasoning, or generic long-context QA, but fail to directly evaluate synthesis-heavy, abstract queries that dominate realistic document analysis and scientific reporting settings. ASTRA-QA fills this evaluation gap by focusing on complex information synthesis, comparison, thematic organization, and temporally grounded reasoning.

ASTRA-QA uniquely structures each QA instance as a tuple $(Q, A, H, M)$, where $Q$ is the abstract question, $A$ is an answer topic set representing required key points, $H$ is a curated hallucination set of plausible but unsupported topics, and $M$ includes evidence alignment and retrieval scope metadata. This design provides both positive and negative supervision for fine-grained, reference-based evaluation—enabling explicit measurement of answer coverage and unsupported content, and supporting robust diagnostic analyses across various retrieval scenarios.

Dataset Construction and Annotation Methodology

ASTRA-QA is grounded in two document domains: academic papers (including ICLR OpenReview submissions, arXiv surveys, and corpus-tagged scientific literature) and news reports (curated via mediastack and domain-specific aggregators). The construction pipeline consists of three key stages:

1. Data Collection: Source documents are paired with reviewer comments, tags, and user-generated annotation signals to increase semantic coverage and evidence grounding.
2. QA Instance Generation: LLMs (primarily GPT-4o) are used to synthesize abstract questions and candidate topic sets, employing type-specific prompt templates for five targeted forms of abstract QA (single-document summarization, pairwise comparison, multi-way comparison, enumeration, and temporal reasoning).
3. QA Refinement: Instance quality is iteratively enhanced through LLM-in-the-loop topic enrichment, deduplication, and manual verification. Hallucination sets are assembled by cataloging rejected but plausible topics produced by the LLM, while final answer topics are strictly aligned with source evidence.

Post-filtering, the dataset comprises 869 instances (over 2,000 unique documents and 16M tokens), with a highly structured topic-set representation (4–18 topics per answer, generally 0–3 hallucination targets per question).

Topic-Based Evaluation Protocol

ASTRA-QA introduces a topic-set evaluation protocol that overcomes the limitations of traditional reference-based metrics (ROUGE, BERTScore) and the cost/instability of LLM head-to-head grading. For any system response $Y$:

• The response is parsed into a predicted topic set $\hat{A}(Q, Y)$ using LLM-based extraction with semantic normalization against the ground-truth $A$ and hallucination set $H$.
• Coverage metrics: Topic Precision (T-Prec) and Recall (T-Rec), combined via T-F1, measure which ground-truth topics are addressed and the proportion of response content that is supported.
• Hallucination metrics: $H_{topic}$ (proportion of hallucination targets included in response) and $Q$0 (binary presence/absence) assess unsupported content.

This approach enables $Q$1 independent evaluation for $Q$2 methods, enhances interpretability by explicitly tracing matched/missing topics, and drastically reduces annotation and inference costs relative to $Q$3 head-to-head comparisons. However, the method's reliability depends on careful curation of $Q$4 and $Q$5 sets and semantic robustness of the topic extraction stage.

Empirical Evaluation of RAG Methods

A broad suite of contemporary RAG approaches—including Vanilla RAG, RAPTOR, HippoRAG, various GraphRAG and LightRAG variants, KET-RAG, ArchRAG, and HiRAG—was evaluated on ASTRA-QA using the Qwen3-8B LLM backbone and nomic-embed-text for retrieval embeddings. Performance was measured under three retrieval scopes (SIMPLE, MIDDLE, HARD) to diagnose robustness against distractor evidence.

Numerical Performance Highlights

• Topic Coverage (T-F1): HippoRAG achieved the highest overall T-F1 (56.6), with HiRAG excelling in the SIMPLE retrieval setting (T-F1=68.9).
• Hallucination Control: KET-RAG yielded the lowest hallucination metrics (Htopic=4.3, Hresp=8.5) but correspondingly low coverage (T-F1=24.5), showing a precision–recall tradeoff. Methods optimizing T-F1 (e.g., HippoRAG, RAPTOR, HiRAG) generally displayed higher hallucination rates, particularly as retrieval scope broadened.
• Retrieval Sensitivity: All methods exhibited significant coverage and hallucination degradation from SIMPLE to HARD, demonstrating ASTRA-QA's capability to stress-test retrieval generalization.
• Question Type Analysis: Performance was question-type dependent—enumeration and single-sum favored higher coverage, while temporal remained most challenging (best T-F1=33.5; highest Hresp=63.2).

ASTRA-QA's topic-based rankings were consistent with head-to-head LLM judgments while achieving a >21x reduction in evaluation cost and eliminating the instability introduced by LLM presentation order.

Theoretical and Practical Implications

ASTRA-QA exposes key limitations of current RAG systems on abstract QA: coverage–hallucination tradeoffs, poor generalization across noisy or distractor-rich corpora, and varying performance by synthesis type. The topic-set evaluation protocol formalizes a critical axis of abstraction missing from prior benchmarks and metrics—the ability to diagnose answer completeness and faithfulness in a semantically interpretable way. This directly facilitates targeted development of RAG methods that can optimize for comprehensive, non-hallucinatory synthesis over long or multisource evidence.

Practically, ASTRA-QA's transparency and granular annotations may improve reproducibility and foster more interpretable assessment pipelines in scientific, technical, and policy-focused document analysis. However, overfitting to curated hallucination sets or narrow topic granularity could underrepresent some generative failure modes.

Future Directions

Key avenues for advancing RAG and LLM-based abstract QA using ASTRA-QA include:

• Developing and benchmarking dynamic retrieval architectures with better scope sensitivity and distractor robustness.
• Enhancing answer organization/discourse coherence metrics alongside topic coverage.
• Automating or semi-automating topic-set and hallucination set construction for broader domains.
• Integrating ASTRA-QA evaluations in production scientific/research workflows, especially for high-stakes document synthesis tasks.

Conclusion

ASTRA-QA establishes a new standard for evaluating abstract QA in retrieval-augmented document analysis, coupling targeted question types with explicit, reference-anchored annotation and an interpretable, scalable topic-based evaluation protocol. Current RAG methods exhibit marked deficiencies in both coverage and hallucination avoidance as retrieval complexity increases. ASTRA-QA offers both the diagnostic granularity and cost-effective evaluation paradigm required to guide future work on retrieval-augmented synthesis and to foster more robust, transparent LLM evaluation practices in research and applied settings (2605.10168).