ArxivFullQA Benchmark

• ArxivFullQA is a benchmark for full-paper QA, evaluating seven question categories across multi-page ArXiv documents with integrated evidence retrieval.
• It combines document retrieval, structured parsing, and multimodal analysis of text, tables, and figures to challenge standard QA systems.
• The benchmark is built with a two-stage annotation pipeline that ensures consistency between LaTeX sources and visual renderings, reflecting both extractive and generative QA innovations.

Searching arXiv for the cited papers and closely related work on full-paper QA benchmarks, scientific article QA generation, and multimodal ArXiv datasets. arxiv_search queries:

1. id:(Sipilä et al., 2024)
2. id:(Azarbonyad et al., 18 Jul 2025)
3. id:(Huang et al., 21 Sep 2025)
4. id:(Li et al., 2024)
5. id:(Xiong et al., 10 Aug 2025) ArxivFullQA is a benchmark and problem setting for question answering over full scientific papers from ArXiv, introduced to evaluate multi-page document understanding under realistic scientific reading conditions. In its benchmark form, it comprises 8,648 question–answer instances drawn from full ArXiv scientific papers spanning 1–30 pages, and it targets seven question categories: factual, reasoning, comparison, summary, procedural, motivation, and result. Its stated motivation is to address a gap left by datasets centered on single-page tasks or narrow multi-page domains, by requiring coarse-to-fine retrieval, evidence localization, and multi-hop reasoning across textual, tabular, and figure content (Xiong et al., 10 Aug 2025). In related usage, ArxivFullQA also denotes an end-to-end full-paper QA pipeline extrapolated from earlier extractive QA work on scientific literature, where passage retrieval, domain-adapted encoders, and structured preprocessing are combined to extract answers from entire articles rather than short snippets (Sipilä et al., 2024).

1. Definition and problem scope

ArxivFullQA was conceived as a response to the mismatch between existing multimodal document benchmarks and the requirements of scientific paper comprehension. The benchmark explicitly targets real-world tasks in which a model must first identify which pages contain relevant evidence and then reason over content dispersed across multiple pages. The benchmark objective is two-fold: to provide a large, high-quality evaluation set of full scientific papers from ArXiv, and to cover seven distinct question categories so that evaluation is not restricted to simple fact lookup (Xiong et al., 10 Aug 2025).

The problem formulation is broader than conventional passage QA. A system is expected to handle questions such as “What is the main motivation?” and “Summarize the experimental results,” not merely recover a local text span. This places ArxivFullQA at the intersection of document retrieval, passage selection, multimodal parsing, and answer generation. A plausible implication is that the benchmark is designed not only to test answer correctness, but also to expose whether a model can recover and use the correct evidence pages before answering.

2. Antecedents in scientific extractive QA

A direct methodological precursor to ArxivFullQA is extractive QA over scientific text, as demonstrated in work on perovskite bandgap extraction from the materials-science literature. That workflow harvested publications from five repositories, beginning with approximately 238,000 raw articles and producing 194,322 unique texts after conversion failure filtering, DOI-based deduplication, and vector-based near-duplicate removal. The evaluation set consisted of 600 manually annotated seven-sentence snippets balanced across five perovskite materials; 175 snippets contained 209 bandgap mentions and 425 contained none. Each example used the fixed natural-language query template, “What is the numerical value of the bandgap of material X?”, with the answer defined as the text span containing the numeric value and “eV,” or the empty string if no value was present (Sipilä et al., 2024).

That study compared five encoder-only transformer models fine-tuned on SQuAD 2.0 for extractive QA: BERT-base, SciBERT, MatBERT, MaterialsBERT, and MatSciBERT. The input format was standard span extraction, [CLS] question tokens [[SEP](https://www.emergentmind.com/topics/semantic-entropy-production-sep-metric)] snippet tokens [SEP], with start- and end-token logits and no-answer mapped to the [CLS] position. For snippets with multiple bandgaps, the system retrieved up to top-$K$ spans with $K = 6$, filtered by a confidence threshold $\tau$ selected by four-fold cross-validation from $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$. On the held-out test set, MatSciBERT achieved the highest reported F1 in the detailed results table at $63.9 \pm 3.5$, with precision $65.4 \pm 8.0$ and recall $64.4 \pm 9.3$, while the rule-based CDE2 baseline had F1 $48.2$ and GPT-4, in the zero-shot generative comparison, reached F1 $69.8$ (Sipilä et al., 2024).

The same work explicitly generalizes from snippet QA to full-paper QA over ArXiv. The recommended pipeline begins with harvesting full PDFs or source LaTeX via the ArXiv OAI-PMH API, parsing structure with GROBID or ScienceParse, cleaning text while preserving headings and captions, and normalizing scientific notation, subscripts, superscripts, and units. It then applies a retrieval-plus-reading architecture: coarse document retrieval with BM25 or ElasticSearch, passage chunking into overlapping windows of at most 512 tokens, dense passage ranking with DPR or SBERT embeddings, and answer extraction with a fine-tuned QA encoder such as MatSciBERT-QA. The same outline also identifies table flattening, caption use for figures, coreference resolution, multi-hop retrieval, and long-context alternatives such as Longformer or BigBird as required extensions for the full-paper setting. In that sense, ArxivFullQA inherits a concrete systems blueprint from earlier scientific IE-by-QA work.

One noteworthy detail is an internal reporting tension in the source description: the abstract states that “Best extraction accuracy was achieved with the QA MatBERT,” whereas the held-out test results table reports the highest F1 for MatSciBERT. The two statements coexist in the source record (Sipilä et al., 2024).

3. Benchmark construction and annotation

In the benchmark introduced with DocR1, ArxivFullQA is built with a two-stage annotation pipeline designed to enforce agreement between source-text and rendered-document views. In Stage 1, the raw LaTeX source of each paper, extracted via DocMatrix, is used to generate one question–answer pair per sample while cycling through the seven question types. Generated answers are filtered by comparison against an internal ground truth when available, or checked for consistency such as numeric values in methodology sections. Only samples whose answers fully satisfy the prompt constraints advance to the next stage (Xiong et al., 10 Aug 2025).

In Stage 2, the same paper is rendered as images, with one to thirty page images, and the model is prompted to answer the Stage-1 question based solely on those visual pages. A sample is accepted only if the visual-format answer exactly matches the Stage-1 LaTeX-based answer, judged either by string comparison or by $\mathrm{ANLS} \geq 1.0$. Samples failing this self-consistency criterion are rejected. The result is a benchmark whose examples are grounded simultaneously in the paper’s textual source and its visual rendering (Xiong et al., 10 Aug 2025).

The benchmark examples are intended to cover the range of scientific reading tasks encountered in practice. Representative examples include a factual question asking for the dataset size used for pre-training in Section 3.1, a reasoning question asking why the authors switched to LaTeX input in Stage 1 of the annotation pipeline, a comparison question contrasting GRPO and EviGRPO on MultiChartQA, a summary question about the two-stage curriculum training strategy, a procedural question listing the validation steps in the verification stage, a motivation question about why ArxivFullQA was created, and a result question asking how many QA pairs remain after verification. These examples clarify that ArxivFullQA is not limited to atomic extraction: some questions are numerical, some are explanatory, and some require synthesis across sections or across pages.

4. Evaluation protocol and relation to evidence-guided training

ArxivFullQA is evaluated with ANLS, the Average Normalized Levenshtein Similarity, which measures string similarity between a model answer and the ground truth and takes the value $K = 6$0 for an exact match. The baselines reported on the benchmark include LLaVA-NeXT-Inter (7B), LEOPARD-Idefics2 (8B), mPLUG-DocOwl2 (8B), LLaVA-oneVision (7B), InternVL3-Instruct (38B), and Qwen2.5-VL-Instruct in 32B and 7B variants. DocR1 reports an ANLS of 40.65, while the nearest baseline reported in the overview is Qwen2.5-VL-Instruct 7B at 29.57. Because the official test split is not public, the reported ArxivFullQA numbers come from the held-out validation portion (Xiong et al., 10 Aug 2025).

The inference format is also structured. Models generate > …, <evidence_page>…</evidence_page>, and <answer>…</answer> in one shot. Page images are resized with a maximum of $K = 6$1 tokens. Evidence page accuracy may be logged, but only ANLS is reported in the main tables. This means that the benchmark operationalizes evidence selection without making evidence-page accuracy the headline metric (Xiong et al., 10 Aug 2025).

Although ArxivFullQA is an evaluation set, it is tightly coupled to the Evidence Page-Guided GRPO training regime used by DocR1. The total reward for an output $K = 6$2 is defined as

$K = 6$3

where $K = 6$4 is the ANLS score between generated and ground-truth answers, and $K = 6$5 enforces the three-part output format. The evidence-page reward uses an F1-style overlap:

$K = 6$6

Advantage normalization and a clipped surrogate objective with KL regularization are then applied during optimization. The benchmark is therefore embedded in a training philosophy that treats evidence localization and answer generation as jointly rewardable behaviors rather than separable stages (Xiong et al., 10 Aug 2025).

5. QA generation pipelines for scientific articles

A complementary line of work approaches scientific article QA not as benchmark construction but as automatic generation of question–answer pairs that capture an article’s main ideas and contributions. One pipeline, termed Core Context–Based QA Generation (CCQG), starts from a scientific article consisting of title, abstract, keywords, and author-provided paragraphs. It defines a proxy $K = 6$7, embeds $K = 6$8 and each paragraph with all-mpnet-base-v2, computes cosine similarity, and selects as salient all paragraphs with similarity at least $K = 6$9. GPT-3.5 is then prompted to generate concise, moderately specific questions from the salient paragraphs, after which questions are greedily ranked against paragraphs by embedding similarity, filtered to remove references to authors, figures, tables, or section numbers, and capped at two questions per article section. Answer generation uses GPT-3.5 again, with instructions to answer in 40–100 words, cite exact source sentences, and output JSON; SelfCheckGPT’s NLI-based hallucination detector discards answers flagged as hallucinated (Azarbonyad et al., 18 Jul 2025).

The knowledge-graph-based variant augments this with corpus-wide entity–relationship extraction. It starts from REBEL, samples a representative scientific corpus through TopicBERT clustering followed by KMeans, uses an LLM to produce gold triplets, and fine-tunes REBEL for 10 epochs with learning rate $\tau$0 and batch size 8. Extracted triplets are uploaded to Neo4j, and each triplet receives a saliency score

$\tau$1

where the graph term combines TF-IDF-like entity and triplet statistics with PageRank and the semantic term is the cosine similarity between the triplet embedding and the title–abstract–keywords embedding. The top $\tau$2 triplet groups then drive focused question generation (Azarbonyad et al., 18 Jul 2025).

The empirical results matter for ArxivFullQA because they quantify how saliency selection and domain adaptation alter QA quality over scientific articles. On SME evaluation normalized to $\tau$3, the KG-based method outperformed CCQG on question quality in both Computer Science and Life Sciences: in Computer Science, relevance, specificity, and clarity increased from $\tau$4 to $\tau$5; in Life Sciences, from $\tau$6 to $\tau$7. Fine-tuning the ER extractor raised triplet extraction accuracy from $\tau$8 to $\tau$9 and win-rate from $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$0 to $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$1. A plausible implication is that future ArxivFullQA-style systems can benefit not only from better readers, but also from better question selection and evidence saliency models.

6. Position within the benchmark landscape

ArxivFullQA belongs to a broader family of scientific QA resources that differ in document scope, modality, and evaluation protocol. AirQA is a human-annotated comprehensive paper QA dataset in AI with 13,948 papers and 1,246 questions, covering single-doc detail, multi-doc analysis, conference-level retrieval, and comprehensive retrieval-plus-detail questions. Its modal breakdown includes text-only, table-based, image/figure-based, formula-based, and metadata examples, and it uses instance-level evaluation in which each example is scored by a custom function yielding 0 or 1; the benchmark defines 19 custom functions, spanning objective, subjective, and logical-combinator evaluators (Huang et al., 21 Sep 2025). ArXivQA, by contrast, is a figure-centered dataset generated from scientific figures and captions: 100,000 multiple-choice QA pairs drawn from 32,000 unique figures and 16,600 papers across 32 domains, with question types distributed across descriptive, inferential, and quantitative figure reasoning (Li et al., 2024).

Resource	Coverage	Distinctive evaluation or design
ArxivFullQA	8,648 QA pairs from full ArXiv papers, 1–30 pages, 7 question categories	ANLS on multi-page scientific documents (Xiong et al., 10 Aug 2025)
AirQA	13,948 AI papers, 1,246 human-written questions, multi-task and multi-modal	Instance-level scoring with 19 custom functions (Huang et al., 21 Sep 2025)
ArXivQA	100,000 MCQs from 32,000 figures and 16,600 papers across 32 domains	Figure-grounded multiple-choice QA with rationale (Li et al., 2024)

These resources illuminate different aspects of the same problem space. AirQA emphasizes realistic agent workflows, with single-turn QA, multi-turn dialogue, and tool use through Retrieve, Query, and Answer, backed by DuckDB, a Milvus vectorstore, and optional image viewing. Its reported results show that trivial text-only or title/abstract prompting yields less than $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$2, RAG gives about $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$3, and tool-enabled agentic methods reach about $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$4, while Gemini-2.5-Pro reports the highest average score at $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$5 on the benchmark table (Huang et al., 21 Sep 2025). ArXivQA instead isolates scientific visual understanding; fine-tuning Qwen-VL-Chat-7B for three epochs on ArXivQA raises overall performance on MathVista testmini from $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$6 to $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$7, with large gains in Geometry and Textbook QA (Li et al., 2024). This suggests that ArxivFullQA occupies the full-document end of a spectrum whose other endpoints are agentic corpus navigation and figure-specific reasoning.

7. Limitations and open directions

Several limitations are explicit in the surrounding literature. For ArxivFullQA itself, reported results are on the held-out validation portion because the official test split is not public (Xiong et al., 10 Aug 2025). In the scientific article QA-generation setting, SME evaluation covered 200 QAs across two domains, the paragraph saliency threshold of $\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$8 and top-$\{0.0125, 0.025, 0.05, 0.1, 0.2, 0.4, 0.8\}$9 choices were set empirically, and no formal statistical significance tests were reported (Azarbonyad et al., 18 Jul 2025). In AirQA, the remaining gaps are identified as cross-paper multi-hop planning, multi-modal reasoning, and error-robust action generation (Huang et al., 21 Sep 2025).

On the systems side, long-document context remains a structural bottleneck because standard BERT is limited to 512 tokens, and formula-heavy text, figures and tables with embedded numbers, hallucinations in generative pipelines, and scalability to approximately 2 million ArXiv papers all require specialized handling (Sipilä et al., 2024). For scientific figures, error analysis in Multimodal ArXiv found context misinterpretation, recognition errors, and over-simplification, and the recommended remedies include OCR grounding, metadata conditioning, broader question types, targeted sampling for low-resource domains, and multi-figure reasoning chains (Li et al., 2024).

Taken together, these results delineate ArxivFullQA as a convergent research target rather than a single isolated benchmark. It combines the retrieval demands of full-paper QA, the evidence-grounding demands of document understanding, the saliency and novelty modeling explored by knowledge-graph-based QA generation, and the modality-specific requirements surfaced by figure, table, formula, and metadata benchmarks. A plausible implication is that progress on ArxivFullQA will depend less on any single model family than on the integration of evidence-page retrieval, structured article parsing, multimodal grounding, and task-specific evaluation.