VersionQA Benchmark Overview

• VersionQA Benchmark is a structured evaluation framework for version-aware question answering that leverages 100 curated QA pairs over 34 documents with version histories.
• It employs hierarchical graph indexing and tailored query types—content retrieval, version listing, and change detection—to overcome limitations of traditional RAG systems.
• Empirical results show VersionRAG excelled with 90% overall accuracy and perfect scores on content and version listing, highlighting its practical impact on document evolution.

VersionQA Benchmark refers to a systematic evaluation framework and dataset targeting question answering systems over versioned or evolving documents. It is integral to distinguishing version-aware question answering (QA) as a unique task, primarily motivated by the limitations of conventional retrieval-augmented generation (RAG) systems in environments where document versions and changes are crucial—such as technical documentation, regulatory materials, or codebases. The VersionQA Benchmark is most prominently cited in the context of assessing the performance of the VersionRAG framework (Huwiler et al., 9 Oct 2025).

1. Motivation and Definition

The emergence of versioned document repositories and rapid document evolution mandates QA systems capable of accurately handling version-sensitive queries. Conventional RAG architectures retrieve semantically similar content but lack mechanisms for temporal validity. Recent empirical findings show only 58–64% accuracy on version-sensitive questions with baseline RAG methods, largely due to version conflation and inability to track content evolution.

The VersionQA Benchmark is explicitly designed to address these deficits. It constitutes a manually curated set of 100 diverse question–answer pairs spanning 34 technical documents exhibiting explicit version history, structured so as to probe model sensitivity to version-specific, content, and change-related queries.

2. Benchmark Construction and Content

The curation process involves collecting documents with rich version history, constructing a hierarchical representation of their evolution (e.g., via category, document, version, content, and change nodes in a graph), and authoring QA pairs targeting three core query types:

• Content Retrieval: Focused on extracting correct content from a specific version or boundary of a document.
• Version Listing: Requiring systems to enumerate, identify, or select among available document versions.
• Change Detection: Assessing the ability to identify, describe, or infer explicit and implicit modifications between versions.

Change detection tasks present particular complexity, including implicit change detection where differences are not documented but must be inferred through semantic or structural analysis (e.g., leveraging change extraction tools such as DeepDiff).

3. Evaluation Methodology

The VersionQA Benchmark is administered by routing queries through candidate QA systems—most notably VersionRAG, naive RAG, and GraphRAG—using standardized intent classification. Query type is first assessed and retrieval is then performed via appropriate graph traversal or vector search mechanisms. Evaluation focuses on exact match accuracy for question–answer pairs, with additional breakdown by query type.

The formal evaluation metric for version-aware retrieval is:

$p(c \mid q, v)$

where $c$ is the retrieved context, $q$ the query, and $v$ the intended version scope. This contrasts with the standard $p(c \mid q)$ formulation, incorporating version constraints directly into retrieval.

4. Empirical Results and Comparative Performance

Empirical evaluation on the VersionQA Benchmark demonstrates that:

• VersionRAG achieves 90% overall accuracy, outperforming naive RAG (58%) and GraphRAG (64%).
• For content queries constrained by version, VersionRAG reaches up to 100% accuracy.
• For version listing, perfect scores are observed for VersionRAG (100%), indicating robust handling of version metadata.
• For change retrieval, especially implicit change detection, VersionRAG achieves 60% accuracy, whereas baselines score between 0–10%.

These results highlight both the necessity and effectiveness of explicit version modeling and query intent classification in QA systems for evolving documents.

5. Technical Framework Integration

The benchmark necessitates architectural modifications to standard retrieval and indexing protocols. VersionRAG incorporates hierarchical graph indexing, compressing tokenization cost by 97% relative to GraphRAG (186K tokens vs. 2,970K tokens for equivalent document sets). This reduction directly impacts deployment feasibility for large document collections subject to frequent updates, minimizing both API cost (\$0.17 vs. \$6.67 for indexing) and processing latency.

Change nodes in the graph structure are populated both from explicit changelog entries and from implicit semantic difference analysis (e.g., DeepDiff), with LLMs providing natural language descriptions of detected differences.

6. Research Impact and Implications

The VersionQA Benchmark establishes versioned document QA as a distinct research domain, characterized by temporal validity, change tracking, and hierarchical modeling requirements. Its introduction has several consequences:

• It exposes the inadequacy of generic RAG systems for tasks requiring fine-grained version awareness.
• It catalyzes interest in hybrid retrieval architectures that blend efficient graph traversal, intent classification, and advanced change detection (both explicit and latent).
• It provides a reference ground truth for model development, evaluation, and benchmarking in document-centric, revision-sensitive applications.

A plausible implication is that similar benchmarks will emerge for legal, scientific, and regulatory corpora where document provenance and change auditability are essential.

7. Availability and Standardization

The full VersionQA Benchmark is released for reproducibility and extension by the research community. Its presence enables standardized comparison across competing architectures and supports research into scalable, efficient, and accurate retrieval for evolving document sets.

QA System	Overall Accuracy	Change Detection Accuracy	Indexing Tokens (K)
VersionRAG	90%	60%	186
Naive RAG	58%	0–10%	—
GraphRAG	64%	0–10%	2,970

Conclusion

The VersionQA Benchmark is a pivotal development for version-aware QA, serving as both an evaluative ground truth and motivator for architectural innovation. By formalizing version-aware retrieval and change detection, it provides a rigorous basis for advancing retrieval-augmented generation methods in settings where document evolution is central, supporting both practical deployment and future research directions in this rapidly expanding area.