WikiWeb-ERP: Exploratory Reasoning Benchmark

• WikiWeb-ERP is a benchmark that evaluates retrieval-augmented generation on exploratory reasoning problems characterized by high uncertainty, ambiguity, and emergent dependencies.
• The dataset comprises 3,536 queries and 53,682 documents sourced from Wikipedia dumps and live web crawls, emphasizing non-linear query planning and structural reasoning.
• Empirical results show that PlanRAG with retrieval enhancements outperforms both iteration-based and graph-based baselines, validating the planning approach in noisy, heterogeneous evidence settings.

WikiWeb-ERP is a benchmark for retrieval-augmented generation on exploratory reasoning problems (ERPs), introduced to evaluate systems on natural-language queries characterized by high uncertainty, ambiguity, difficulty in reduction, and fuzzy, emergent dependencies between entities. Each benchmark instance combines a natural-language ERP query, a set of atomic queries derived from it, and an evidence corpus assembled from both Wikipedia dumps and live webpages crawled from the web. Its purpose is to measure performance in settings where evidence is noisy and heterogeneous and where reasoning does not collapse into a simple linear multi-hop chain (Xu et al., 1 Jul 2026).

1. Definition and research motivation

WikiWeb-ERP is presented as a complex RAG benchmark specifically tailored to ERPs. The benchmark was constructed because existing evaluation sets were judged to contain only a relatively low proportion of ERPs. The cited proportions are 23.8% for BrowseComp, 23.6% for BrowseComp-Plus, 29.2% for BrowseComp Long Context, and 16.1% for GAIA. On that basis, WikiWeb-ERP was introduced to provide a benchmark where query planning matters more than simple sequential retrieval, to support systematic evaluation of PlanRAG on ERPs, and to enable comparisons against both iteration-based and graph-based RAG systems under ERP conditions (Xu et al., 1 Jul 2026).

The benchmark’s conceptual premise is that ERP queries differ materially from ordinary multi-hop questions. In the benchmark formulation, ordinary multi-hop settings often expose a relatively clear reasoning chain, whereas ERPs require identification of latent subgoals, retrieval of partial evidence from multiple sources, reconciliation of constraints across those subgoals, and planning of retrieval and aggregation order. This makes WikiWeb-ERP a benchmark not only of retrieval quality, but of structural reasoning and planning quality.

2. Dataset composition and construction

WikiWeb-ERP contains 3,536 queries and 53,682 documents. Query collection combines dataset sampling from prior benchmarks with generative expansion using a WebSailor-style approach, while document collection combines Wikipedia dump retrieval with web crawling. The benchmark is therefore designed to mix structured encyclopedic evidence with noisier open-web evidence (Xu et al., 1 Jul 2026).

Item	Value
Queries	3,536
Documents	53,682
Query origin	2,405 from existing datasets; 1,131 via WebSailor-style generation
Document origin	24,050 from Wikipedia dumps; 29,632 from crawled webpages
Average length	93.48 words per query; 121.09 words per document

The construction process follows two complementary collection strategies. On the query side, the benchmark draws from BrowseComp, BrowseComp-Plus, BrowseComp Long Context, and GAIA, then extends these materials through WebSailor-style generation to create more ERP-like open-web queries. On the document side, the evidence corpus is built through dense and sparse retrieval techniques over Wikipedia dumps together with WebSailor-style web collection from the internet. This dual-source design is explicitly intended to improve diversity and representativeness by combining comparatively structured sources with realistic web noise (Xu et al., 1 Jul 2026).

3. Query structure and exploratory reasoning characteristics

The benchmark defines ERPs as queries involving high uncertainty, ambiguity, difficulty in reduction, and fuzzy, emergent dependencies between entities. Such queries typically do not admit a clean, explicit reasoning path. Instead, they require decomposition into multiple latent subproblems whose relationships may only become clear during retrieval and aggregation (Xu et al., 1 Jul 2026).

A representative example is:

“There was an early Christian poetic hymn composed by a late antique writer who passed away around the mid-5th century. The year of this writer’s death coincides with the last year of a scientific chronology that reconstructs environmental conditions from several centuries before the modern era. What is the name of this chronology?”

The example is accompanied by atomic queries such as:

• “What were the early Christian hymns?”
• “Which late antique authors composed Christian hymns?”
• “Which authors passed away around the middle of the 5th century?”
• “Which authors’ years of death coincide with the final year of a scientific chronology?”
• “Which scientific chronologies reconstruct environmental conditions from several centuries before the modern era?”
• “What is the name of the scientific chronology?”

This structure illustrates the benchmark’s central distinction from standard factoid QA. Evidence is distributed across sources, intermediate goals are only partially visible in the original query, and success depends on discovering and coordinating those goals rather than following a fixed chain.

4. Logical query trees and the PlanRAG formalization

WikiWeb-ERP is the benchmark used to validate the claim that natural-language ERPs can be modeled as logical query trees (LQTs). In the formalization used by PlanRAG, an LQT is defined as a DAG

$\mathcal{T} = (V, E)$

where $V$ denotes nodes corresponding to atomic queries and $E$ denotes directed semantic dependencies between them (Xu et al., 1 Jul 2026).

The planning mechanism is inspired by classical database query optimization, including the Selinger-style dynamic-programming recurrence

$$C(S) = \min_{A \subset S} \left\{ C(A) + C(S \setminus A) + J(A, S \setminus A) \;\middle|\; A \neq \emptyset \right\}.$$

To score candidate trees, the paper defines a multi-dimensional heuristic cost:

$$\text{Cost}(\mathcal{T}) = - \Bigl( |V| + |E| - \text{Depth}(\mathcal{T}) - \text{Balance}(\mathcal{T}) + \alpha \, \text{Sim}(\mathcal{T}) \Bigr)$$

where $|V|$ is tree size, $|E|$ is structural density, $\text{Depth}(\mathcal{T})$ is tree depth, $\text{Balance}(\mathcal{T})$ is a balance measure, $\text{Sim}(\mathcal{T})$ is semantic similarity between the textualized LQT and the original query, and $V$0 is a scaling factor set to 10 in the experiments. Semantic similarity is computed with BGE embeddings as

$V$1

Operationally, PlanRAG first decomposes ERPs into atomic queries and then organizes them into LQTs using dynamic programming guided by the cost model. It subsequently performs iterative aggregation, rewriting, retrieval, and generation over the trees, processing independent nodes concurrently and propagating intermediate results upward, with further parallelization across multiple threads for efficiency. WikiWeb-ERP is thus not merely a test corpus; it is the empirical substrate for a planning-oriented formulation of RAG (Xu et al., 1 Jul 2026).

5. Evaluation protocol and empirical results

Evaluation on WikiWeb-ERP uses four metrics: Acc, EM, F1, and Acc†, where Acc† is semantic accuracy judged by GPT-3.5-turbo-instruct. The implementation details reported for the main comparison are a BM25 retriever, LLaMA-3-8B as the base model, GPT-4o-mini for decomposition, relation preprocessing, context-aware merging, and rewriting, treatment of documents as single passages because they are short, and multi-threaded execution for independent LQT nodes (Xu et al., 1 Jul 2026).

The benchmark compares PlanRAG against two baseline families. The iteration-based baselines are RetGen, GenGround, DualRAG, and KiRAG. The graph-based baselines are ChainRAG, HopRAG, and LEGO-GraphRAG. The comparison also includes DirectLLM and NaiveRAG.

The strongest reported result is PlanRAG w/ Ret, which achieves 26.43 Acc, 30.15 F1, 23.36 EM, and 30.68 Acc†. By comparison, LEGO-GraphRAG attains 25.48 Acc, 28.92 F1, 21.27 EM, and 28.30 Acc†; DualRAG attains 24.85 Acc, 26.65 F1, 20.47 EM, and 28.35 Acc†; KiRAG attains 24.72 Acc, 27.83 F1, 19.65 EM, and 27.92 Acc†; NaiveRAG attains 17.54 Acc, 20.33 F1, 14.65 EM, and 22.19 Acc†; and DirectLLM attains 12.31 Acc, 16.55 F1, 10.48 EM, and 16.88 Acc†. The ablation PlanRAG w/o Ret achieves 21.28 Acc, 25.95 F1, 19.82 EM, and 26.74 Acc† (Xu et al., 1 Jul 2026).

The paper’s interpretation is threefold. First, PlanRAG outperforms all iteration-based baselines. Second, it also outperforms all graph-based baselines. Third, the performance of PlanRAG w/o Ret indicates that planning structure itself contributes meaningfully, while the superior performance of PlanRAG w/ Ret indicates that the strongest gains arise from the combination of planning and retrieval rather than retrieval alone.

6. Curation quality, benchmark significance, and acronym disambiguation

The benchmark description does not report a large-scale human annotation pipeline for answer labels in the style of some QA datasets. Instead, curation is described in terms of selecting ERP-style queries from existing benchmarks, generating additional queries with WebSailor-style expansion, retrieving documents from Wikipedia and real webpages, and then using the resulting corpus for system evaluation. The paper does, however, provide structural quality indicators for the broader PlanRAG pipeline on WikiWeb-ERP: atomic query generation is manually assessed at 98.5% accuracy, Fleiss’s kappa = 0.82 is reported for agreement on atomic query correctness, atomic query coverage is 94.2%, the near-linear LQT degradation rate is 3.2%, and relationship classification accuracy is 96.4% for unrelated, 83.1% for parent-child, and 87.9% for sibling relations (Xu et al., 1 Jul 2026).

The benchmark is significant because it operationalizes a specific difficulty profile: no explicit reasoning chain, high retrieval noise, need for global planning, need for structural optimization, semantic ambiguity, and mixed-source evidence. In that sense, WikiWeb-ERP functions as an evaluation framework for the claim that ERP-solving is fundamentally a planning problem, not merely a retrieval problem.

A persistent source of confusion is the acronym ERP. In WikiWeb-ERP, ERP denotes exploratory reasoning problems, not enterprise resource planning. The latter meaning dominates a large adjacent literature, including work on PeopleSoft-based ERP modernization in digital transformation (Rawat, 2023), implementation methodologies and methodology-level success indicators (Yaghubi et al., 2014), and hybrid cloud ERP architectures for purchasing data processing (Zhang, 2022). This acronym collision is consequential for literature search, indexing, and benchmark discovery: “WikiWeb-ERP” belongs to RAG and reasoning-benchmark research, whereas the other ERP literature concerns enterprise information systems and organizational process infrastructure.