---
title: 'WikiWeb-ERP: Exploratory Reasoning Benchmark'
url: https://www.emergentmind.com/topics/wikiweb-erp
type: topic
---

# WikiWeb-ERP: Exploratory Reasoning Benchmark

WikiWeb-ERP is a benchmark for retrieval-augmented generation on **exploratory reasoning problems (ERPs)**, introduced to evaluate systems on natural-language queries characterized by **high uncertainty**, **ambiguity**, **difficulty in reduction**, and **fuzzy, emergent dependencies between entities**. Each benchmark instance combines a natural-language ERP query, a set of atomic queries derived from it, and an evidence corpus assembled from both **Wikipedia dumps** and **live webpages crawled from the web**. Its purpose is to measure performance in settings where evidence is noisy and heterogeneous and where reasoning does not collapse into a simple linear multi-hop chain [2607.00508].

## 1. Definition and research motivation

WikiWeb-ERP is presented as a **complex RAG benchmark** specifically tailored to ERPs. The benchmark was constructed because existing evaluation sets were judged to contain only a **relatively low proportion of ERPs**. The cited proportions are **23.8%** for BrowseComp, **23.6%** for BrowseComp-Plus, **29.2%** for BrowseComp Long Context, and **16.1%** for GAIA. On that basis, WikiWeb-ERP was introduced to provide a benchmark where **query planning** matters more than simple sequential retrieval, to support systematic evaluation of PlanRAG on ERPs, and to enable comparisons against both **iteration-based** and **graph-based** RAG systems under ERP conditions [2607.00508].

The benchmark’s conceptual premise is that ERP queries differ materially from ordinary multi-hop questions. In the benchmark formulation, ordinary multi-hop settings often expose a relatively clear reasoning chain, whereas ERPs require identification of latent subgoals, retrieval of partial evidence from multiple sources, reconciliation of constraints across those subgoals, and planning of retrieval and aggregation order. This makes WikiWeb-ERP a benchmark not only of retrieval quality, but of structural reasoning and planning quality.

## 2. Dataset composition and construction

WikiWeb-ERP contains **3,536 queries** and **53,682 documents**. Query collection combines **dataset sampling** from prior benchmarks with **generative expansion** using a WebSailor-style approach, while document collection combines **Wikipedia dump retrieval** with **web crawling**. The benchmark is therefore designed to mix structured encyclopedic evidence with noisier open-web evidence [2607.00508].

| Item | Value |
|---|---|
| Queries | **3,536** |
| Documents | **53,682** |
| Query origin | **2,405** from existing datasets; **1,131** via WebSailor-style generation |
| Document origin | **24,050** from Wikipedia dumps; **29,632** from crawled webpages |
| Average length | **93.48 words per query**; **121.09 words per document** |

The construction process follows two complementary collection strategies. On the query side, the benchmark draws from **BrowseComp**, **BrowseComp-Plus**, **BrowseComp Long Context**, and **GAIA**, then extends these materials through WebSailor-style generation to create more ERP-like open-web queries. On the document side, the evidence corpus is built through **dense and sparse retrieval techniques** over Wikipedia dumps together with **WebSailor-style web collection from the internet**. This dual-source design is explicitly intended to improve diversity and representativeness by combining comparatively structured sources with realistic web noise [2607.00508].

## 3. Query structure and exploratory reasoning characteristics

The benchmark defines ERPs as queries involving **high uncertainty**, **ambiguity**, **difficulty in reduction**, and **fuzzy, emergent dependencies between entities**. Such queries typically do not admit a clean, explicit reasoning path. Instead, they require decomposition into multiple latent subproblems whose relationships may only become clear during retrieval and aggregation [2607.00508].

A representative example is:

> “There was an early Christian poetic hymn composed by a late antique writer who passed away around the mid-5th century. The year of this writer’s death coincides with the last year of a scientific chronology that reconstructs environmental conditions from several centuries before the modern era. What is the name of this chronology?”

The example is accompanied by atomic queries such as:

- **“What were the early Christian hymns?”**
- **“Which late antique authors composed Christian hymns?”**
- **“Which authors passed away around the middle of the 5th century?”**
- **“Which authors’ years of death coincide with the final year of a scientific chronology?”**
- **“Which scientific chronologies reconstruct environmental conditions from several centuries before the modern era?”**
- **“What is the name of the scientific chronology?”**

This structure illustrates the benchmark’s central distinction from standard factoid QA. Evidence is distributed across sources, intermediate goals are only partially visible in the original query, and success depends on discovering and coordinating those goals rather than following a fixed chain.

## 4. Logical query trees and the PlanRAG formalization

WikiWeb-ERP is the benchmark used to validate the claim that natural-language ERPs can be modeled as **logical query trees (LQTs)**. In the formalization used by PlanRAG, an LQT is defined as a DAG

$$
\mathcal{T} = (V, E)
$$

where \(V\) denotes nodes corresponding to atomic queries and \(E\) denotes directed semantic dependencies between them [2607.00508].

The planning mechanism is inspired by classical database query optimization, including the Selinger-style dynamic-programming recurrence

$$
C(S) = \min_{A \subset S} \left\{ C(A) + C(S \setminus A) + J(A, S \setminus A) \;\middle|\; A \neq \emptyset \right\}.
$$

To score candidate trees, the paper defines a multi-dimensional heuristic cost:

$$
\text{Cost}(\mathcal{T}) = - \Bigl( |V| + |E| - \text{Depth}(\mathcal{T}) - \text{Balance}(\mathcal{T}) + \alpha \, \text{Sim}(\mathcal{T}) \Bigr)
$$

where \(|V|\) is tree size, \(|E|\) is structural density, \(\text{Depth}(\mathcal{T})\) is tree depth, \(\text{Balance}(\mathcal{T})\) is a balance measure, \(\text{Sim}(\mathcal{T})\) is semantic similarity between the textualized LQT and the original query, and \(\alpha > 1\) is a scaling factor set to **10** in the experiments. Semantic similarity is computed with BGE embeddings as

$$
\text{sim}(x, y) = \langle E(x), E(y) \rangle.
$$

Operationally, PlanRAG first decomposes ERPs into atomic queries and then organizes them into LQTs using dynamic programming guided by the cost model. It subsequently performs **iterative aggregation, rewriting, retrieval, and generation** over the trees, processing independent nodes concurrently and propagating intermediate results upward, with further parallelization across multiple threads for efficiency. WikiWeb-ERP is thus not merely a test corpus; it is the empirical substrate for a planning-oriented formulation of RAG [2607.00508].

## 5. Evaluation protocol and empirical results

Evaluation on WikiWeb-ERP uses four metrics: **Acc**, **EM**, **F1**, and **Acc†**, where **Acc†** is semantic accuracy judged by **GPT-3.5-turbo-instruct**. The implementation details reported for the main comparison are a **BM25 retriever**, **LLaMA-3-8B** as the base model, **GPT-4o-mini** for decomposition, relation preprocessing, context-aware merging, and rewriting, treatment of documents as **single passages** because they are short, and **multi-threaded execution** for independent LQT nodes [2607.00508].

The benchmark compares PlanRAG against two baseline families. The **iteration-based** baselines are **RetGen**, **GenGround**, **DualRAG**, and **KiRAG**. The **graph-based** baselines are **ChainRAG**, **HopRAG**, and **LEGO-GraphRAG**. The comparison also includes **DirectLLM** and **NaiveRAG**.

The strongest reported result is **PlanRAG w/ Ret**, which achieves **26.43 Acc**, **30.15 F1**, **23.36 EM**, and **30.68 Acc†**. By comparison, **LEGO-GraphRAG** attains **25.48 Acc**, **28.92 F1**, **21.27 EM**, and **28.30 Acc†**; **DualRAG** attains **24.85 Acc**, **26.65 F1**, **20.47 EM**, and **28.35 Acc†**; **KiRAG** attains **24.72 Acc**, **27.83 F1**, **19.65 EM**, and **27.92 Acc†**; **NaiveRAG** attains **17.54 Acc**, **20.33 F1**, **14.65 EM**, and **22.19 Acc†**; and **DirectLLM** attains **12.31 Acc**, **16.55 F1**, **10.48 EM**, and **16.88 Acc†**. The ablation **PlanRAG w/o Ret** achieves **21.28 Acc**, **25.95 F1**, **19.82 EM**, and **26.74 Acc†** [2607.00508].

The paper’s interpretation is threefold. First, PlanRAG outperforms all **iteration-based** baselines. Second, it also outperforms all **graph-based** baselines. Third, the performance of **PlanRAG w/o Ret** indicates that planning structure itself contributes meaningfully, while the superior performance of **PlanRAG w/ Ret** indicates that the strongest gains arise from the combination of planning and retrieval rather than retrieval alone.

## 6. Curation quality, benchmark significance, and acronym disambiguation

The benchmark description does not report a large-scale human annotation pipeline for answer labels in the style of some QA datasets. Instead, curation is described in terms of selecting ERP-style queries from existing benchmarks, generating additional queries with WebSailor-style expansion, retrieving documents from Wikipedia and real webpages, and then using the resulting corpus for system evaluation. The paper does, however, provide structural quality indicators for the broader PlanRAG pipeline on WikiWeb-ERP: **atomic query generation** is manually assessed at **98.5% accuracy**, **Fleiss’s kappa = 0.82** is reported for agreement on atomic query correctness, **atomic query coverage** is **94.2%**, the **near-linear LQT degradation rate** is **3.2%**, and relationship classification accuracy is **96.4%** for unrelated, **83.1%** for parent-child, and **87.9%** for sibling relations [2607.00508].

The benchmark is significant because it operationalizes a specific difficulty profile: **no explicit reasoning chain**, **high retrieval noise**, **need for global planning**, **need for structural optimization**, **semantic ambiguity**, and **mixed-source evidence**. In that sense, WikiWeb-ERP functions as an evaluation framework for the claim that ERP-solving is fundamentally a **planning problem**, not merely a retrieval problem.

A persistent source of confusion is the acronym **ERP**. In WikiWeb-ERP, ERP denotes **exploratory reasoning problems**, not **enterprise resource planning**. The latter meaning dominates a large adjacent literature, including work on PeopleSoft-based ERP modernization in digital transformation [2303.03224], implementation methodologies and methodology-level success indicators [1402.0780], and hybrid cloud ERP architectures for purchasing data processing [2202.10786]. This acronym collision is consequential for literature search, indexing, and benchmark discovery: “WikiWeb-ERP” belongs to RAG and reasoning-benchmark research, whereas the other ERP literature concerns enterprise information systems and organizational process infrastructure.

Source: https://www.emergentmind.com/topics/wikiweb-erp