LLM-WikiRace Benchmark

Updated 4 July 2026

LLM-WikiRace is a benchmark that evaluates long-horizon planning, reasoning, and world knowledge by requiring models to navigate Wikipedia’s hyperlink graph step by step.
The task involves selecting filtered outgoing links based on oracle shortest-path data, making it interactive, partially observable, and sensitive to replanning failures.
Empirical results show frontier models excel on easy instances (>90% success) but struggle on harder tasks, highlighting significant challenges in adaptive sequential decision-making.

LLM-WikiRace is a benchmark for evaluating long-horizon planning, reasoning, and world knowledge in LLMs by requiring stepwise navigation over the Wikipedia hyperlink graph. Rather than answering a question directly, a model must move from a source page to a target page by selecting hyperlinks one step at a time, making the task interactive, partially observable, and sensitive to replanning failures. The benchmark is designed to test whether models can convert semantic and factual knowledge into successful action sequences under a fixed budget, and its central empirical result is that frontier systems are already very strong on easy instances while remaining sharply limited on hard ones: Gemini 3, GPT-5, and Claude Opus 4.5 exceed 90% success on the easy split, but the best hard-split result is only 23.0% (Ziomek et al., 18 Feb 2026).

1. Task definition and benchmark construction

Each LLM-WikiRace episode consists of a source Wikipedia page and a target Wikipedia page. At every timestep, the model is shown the current page, the target page, the history of previously visited pages, and a numbered list of outgoing hyperlinks from the current page; it must reply with the index of the link it wishes to click. The environment then updates the current page accordingly and continues until the target is reached or the step budget is exhausted (Ziomek et al., 18 Feb 2026).

The graph is derived from a fixed snapshot of Wikipedia taken on 23 June 2025. The benchmark retains the largest strongly connected component, containing 549,232 pages, so that source-target reachability is guaranteed within the evaluated subgraph. To control prompt length and cost, the environment does not expose all outgoing links. Instead, it presents the 50 outgoing links with the shortest precomputed shortest-path distance to the target and randomizes their order before showing them to the model. The shortest-path information used for this filtering is oracle information available only to the environment, not to the model (Ziomek et al., 18 Feb 2026).

The maximum episode length is $T=30$. A run is successful if and only if the target is reached within those 30 steps. Difficulty is defined graph-theoretically by shortest-path distance between source and target:

Split	Episodes	Shortest path length
Easy	200	3 or 4
Medium	150	5 or 6
Hard	100	7 or 8

This construction gives the task a clear operational meaning. The longest optimal path in the benchmark is 8, so the 30-step cap provides at least a threefold budget over optimal length while still making inefficient wandering and loop formation consequential (Ziomek et al., 18 Feb 2026).

2. Evaluation protocol and measured quantities

The benchmark reports three main metrics. Success Rate is the fraction of episodes solved within 30 steps. Suboptimal Steps measures how many extra steps beyond the shortest path are taken, averaged over successful runs. Usage per Step tracks average token usage per step, and the benchmark also reports monetary cost for closed models. The appendix extends the analysis with loop-sensitive diagnostics: Loop Frequency, Recovery Rate, and Average Maximum Visitation (Ziomek et al., 18 Feb 2026).

The evaluated model set spans over 20 systems. Closed-source models include the GPT-5 family, Gemini 2.0–3, Claude Sonnet 4.5, Claude Opus 4.5, and Grok 4.1-Fast. Open-source models include DeepSeek R1, Kimi K2, LLaMA 3 variants from 1B to 70B, Gemma 3 variants from 4B to 27B, Apertus 8B and 70B, Mistral 7B, Ministral 8B, Qwen 2.5-7B, Dream-v0-Inst. 7B, and LLaDA-Inst. 8B; all open models are evaluated in Instruct-finetuned form (Ziomek et al., 18 Feb 2026).

The paper also includes a preliminary fine-tuning ablation on Qwen-2.5-7B using DAPO. Training uses 1,000 source-target pairs with shortest-path distances from 2 to 6, 16 rollouts per prompt, and checkpoints at 50, 100, and 300 steps. This study is informative because it isolates whether simple preference-style improvement can close the planning gap exposed by the benchmark (Ziomek et al., 18 Feb 2026).

3. Performance profile across difficulty levels

The dominant empirical pattern is a sharp difficulty gradient. On the easy split, the strongest systems exceed 90% success; on the medium split, top performance falls into the 50–70% range; on the hard split, all reported systems remain below 25%. Gemini 3 is the strongest reported model overall, with 95.0% on easy, 66.0% on medium, and 23.0% on hard. GPT-5 reaches 92.5%, 60.0%, and 15.0%; Claude Opus 4.5 reaches 91.5%, 56.0%, and 18.0%; DeepSeek R1 reaches 91.0%, 54.7%, and 17.0%; and Kimi K2 reaches 87.5%, 45.3%, and 7.0%. Among open models, LLaMA 3 70B reaches 84.5% on easy, 39.3% on medium, and 7.0% on hard, while smaller LLaMA and Gemma variants are near zero on hard tasks (Ziomek et al., 18 Feb 2026).

Efficiency degrades along with success. Gemini 3 averages 0.8 suboptimal steps on easy, 1.9 on medium, and 4.7 on hard. GPT-5 averages 1.8, 2.7, and 6.3; Claude Opus 4.5 averages 1.3, 2.3, and 8.1. These figures indicate that even successful hard runs are often substantially less direct than shortest-path routes (Ziomek et al., 18 Feb 2026).

The fine-tuning ablation reinforces the same conclusion. Base Qwen-2.5-7B scores 22.5% on easy, 1.3% on medium, and 0.0% on hard. FineTuned-300 improves easy performance to 67.5% and medium performance to 4.6%, but hard performance remains 0.0%. The paper therefore concludes that simple one-step preference-style fine-tuning improves easy behavior without solving the long-horizon planning problem that dominates the hard split (Ziomek et al., 18 Feb 2026).

A human comparison is included using a reference corpus from The WikiGame Daily. Humans achieve 98.5% success, while Gemini 3, GPT-5, and Claude Opus 4.5 all achieve 100%. However, the corpus is easier than the benchmark’s easy split, so the paper treats success saturation cautiously and compares suboptimal steps instead; on that easier reference set, the models take fewer suboptimal steps than humans (Ziomek et al., 18 Feb 2026).

4. Knowledge, planning, and trajectory-level failure modes

A central contribution of LLM-WikiRace is its separation of world knowledge from long-horizon control. The paper evaluates graph knowledge with a separate binary task asking whether a source page directly links to a target page, using a dataset of 1,000 source-target pairs including connected and unconnected cases, targets at distances 2, 3, or 4, and reversed-link cases. The reported conclusion is that stronger world knowledge improves navigation performance, but only up to a point; beyond that threshold, planning and long-horizon reasoning become the dominant factors (Ziomek et al., 18 Feb 2026).

The authors describe this as a transition from a knowledge-limited regime to a planning-limited regime. This framing is supported by model contrasts: systems such as LLaMA 3 70B can display competitive graph-connection understanding yet still underperform frontier models in actual navigation. The implication is that recognizing relevant conceptual relations is not equivalent to executing a successful multi-step route through a large graph (Ziomek et al., 18 Feb 2026).

Trajectory analysis sharpens this distinction. Gemini 3 explicitly frames a hub-seeking strategy in 72% of inspected games, and Claude Opus 4.5 does so in 94%. The strongest models often move from narrow topics toward broad hub pages and reason about likely bridge concepts. Yet replanning after a wrong turn is a persistent weakness. Models may acknowledge that they are stuck, identify a loop in their own reasoning, and still fail to escape it (Ziomek et al., 18 Feb 2026).

Looping is the clearest failure mode. In inspected trajectories, Gemini 3 encounters loops in 66% of cases and Claude Opus 4.5 in 61%. On the hard split, Gemini 3 loops in 86% of tasks; only 10% of those looped trajectories are eventually solved, while all hard tasks without loops are solved. Claude Opus 4.5 loops in 92% of hard tasks, and only 1% of those looped trajectories succeed. The appendix reports a strong negative relationship between loop frequency and success, with regression coefficient $-1.02$ and 95% confidence interval $(-1.13, -0.91)$. The paper uses the term “blind looping” for cases in which models revisit the same page six or more times without effective recovery (Ziomek et al., 18 Feb 2026).

The case studies show that not all failures are purely planning failures. One illustrative contrast involves a target village that Gemini 3 correctly places in Croatia while Claude incorrectly places it in Serbia, causing Claude to fail. Even so, the benchmark’s general conclusion is that hard-task failures are dominated less by missing isolated facts than by failures of adaptive sequential decision-making (Ziomek et al., 18 Feb 2026).

LLM-WikiRace belongs to a broader line of work on goal-directed navigation in Wikipedia-like information spaces. A closely related study, "Textual understanding boost in the WikiRace," evaluates navigation on a Wikipedia hyperlink subgraph using only local outgoing links and compares structural heuristics, semantic title embeddings, and hybrid methods. In that setting, a greedy semantic policy using article-title embeddings from all-MiniLM-L6-v2 plus a visited-set loop-avoidance mechanism—denoted LLM*—achieves 155.3 average hops with 100% success, while pure betweenness centrality achieves 5000.0 average hops with 0% success and raw semantic search without loop avoidance achieves 3502.1 hops with 30% success (Ebrahimi et al., 13 Nov 2025).

That result is methodologically important for interpreting LLM-WikiRace. The earlier study shows that semantic title similarity can provide a strong zero-shot navigation signal and that loop avoidance can be decisive. LLM-WikiRace, by contrast, shows that general-purpose frontier models still struggle when the task emphasizes interactive long-horizon planning, replanning after failure, and repeated local decisions over a much larger real-world graph. This suggests that semantic proximity and world knowledge are necessary components of successful navigation, but they do not by themselves eliminate the control problem exposed by harder benchmark instances (Ebrahimi et al., 13 Nov 2025).

An adjacent architectural comparison appears in "Vector RAG vs LLM-Compiled Wiki," which is not a hyperlink-navigation benchmark but does study whether precompiled wiki structure helps LLMs synthesize information over a 24-paper corpus. There, an LLM-compiled markdown wiki clearly outperforms single-round Vector RAG on inter_paper_mapping, with judge-averaged Wiki − RAG = +6.625 on the multi-hop plus emergence subset, and its cited claims are supported 40.2% of the time versus 18.9% for RAG; however, the wiki uses 1,651,357 query tokens against 78,093 for RAG (Cochran, 18 May 2026). A plausible implication is that LLM-WikiRace sits at the intersection of two separable issues: navigation through linked knowledge structures and the representation of evidence in forms that facilitate multi-step reasoning. The existing literature indicates that improvements in one dimension do not automatically solve the other.

6. Position within the wider Wikipedia–LLM research landscape

LLM-WikiRace isolates one narrow but consequential capability: online, goal-directed navigation over a real-world hyperlink graph. Other Wikipedia-centered LLM research addresses different layers of the same ecosystem. "LLMs in Wikipedia: Investigating How LLMs Impact Participation in Knowledge Communities" studies editors rather than navigators and finds an expertise-based participation divide: LLMs help experienced editors expand and improve contributions, but newcomers face increased burdens because meaningful participation requires tacit knowledge of verifiability, neutrality, and community norms. The paper characterizes this as a “paradox of participation,” in which LLMs lower the barrier to entry while raising the demands of legitimate participation (Zhou et al., 9 Sep 2025).

A different adjacent strand concerns structured knowledge infrastructure rather than community participation. "Scholarly Wikidata: Population and Exploration of Conference Data in Wikidata using LLMs" presents a semi-automatic extraction and ingestion pipeline—PDF or web crawl, chunking, LLM extraction, CSV export, OpenRefine reconciliation and validation, schema mapping, QuickStatements, then Wikidata population—that was applied to 105 Semantic Web-related conferences, extending more than 6,000 entities and more than 30,000 statements. The paper emphasizes that LLM outputs require human validation to eliminate noisy extraction and ensure accuracy (Mihindukulasooriya et al., 2024).

Taken together, these neighboring works clarify the scope of LLM-WikiRace. The benchmark does not measure Wikipedia editing competence, social alignment with community norms, ontology engineering, or data curation. It measures whether an LLM can transform knowledge into adaptive action over a large, sparse, real-world graph. Its enduring significance lies in the gap it exposes: models can often articulate sensible strategies, identify broad semantic hubs, and perform at or above non-expert human level on very easy instances, yet still fail abruptly when the problem requires robust replanning under partial observability and loop risk. Within the current literature, LLM-WikiRace therefore functions as a planning benchmark whose difficulty is grounded in a real knowledge graph rather than a synthetic puzzle, and whose main lesson is that world knowledge, semantic relatedness, and fluent reasoning are not equivalent to stable long-horizon control (Ziomek et al., 18 Feb 2026).