HalluWorld: Controlled Hallucination Benchmark

• HalluWorld is a controlled benchmark that defines hallucination via an explicit reference-world formulation, allowing mechanical truth assessment.
• It categorizes errors into perceptual, memory, causal, uncertainty, and compound probes across diverse domains like gridworlds, chess, and terminals.
• The benchmark systematically varies world complexity, observability, temporal change, and source-conflict policies to isolate distinct hallucination failure modes.

HalluWorld is a controlled benchmark for hallucination in LLMs that grounds evaluation in an explicit reference-world formulation: a model hallucinates when it produces an observable claim that is false with respect to a fully specified world. Introduced to address fragmentation across summarization, question answering, retrieval-augmented generation, and agentic interaction, HalluWorld constructs synthetic and semi-synthetic environments in which the reference world is explicit, the model’s view is controlled, and hallucination labels are generated automatically. It spans gridworlds, chess, and realistic terminal tasks, and is designed to vary world complexity, observability, temporal change, and source-conflict policy while separating perceptual, memory, causal, uncertainty, and compound errors (Liu et al., 19 May 2026).

1. Reference-world semantics

The benchmark adopts a unified world-model view in which a world is represented as $W=(S,H,R)$, where $S$ is the state, $H$ is the action/history trace, and $R$ is the rule system or transition semantics. A view function $V$ determines what the model can observe, and a truth function $T_{w,p}$ determines whether a probe $p$ is true in world $w$ under a given policy $P$. Within this formulation, hallucination is not defined relative only to a prompt, a retrieved passage, or a human-written reference answer. It is defined relative to an explicit simulator state plus transition history, so truth can be computed mechanically rather than inferred from annotation (Liu et al., 19 May 2026).

This formulation is also policy-sensitive. HalluWorld emphasizes that conflict policy matters when multiple sources disagree. In gridworlds, a model should trust its own direct observation over a misleading signboard; in chess, the board state can override a corrupted FEN string; in terminals, the live terminal trace is the authority over stale or contradictory artifacts. The benchmark therefore treats source conflict not as incidental noise but as a controlled dimension of evaluation.

A central implication of this design is diagnostic specificity. HalluWorld is motivated by the claim that hallucinations are not one monolithic failure. Misreading what is directly visible, forgetting a prior observation, reasoning incorrectly about future state changes, or over-trusting stale written testimony are qualitatively different errors. The benchmark is organized to separate these failure modes rather than collapse them into a single undifferentiated score.

2. Benchmark construction and domains

HalluWorld has three domains: synthetic gridworlds, chess, and realistic terminal tasks. Each domain instantiates the same reference-world logic but at a different point on the spectrum from fully synthetic control to realistic operational complexity (Liu et al., 19 May 2026).

Domain	Reference world and view	Scale
HalluWorld-Grid	MiniGrid state with controllable serializers	33 unique levels, 839 probes
HalluWorld-Chess	Chess position and move history with chess rules as $R$	7 probes, each run over 50 positions
HalluWorld-Terminal	Full live Linux filesystem and process state with raw terminal history as view	132 Terminal-Bench trajectories spanning 110 tasks, 529 probes

In HalluWorld-Grid, the reference world is the grid state itself: agent position, object locations, door states, and action history, with custom mechanics such as rivers, fire, flood tiles, pressure plates, and dark zones. Observability is varied through serializers: a symbolic structured object list, an ASCII grid, or a memory format that combines current observation with prior observations. Because the simulator state is explicit, labels are automatic. The grid suite includes levels such as Dense Array, Corridor Gauntlet, Rotation Challenge, River Field, Witness Stand, Incident Report, Unreliable Narrator, Persistent Chain, Continuous Chain, Flood Room, Forking Paths, Adversarial Board, Flood-Fire Escape, Fog of War, Oracle Problem, Amnesiac, Facility Tour variants, and Dragon Keep.

In HalluWorld-Chess, the world is the chess position and move history, encoded through FEN plus PGN history, with chess rules as $S$0. The observation usually includes an ASCII board and side to move; sometimes a FEN is shown, and sometimes an incorrect FEN is injected as a controlled conflict source. Positions are drawn from real Lichess puzzles and then advanced by $S$1 random legal moves, default 14, to reduce trivial opening-book behavior. Chess serves as a middle ground between synthetic control and real-world priors.

In HalluWorld-Terminal, the reference world is the full live Linux filesystem and process state, while the model’s view is the raw terminal history visible at a trajectory step. This domain is built from 132 Terminal-Bench trajectories spanning 110 software-engineering tasks, and the authors run the original trajectories with GPT-4o-mini to elicit messy, failure-prone traces rather than optimized solutions. An LLM-based probe generator grounds questions in file-system diffs, hashes, mtimes, directory listings, compile outputs, and command traces, and probes are retained only if they are answerable and sufficiently difficult.

3. Probe taxonomy and controlled variables

Across the benchmark, HalluWorld organizes probes into five categories: perceptual, memory, causal, uncertainty, and compound. Perceptual probes test directly observed facts; memory probes test tracking across time; causal probes test forward simulation; uncertainty probes test whether the model abstains when evidence is insufficient; and compound probes test integration across multiple sources or steps. In the terminal domain, the same taxonomy is mapped to more operational labels: stale memory, cross-category reasoning, uncertainty overclaim, causal shortcut, and version/API hallucination (Liu et al., 19 May 2026).

Within gridworlds, probes use six closed-form answer types: presence, count, state/attribute, location, causal outcome, and uncertainty or “can’t determine.” Chess omits uncertainty probes because the board fully specifies the current position. The memory probe removes the board entirely and requires tracking long move sequences, while the compound probe asks for a legal move from SAN-only history, forcing reconstruction of the hidden board state.

A major methodological contribution is controlled variation along four axes. World complexity increases from simple local grid scenes to multi-zone navigation, then to chess histories, and then to long terminal traces. Observability is varied through serializers and through omission of boards or other state cues. Temporal change is varied through moving objects, persistent versus continuous mechanics, repeated room visits, move sequences in chess, and evolving terminal sessions. Source-conflict policy is varied through misleading noticeboards, signposts, incorrect FENs, and stale metadata. This is the mechanism by which HalluWorld disentangles hallucination into fine-grained categories.

The benchmark’s treatment of uncertainty is especially notable. In many benchmark settings, the correct model behavior is assumed to be answer production. HalluWorld instead includes cases where the correct answer is “can’t determine.” This shifts part of hallucination evaluation from factual recall to epistemic calibration.

4. Evaluation protocol and empirical results

The experimental evaluation includes frontier and open-weight models, with non-thinking and thinking variants where available. The reported models are GPT-5.5, o3, o4-mini, GPT-4o, GPT-4o-mini, Claude Sonnet 4.6, Claude Opus 4.6, Qwen-3-30B, GLM-5, DeepSeek-V3.1/0324, and Kimi K2.6. For gridworlds, the evaluation uses 10 seeds for nondeterministic levels; for chess, each probe is run over 50 positions; for terminals, the entire runtime context is passed in, truncated to 60k characters when necessary. Unless otherwise noted, answer length is constrained and thinking models are given much larger reasoning budgets (Liu et al., 19 May 2026).

The principal empirical pattern is stable across domains. Perceptual hallucination on directly observed information is near-solved for strong frontier models, but multi-step state tracking and causal forward simulation remain difficult. In gridworlds, GPT-5.5 has 5.0% perceptual hallucination overall; Claude Opus 4.6 is at 5.9%, o3 at 6.8%, and Claude Sonnet 4.6 at 7.2%. On uncertainty probes in gridworlds, GPT-5.5 reaches 0.1% hallucination and o3 0.3%. By contrast, memory and causal probes are harder: GPT-5.5 is at 5.9% on memory and 25.3% on causal, while Claude Opus 4.6 is 3.8% on memory and 20.1% on causal. The paper summarizes this pattern as roughly $S$2 for most models.

In chess, the same decomposition appears in a different form. With no FEN corruption, GPT-5.5 (T) is at 0.3% overall hallucination on visible-board perceptual and causal tasks, o3 is at 2.0%, and GPT-5.4 at 2.6%. Weaker or non-thinking models degrade sharply, especially on memory and compound probes. When an incorrect FEN is included, even strong models worsen slightly, showing sensitivity to contradictory textual metadata.

The terminal setting is the hardest overall. GPT-5.5 (T) reaches 5.9% hallucination overall, o3 (T) 10.2%, o4-mini (T) 14.2%, and Claude Opus 4.6 (T) 15.7%. Non-thinking and many open-weight models are far worse: GPT-4o is 38.4%, DeepSeek-V3.1 is 45.6%, GPT-5.4-mini is 50.3%, Qwen3-30B is 51.0%, GPT-4o-mini is 51.8%, and Kimi-K2.6 is 56.5%. In terminals, the category ordering is roughly perceptual and causal best, then compound and memory, and uncertainty worst. Even GPT-5.5 and o3 are only 76.9% accurate on terminal uncertainty probes, indicating that abstention remains difficult in realistic tool-use settings.

The benchmark also isolates several interaction effects. Serializer choice is an independent driver of hallucination in gridworlds: grid serialization helps allocentric rotation tasks but hurts multi-step attribute tracking, while memory serialization can help temporal tracking but make some causal tasks harder. Extended thinking does not uniformly help. In gridworlds, Claude Sonnet 4.6 improves on perception from 2.8% to 1.1% hallucination when thinking is enabled, and Claude Opus 4.6 improves similarly on perception, but both worsen on causal probes: Sonnet goes from 19.7% to 24.0%, and Opus from 20.1% to 25.2%. In terminals, by contrast, extended thinking helps more consistently: Sonnet 4.6 rises from 79.2% accuracy with no thinking to 86.2% at max thinking, while GPT-5.5 rises from 86.4% with no reasoning to 94.9% with low reasoning and reaches 96.6% at high reasoning.

A further analysis compares In-Navigation to a Controlled Static condition with identical trajectories. Overall, navigation usually helps reasoning models, though not all models. For GPT-5.5, hallucination drops by 13.2 points in navigation versus static; GLM-5, o3-mini, o4-mini, and o3 also improve. The reported interpretation is that active interaction can improve grounding in some settings while imposing cognitive load in others.

5. Position within hallucination research

HalluWorld belongs to a broader body of work that treats hallucination as an evaluation-design problem as much as a model-design problem. The benchmark’s motivation partly parallels the psychometric critique in “Evaluating the Quality of Hallucination Benchmarks for Large Vision-LLMs,” which proposes the Hallucination benchmark Quality Measurement framework and argues that benchmark quality should be assessed through test-retest reliability, parallel-forms reliability, criterion validity, and coverage of hallucination types (Yan et al., 2024). HalluWorld addresses a different problem, but both works reject the assumption that any single benchmark score is automatically trustworthy.

Its contribution also differs from task-specific hallucination benchmarks. “Hallucination Localization in Video Captioning” defines a span-level task in which the model outputs token labels indicating whether each caption token is hallucinated, and introduces HLVC-Dataset for fine-grained video-caption error localization (Nakada et al., 29 Oct 2025). “WildHallucinations” evaluates long-form factuality by prompting models to generate paragraphs about entities mined from real user-chatbot conversations and then fact-checking atomic claims against web-mined knowledge sources, with 52% of entities lacking corresponding Wikipedia pages (Zhao et al., 2024). “HalluJudge” treats hallucination in automated code review as context misalignment between a pull-request diff and an LLM-generated review comment, and uses a reference-free LLM-as-a-Judge framework to assess grounding without a gold reference comment (Tantithamthavorn et al., 27 Jan 2026).

These neighboring benchmarks expose different surfaces of the hallucination problem: token-span grounding in video captions, long-form factuality over real-world entities, context alignment in software review, and benchmark reliability for LVLM evaluation. HalluWorld’s distinctive move is to make the reference world explicit and executable. This suggests a complementary role rather than direct substitution: task-specific benchmarks capture domain realism, whereas controlled reference worlds isolate failure mechanisms with automatic labels and reproducible ground truth.

6. Interpretation, misconceptions, and implications

One common misconception is that hallucination is a single capability deficit that should decline uniformly as models improve. HalluWorld argues against this directly. The same model can be near-perfect on direct observation yet fail badly on forward simulation, long-horizon state tracking, or abstention. Open-weight models are not merely worse on the same axis; some exhibit uneven profiles. Kimi K2.6, for example, has 47.4% causal hallucination in gridworlds while its other categories are much lower (Liu et al., 19 May 2026).

A second misconception is that extended thinking uniformly reduces hallucination. HalluWorld reports a mixed picture. In gridworlds, more reasoning can raise causal error, and the authors interpret this as extra reasoning creating more room for confabulated intermediate states. In terminals, extended thinking helps more consistently, but uncertainty remains the hardest category across all reasoning budgets, and more reasoning can still hurt epistemic abstention in some configurations.

A third misconception is that perceptual success implies the hallucination problem is largely solved. HalluWorld’s data do not support that conclusion. The paper reports that perceptual hallucination on directly observed information is near-solved for frontier models, but multi-step state tracking and causal forward simulation remain difficult and are not generally solved by extended thinking. In the terminal setting, models also struggle with when to abstain. The uneven profile of failures across probe types and domains suggests that hallucinations arise from distinct failure modes rather than a single capability.

The benchmark’s broader significance lies in reproducibility and scale. Because the reference worlds are fully specified and labels are generated automatically, HalluWorld offers a controlled alternative to benchmarks that require human annotation or fixed references that may be memorized. A plausible implication is that interventions can be evaluated more cleanly against specific hallucination mechanisms: perceptual reading, memory maintenance, causal simulation, uncertainty calibration, or multi-source integration. HalluWorld therefore positions controlled reference worlds as a scalable and reproducible path toward measuring and reducing hallucinations in modern LLMs (Liu et al., 19 May 2026).