Chess-World-Model Research

• Chess-World-Model is a research framework that uses chess’s deterministic, fully observable domain to study whether sequence models maintain an accurate latent state and adhere to the official rules.
• It implements diverse methodologies—ranging from UCI move sequences and SAN board descriptions to FEN-based evaluations and adversarial robustness tests—to probe move legality and state reconstruction.
• Empirical results show that high legal-move accuracy can mask deficiencies in exact state tracking and robustness under adversarial conditions, underscoring the importance of architecture, scale, and evaluation criteria.

Chess-World-Model denotes a line of research in which chess is used as a controlled domain for studying whether learned sequence models maintain a correct latent state across action sequences. In this setting, the “world model” may refer to the exact transition system induced by the official rules of chess, or to a model’s implicit approximation to that system as evidenced by move prediction, board-state probes, affordance distributions, adversarial robustness, or exact state reconstruction. The topic encompasses early language-model testbeds based on UCI move sequences, probe-based analyses of PGN-trained transformers, instruction-finetuned small LLMs trained on SAN board descriptions, model-agnostic FEN-based state evaluation, adversarial soundness verification, geometry-aligned chess architectures, and the large-scale “Chess-World-Model” benchmark built from 10 million real games (Toshniwal et al., 2021, Karvonen, 2024, Fauber, 2024, Harang et al., 27 Aug 2025, Balogh et al., 5 Feb 2026, Monroe et al., 18 May 2026, Walker et al., 28 May 2026).

1. Formal object and conceptual scope

Chess is especially suitable for world-model research because its notation describes a simple, constrained, and deterministic domain, and because the domain is fully observable and rule-based (Toshniwal et al., 2021). This makes it possible to distinguish surface-level sequence regularities from genuine state tracking more cleanly than in natural language. A central formalization treats the world model as the legality predicate over prefixes. Let $\Sigma$ denote the finite set of all UCI-encoded chess moves; then a world model $W$ assigns to each prefix $s \in \Sigma^*$ the set $W(s) \subseteq \Sigma$ of legal continuations. A sequence $s = a_1 \ldots a_k$ is valid iff every prefix obeys the transition rule, and a generative model with deterministic decoding is sound with respect to $W$ iff, for every valid prefix with at least one legal continuation, its next move is legal (Balogh et al., 5 Feb 2026).

An alternative state-centric formalization represents positions directly. In the FEN-based evaluation framework, a valid chess state is an element of a finite set $\mathcal{S}$, and malformed or illegal FEN is mapped to a special sink state $0$ in the augmented state space $\mathcal{S}^* = \mathcal{S} \cup \{0\}$ (Harang et al., 27 Aug 2025). The same paper summarizes the induced chess world model as a finite-state automaton over FEN states. This emphasizes exact semantic state rather than only next-token prediction.

Formal specification work supplies a third perspective: the world model as an executable symbolic semantics. In VDM++, a board is a set of pieces with an invariant forbidding two pieces on the same square, a move has preconditions and postconditions, and derived predicates such as in_check and in_mate are defined directly over board state (Kristensen et al., 2023). This symbolic formulation is not itself a learned model; rather, it provides an exact reference semantics against which learned models can be compared. A plausible implication is that chess world-model research is best understood as the interaction between an exact transition system and approximate latent state estimators.

2. State representations and benchmark constructions

Different Chess-World-Model papers make different representation choices depending on whether the goal is probing, exact supervision, or model-agnostic evaluation. In the UCI-based state-tracking testbed, moves are encoded in Universal Chess Interface notation, where every move is four characters plus an optional promotion piece, such as f1b5; SAN was found too ambiguous for probing because it omits source-square information. The vocabulary consists of 64 square tokens, 6 piece-type tokens, 4 promotion tokens, and BOS/EOS/PAD, for a total of 77 tokens (Toshniwal et al., 2021).

By contrast, the instruction-finetuning work on small LLMs uses SAN-style board descriptions as prompts. Each training example is an instruction-response pair, where the instruction lists the board state in square:piece tokens and the response is a single white move in SAN, such as Rg3. No chess-specific tokenization was added beyond the pretrained model’s BPE vocabulary (Fauber, 2024). This formulation exposes board state explicitly at training time rather than requiring it to be reconstructed from a move history.

The FEN-based state-evaluation framework adopts the standard Forsyth–Edwards Notation as a textual state representation. A FEN string contains six fields: piece placement, active color, castling availability, en passant target square, halfmove clock, and fullmove number (Harang et al., 27 Aug 2025). Because FEN includes castling rights, en passant, and move counters, it captures semantics that are not recoverable from board occupancy alone.

The large-scale benchmark titled “Chess-World-Model” uses yet another decomposition. Input moves are encoded in UCI style by packing source square, target square, and promotion type into a single categorical token, yielding $64 \times 64 \times 5 = 20{,}480$ move tokens plus <START> and <PAD>, for a total vocabulary of 20,482 tokens. Invalid packings never appear during training, and only 1,968 of the 20,480 non-special tokens occur in both training and the random-uniform test set (Walker et al., 28 May 2026). The target state is represented by 75 categorical labels: 64 square-occupancy labels, 1 side-to-move label, 4 castling-rights labels, 1 en-passant-file label, 1 en-passant-rank label, and byte-wise heads for the halfmove clock and fullmove number. This design makes exact state tracking an explicitly supervised prediction problem rather than an implicit property inferred from behavior.

The same benchmark also introduces two distinct test distributions. Training and in-distribution validation come from 10 million human games from the March 2025 public Lichess monthly PGN exports, while evaluation includes both a held-out real-game split of 10,000 fresh April 2025 Lichess games and an out-of-distribution random-uniform split of 10,000 games generated by uniformly sampling a legal move at each ply under the FIDE rules (Walker et al., 28 May 2026). This separation is central to the benchmark’s purpose: it tests whether models learn transition rules rather than only frequent human positions.

3. Evaluation paradigms

Chess-World-Model research has produced several non-equivalent evaluation paradigms. They target different questions: whether a model predicts legal continuations, whether its hidden state linearly encodes the board, whether a decoded state has the correct affordances, whether every component of a position is exact, and whether the model can be forced into illegality under adversarial play.

Evaluation family	Core quantity	What it tests
Probe-style cloze evaluation (Toshniwal et al., 2021)	ExM, LgM, LgM-R-Prec	Exact next square, legality of prediction, legality among top-$W$0 answers
Affordance-based state evaluation (Harang et al., 27 Aug 2025)	$W$1, $W$2	Semantic fidelity via legal-move distributions
Exact-state reconstruction (Walker et al., 28 May 2026)	ExactState accuracy, labelwise accuracy, trajectory exactness	Whether all state labels match at each prefix
Adversarial soundness verification (Balogh et al., 5 Feb 2026)	Attack Success Rate	Whether legal prefixes can force an illegal model reply

The cloze-style probing framework defines ending-square and starting-square probes. Ending-square probes prompt the model with a source square and ask for its destination; starting-square probes prompt with a piece type and ask for the square of a movable piece of that type. Exact-Move accuracy measures whether the top prediction matches the ground truth, while Legal-Move accuracy checks only legality under the true board state. LgM-R-Prec averages the proportion of legal answers among the top-$W$3 predictions, where $W$4 is the number of legal continuations for the prompt (Toshniwal et al., 2021).

The state-based evaluation framework replaces string matching with legal-move affordances. Given a ground-truth state $W$5 and a predicted state $W$6, the true action distribution is uniform over the legal moves $W$7, while the model distribution over next-move actions is estimated either by re-prompting or by sampling:

$W$8

and

$W$9

The core metric is then the KL divergence

$s \in \Sigma^*$0

Low divergence indicates that the predicted state affords essentially the same moves as the real state (Harang et al., 27 Aug 2025).

The exact-state benchmark evaluates state tracking directly. If $s \in \Sigma^*$1 is the exact state after prefix $s \in \Sigma^*$2 and $s \in \Sigma^*$3 is the model prediction, then $s \in \Sigma^*$4 iff all 75 labels match. ExactState accuracy is the fraction of timesteps recovered exactly; labelwise accuracy averages correctness per label; trajectory exactness requires every prefix in an entire game to be exact (Walker et al., 28 May 2026). This metric is much stricter than legality-only criteria, because a single incorrect castling bit or en-passant field invalidates exact state.

Adversarial verification asks a different question: not whether a model often stays legal in-distribution, but whether it is sound for all legal prefixes. The adversary always chooses a legal white move, while the model plays greedily as black; Attack Success Rate is the fraction of prefixes from which an illegal reply can be forced (Balogh et al., 5 Feb 2026). The paper studies several adversaries, including the Illegal Move Oracle, Board State Oracle, Adversarial Detours, Random Move, and Sequence Model Move, thereby turning soundness into a falsifiable empirical property rather than an average-case statistic.

4. Model classes and representational mechanisms

The earliest learned chess world-model studies use autoregressive transformers trained solely on move sequences. One benchmark trains GPT2-small from scratch with 12 transformer layers, 12 heads, hidden size 768, and context window 512 tokens, using standard language-model cross-entropy on UCI move tokens. A train-time augmentation called Randomly Annotated Piece-type prepends a piece-type token to a UCI move with some probability, permitting direct starting-square probes without requiring the same token at test time (Toshniwal et al., 2021).

A related internal-representation study trains two character-level autoregressive GPTs on raw PGN strings from 16 million human-played Lichess games, one with 8 layers and about 25 million parameters and one with 16 layers and about 50 million parameters. Linear probes are trained for each square and layer to classify one of 13 piece-types or blank from the residual stream, and contrastive activations over high-Elo and low-Elo games define a skill vector $s \in \Sigma^*$5 in activation space (Karvonen, 2024). This line of work treats the world model as an internal representation to be decoded and edited.

Instruction-finetuned small LLMs constitute a different family. TinyStories-28M and OPT-125M are fine-tuned on SAN-derived board/move pairs using standard autoregressive cross-entropy,

$s \in \Sigma^*$6

with all parameters updated and no adapters or layer freezing (Fauber, 2024). Here the model is not reconstructing state from a move history alone; it is learning chess rules and tactical response from explicit board descriptions in natural-language-style prompts.

The exact-state benchmark compares a causal Transformer against three recurrent alternatives under a matched interface and training protocol: SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues. The recurrent models differ in how expressive their input-dependent state-transition mechanisms are, and the paper includes expressivity ablations: block-diagonal versus diagonal SLiCE, Mamba-3 versus Mamba-2, and signed versus positive-only eigenvalues in DeltaNet (Walker et al., 28 May 2026). This benchmark therefore frames chess world modeling as a controlled comparison of sequence-update mechanisms under exact supervision.

A separate architectural line aligns the model directly with board geometry. Chessformer is an encoder-only transformer that represents each legal position as a fixed sequence of 64 tokens, one per board square, with the board rotated so that the side to move faces the model. It augments self-attention with Geometric Attention Bias, a dynamic positional encoding that conditions square-to-square biases on the board state, and uses an attention-based source-destination policy head rather than flattened move tokens (Monroe et al., 18 May 2026). This is not a pure move-sequence state tracker in the style of the benchmark, but it is a chess world model in the broader sense that tokenization, geometry, and action space are all made board-native.

5. Empirical findings

Across this literature, one recurring result is that chess world models do emerge in learned sequence models, but their fidelity depends sharply on training setup, scale, architecture, and evaluation criterion. In the UCI testbed, transformer LLMs trained only on move sequences can learn highly accurate piece-tracking and legal-move generation when given about 200,000 games. On the Train-L split, starting-square Legal-Move accuracy with RAP reaches 100.0% for Actual and 99.6% for Other prompts, while ending-square Legal-Move accuracy for pure UCI reaches 97.7% for Actual prompts and 91.9% for Other prompts (Toshniwal et al., 2021). The same study shows that full attention over the entire move history is critical: Reformer, Performer, and LSTM baselines lag behind the full-attention GPT2 baseline, especially in lower-data regimes.

Internal-representation analyses find near-perfect linear recoverability of board state in PGN-trained GPTs. Best-layer square classification reaches 99.6% accuracy for the 16-layer model and 99.1% for the 8-layer model, while Elo classification between below 1550 and above 2050 reaches 90.5% and 88.6%, respectively (Karvonen, 2024). Interventions on probe directions are also causal in the narrow sense tested there: on a modified board, baseline legality is about 41%, but with board-state intervention it rises to about 92% for the 16-layer model and about 90% for the 8-layer model. Adding the positive skill vector improves win rate against Stockfish 16 at level 0 from 16.7% to 43.2% on a random board, a 2.6-fold increase, while subtracting it reduces performance sharply.

Instruction-finetuned small models exhibit a different profile. With 1 million examples, OPT-125M reaches more than 99% legal moves on the WSM-10K test, while TinyStories-28M plateaus around 90%. On “mate-in-one” puzzles, OPT-125M reaches about 24% under default generation and TinyStories-28M about 15% (Fauber, 2024). Increasing epochs from 1 to 10 at 100,000 examples raises legal-move accuracy on mate-in-one tasks from about 30% to about 55%, though hallucination rates drop more strongly with more unique examples than with extra epochs. This suggests that explicit board descriptions can teach local legality and some tactical competence, but deeper lookahead remains limited without search.

Affordance-based evaluation reveals a sharper long-horizon failure than legality-only statistics suggest. On 10,000 PGN-to-FEN conversions by GPT-4o, the Kendall $s \in \Sigma^*$7 between state precision $s \in \Sigma^*$8 and negative Levenshtein distance is about 0.69 overall, but it steadily falls as games get longer (Harang et al., 27 Aug 2025). Figure 1 in that work shows that mean $s \in \Sigma^*$9 starts around 0.6 after 5 moves but decays exponentially to about 0.015 by 50 moves, whereas a random board has about $W(s) \subseteq \Sigma$0 chance of supporting any given next move. At the same time, the probability of producing a legal next move remains near 1.0 even late in the game. The paper identifies the hidden source of this discrepancy as subtle degradation in castling rights, en passant, and exact piece placement. A common misconception is therefore that “legal next move” implies coherent state tracking; the state-based results directly contradict that interpretation.

The exact-state benchmark makes the distinction even stricter. On the held-out real-game split at about 3 million parameters, exact-state accuracy is about 27% for the Transformer, about 41% for SLiCE, about 52% for Mamba-3, and about 50% for Gated DeltaNet. At about 8 million parameters, the corresponding values are about 74%, about 97%, about 98%, and about 99% (Walker et al., 28 May 2026). By about 18 million parameters all recurrent models reach at least 99.8% exact state in-distribution, while the Transformer reaches about 99.9% only at about 36 million parameters. The out-of-distribution random-uniform split is much harsher: at 8 million parameters, the Transformer is about 8.6%, SLiCE about 33.2%, Mamba-3 about 33.3%, and Gated about 36.2%; at about 40 million parameters, the Transformer reaches about 72.9%, SLiCE about 73.5%, Mamba-3 about 81.1%, and Gated about 61.2%. The paper’s conclusion is that in-distribution saturation conceals failures that remain exposed under legal but atypical trajectories.

Adversarial verification strengthens the negative result. Across 24 GPT-2 models, the Illegal Move Oracle almost always attains Attack Success Rate about 1.0 within fewer than 30 adversarial moves (Balogh et al., 5 Feb 2026). Even models trained on Lichess-16M with standard next-token loss and more than 99.9% legal-move ratio on held-out test games are unsound under adversarial play. Larger datasets improve robustness, and the probability-distribution objective is roughly twice as hard to break as standard next-token training under several attacks, especially on random data. Joint board-state probes, however, yield negligible improvement in Attack Success Rate or legal-move ratio, and gradient analyses show near-orthogonal probe and next-token gradients. Another common misconception is therefore that easy board-state decoding implies that decoded state is causally used for next-token prediction; this study finds that the extracted board states have no causal role in most of the models it examines.

6. Interpretability, significance, and research directions

The topic has broadened from diagnosing LLMs to constructing architectures and benchmarks explicitly aligned with chess semantics. Chessformer exemplifies this shift. Its one-token-per-square representation makes every attention map directly attributable to board geometry; between-position correlation of Geometric Attention Bias rows is about 0.77, while sparse transcoder analyses on MLP activations reveal features corresponding to forks, pins, attacked squares, and endgame blockades (Monroe et al., 18 May 2026). On human move prediction, the 79M-parameter Maia-3 reaches 57.1% move-matching accuracy on the held-out Lichess Blitz benchmark, exceeding a prior 355M-parameter state of the art at 55.9% with fewer than a quarter of the parameters. In engine integration, the 191M-parameter Leela-CF adds $W(s) \subseteq \Sigma$1 Elo over the standard 195M-parameter ResNet in a 200-game round-robin at 600 nodes per move and consistently adds about 105–112 Elo in full LC0 tournaments. This line of work suggests that tokenization, positional encoding, and output design aligned with the board’s geometry can simultaneously support playing strength, human compatibility, and interpretability.

The benchmark literature points in a more cautionary direction. “Chess-World-Model” establishes exact state tracking as a practical large-scale supervised task and shows that out-of-distribution legal-play evaluation is essential, because held-out real-game performance saturates above about 18 million parameters while the random-uniform split remains discriminative up to 40 million (Walker et al., 28 May 2026). The FEN-affordance framework arrives at a similar lesson from a model-agnostic angle: divergence over legal-move distributions is more semantically meaningful than string matching for long sequences (Harang et al., 27 Aug 2025). Together, these results imply that evaluation based only on observed human continuations is insufficient for measuring a chess world model.

The field also contains a clear tension between implicit and explicit rule systems. Learned models can encode board state, skill, and legal-move structure to a high degree, yet adversarial verification shows that sample-trained sequence models remain unsound (Balogh et al., 5 Feb 2026). Formal symbolic models in VDM++ and exact replay engines such as python-chess provide legality-preserving reference semantics that learned systems do not guarantee by themselves (Kristensen et al., 2023). A plausible implication is that future chess world models may be judged less by average next-move quality than by whether they can combine learned representations with exact transition semantics.

Several future directions recur across the literature. One is stronger architectural or inference-time enforcement of rules, such as legality filters or move-decoder heads that enforce chess rules exactly (Fauber, 2024). Another is combining learned models with search or reinforcement learning for deeper tactical planning, including mate-in-two, endgame tablebases, or alternative rule sets such as Chess960 (Fauber, 2024). A third is stronger verification, including automaton-based formal language inclusion tests and adversarial training loops to enforce provable soundness at inference time (Balogh et al., 5 Feb 2026). The benchmark work adds a more specific recommendation: prefer recurrent state-space or state-update architectures with sufficiently expressive transition mechanisms, and always monitor strict exact-state and trajectory-level metrics rather than only per-label or next-move legality statistics (Walker et al., 28 May 2026).

In this broader sense, Chess-World-Model is not merely a single benchmark name but a technical program: using chess as a domain in which exact semantics, structured state, and model behavior can all be checked with unusual precision. Its importance lies in making latent state tracking empirically measurable, its difficulty lies in separating apparent legal play from exact semantic coherence, and its main result so far is mixed: sequence models do learn substantial internal chess structure, but exact, causally used, and adversarially robust world models remain substantially harder to achieve than high in-distribution move legality.