Chess960 Dataset: Overview & Methods

• Chess960 dataset is a collection of Fischer Random Chess resources including expert-annotated positions, engine-games, and full-space analytical evaluations.
• It employs standardized representations like FEN, UCI, and PGN to facilitate comparisons across benchmark types, game outcomes, and engine-based complexity measures.
• The resource landscape reveals fragmented data with implications for model generalization, reproducibility, and the extension of standard-chess benchmarking methods.

A Chess960 dataset is a corpus whose primary objects are Fischer Random Chess starting positions, games, annotations, or engine-derived measurements. Recent arXiv work shows that this label does not denote a single canonical resource, but at least three distinct constructions: an expert-annotated concept benchmark of 240 Chess960 positions, a large engine-game corpus of 480,000 games distributed uniformly across all 960 starting positions, and a reconstructable analytical collection over the full set of 960 legal starts using Stockfish evaluations and information-theoretic opening-complexity measures (Lomaso et al., 29 Oct 2025, Deo et al., 2023, Barthelemy, 16 Dec 2025). Alongside these, several influential chess datasets are explicitly standard-chess resources rather than Chess960 datasets, but their FEN-, UCI-, PGN-, and engine-centered designs have been presented as transferable templates for Chess960 extensions (Zhu et al., 3 Jun 2026, Walker et al., 28 May 2026, Wen et al., 28 Oct 2025).

1. Resource types and data representations

The current literature distinguishes between at least three technically different notions of a Chess960 dataset. One is a position-level conceptual benchmark, where the unit of data is a labeled board state. Another is a game-level corpus, where the unit is a full PGN record or a derived feature table. A third is an analytical ensemble over all 960 legal starting positions, where the primary outputs are engine evaluations and derived complexity statistics rather than raw human or engine game records (Lomaso et al., 29 Oct 2025, Deo et al., 2023, Barthelemy, 16 Dec 2025).

Resource	Core contents	Chess960 status
"Exploring Human-AI Conceptual Alignment through the Prism of Chess" (Lomaso et al., 29 Oct 2025)	240 expert-annotated positions across 6 concepts	Explicit Chess960 dataset
"Machine Learning Algorithms to Predict Chess960 Result and Develop Opening Themes" (Deo et al., 2023)	960 PGN files, 500 games per start, 480,000 games total	Explicit Chess960 dataset
"Not all Chess960 positions are equally complex" (Barthelemy, 16 Dec 2025)	All 960 starting positions with Stockfish-based evaluation and complexity methodology	Reconstructable Chess960 dataset, not explicitly released
"DeliChess" (Zhu et al., 3 Jun 2026)	Multi-party dialogue over standard-chess puzzles	Not Chess960
"Chess-World-Model" (Walker et al., 28 May 2026)	10M standard-chess games with exact-state targets	Not Chess960
"ChessQA" (Wen et al., 28 Oct 2025)	50-task, 3,500-item standard-chess QA benchmark	Not Chess960

Representation choices are unusually stable across these resources. The concept benchmark is built around static positions and labels; the engine-game corpus uses PGN-to-FEN processing and region-count features; the analytical study uses standard Chess960/Scharnagl indexing and engine-generated values; and the standard-chess template benchmarks are all formulated in FEN, UCI, PGN, and engine-evaluation terms (Lomaso et al., 29 Oct 2025, Deo et al., 2023, Barthelemy, 16 Dec 2025, Walker et al., 28 May 2026, Wen et al., 28 Oct 2025). This suggests a de facto representational backbone for Chess960 research: starting-position identifiers or FENs, legal-move encodings in UCI, and engine-grounded supervision.

2. Expert-annotated conceptual benchmark

The paper "Exploring Human-AI Conceptual Alignment through the Prism of Chess" introduces what it describes as the first Chess960 dataset: 240 expert-annotated positions across 6 strategic concepts, with 40 positions per concept, curated from high-level Chess960 games and annotated by experts with Elo > 2200 (Lomaso et al., 29 Oct 2025). The six concepts are Open Files and Diagonals, Knight Outposts, Advancement of f/g/h pawns (kingside), Advancement of a/b/c pawns (queenside), Center Control, and Pawn Play in the Center (Lomaso et al., 29 Oct 2025).

The dataset is designed as a robustness benchmark for conceptual generalization rather than as a game corpus. Each selected position is meant to “clearly exemplify” a target concept while matching the complexity distribution of the Strategic Test Suite used for standard chess (Lomaso et al., 29 Oct 2025). In the paper’s experiments, positions are used for binary concept probing under 5-fold cross-validation, and the central empirical result is that when probes are trained on standard chess and tested on Chess960, concept recognition accuracy drops by about 10–20 percentage points across all methods and concepts (Lomaso et al., 29 Oct 2025). The authors interpret this as evidence that standard-chess concept recognition relies substantially on memorized patterns from conventional openings rather than purely abstract concept formation (Lomaso et al., 29 Oct 2025).

Methodologically, the benchmark is tied to representation probing of a 270M-parameter transformer with 18 transformer layers and hidden size $D = 1024$, but the dataset itself is position-centric rather than model-specific (Lomaso et al., 29 Oct 2025). At minimum, the paper implies the presence of FEN-level board descriptions and concept labels; however, it does not report inter-annotator agreement, annotator counts per position, or a fully specified file schema in the text (Lomaso et al., 29 Oct 2025). Dataset and code are stated to be available at https://github.com/slomasov/ChessConceptsLLM (Lomaso et al., 29 Oct 2025).

3. Engine-game corpus for result prediction and opening themes

The paper "Machine Learning Algorithms to Predict Chess960 Result and Develop Opening Themes" constructs a much larger Chess960 resource from engine games in PGN format (Deo et al., 2023). Its raw organization is explicit: a pipeline from Raw Data → Zipped Folder → Unzipped Folder → 960 .pgn files (one per starting position), with 500 games for each of the 960 starting positions, yielding 480,000 games in total (Deo et al., 2023). All games analyzed are engine games played by top engines “Rated between ELO Chess Rating ~4000 & ~1800” with reference to CCRL 40/2 FRC (Deo et al., 2023).

This corpus is not used directly as a full-sequence learning benchmark. Instead, it is transformed into several tabular datasets by extracting FEN positions at selected moves and reducing them to region-count features. The board is divided into five regions—Centre, White Kingside, White Queenside, Black Kingside, and Black Queenside—and for each position the dataset records the number of White and Black pieces in each region, together with the final game result encoded as 1 for a White win, 0.5 for a draw, and 0 for a Black win (Deo et al., 2023). The authors define three derived datasets: Data Set 1, built from one sampled game per starting position at move 20; Data Set 2, consisting of per-position move-20 tables over all 500 games; and Data Set 3, consisting of per-position tables from moves 10–15 (Deo et al., 2023).

A second use of the corpus is “opening theme” extraction. For each starting position, the authors examine region-count changes at moves 1, 6, 11, and 16, aggregate changes across all 500 games, and assign each starting position a direction of development for White and Black according to the region of maximum positive change (Deo et al., 2023). The result is a taxonomy of theme pairs such as Centre_Centre or Black Q Side_White K Side (Deo et al., 2023).

The paper reports prediction accuracies for KNN, Random Forest, and Gradient Boosted Trees across the three datasets. For Data Set 1, mean accuracy is 0.396 for KNN, 0.485 for Random Forest, and 0.401 for Gradient Trees. For Data Set 2, mean accuracies are 0.398, 0.391, and 0.394 respectively. For Data Set 3, mean accuracies are 0.387, 0.374, and 0.385 (Deo et al., 2023). The authors also note class imbalance, specifically that Black wins are less frequent overall, and acknowledge that the feature set is intentionally coarse and engine-only (Deo et al., 2023). No direct dataset or code link is provided in the text (Deo et al., 2023).

4. Full-space analytical dataset over all 960 starting positions

The paper "Not all Chess960 positions are equally complex" treats the entire space of Chess960 starting positions as the object of analysis (Barthelemy, 16 Dec 2025). It does not explicitly publish a downloadable dataset or code repository, but it provides enough methodological detail to reconstruct a very similar one (Barthelemy, 16 Dec 2025). The central inputs are the 960 legal Chess960 starting positions, represented by the standard Chess960/Scharnagl numbering: $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$ The paper gives concrete examples, including standard chess as position #518 with piece order RNBQKBNR, the most complex opening as #226 with BNRQKBNR, and the most balanced position as #198 with QNBRKBNR (Barthelemy, 16 Dec 2025).

For each starting position, the authors compute a Stockfish 17.1 depth-30 root evaluation $E$, reporting an ensemble mean

$$\langle E \rangle = +0.297 \pm 0.136 \ \text{pawns},$$

often rounded in the paper to approximately $+0.30 \pm 0.14$ pawns (Barthelemy, 16 Dec 2025). To measure decision difficulty, they define a per-ply information cost from the gap $\Delta$ between the best and second-best engine moves: $$S(\Delta) = \log_2\!\left(1 + e^{-\Delta/\Delta_0}\right), \qquad \Delta_0 = 10 \ \text{centipawns}.$$ They then accumulate this over the first $n$ plies: $S(n) = \sum_{i=1}^{n} S(\Delta_i),$ and decompose it by color into $S_W$ and $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$0, with

$$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$1

Here $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$2 is total opening complexity and $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$3 is decision asymmetry (Barthelemy, 16 Dec 2025).

The computations use 5 independent engine games per starting position and, for the main $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$4 analysis, 10 plies per player at depth 14, with depth 20 used for the time-evolution of asymmetry $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$5 (Barthelemy, 16 Dec 2025). Reported heterogeneity is substantial: $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$6 varies by roughly a factor of three across positions, while $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$7 ranges from about $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$8 to $$\mathrm{idn} = (\text{bishop code}) + 16 \times (\text{queen position}) + 96 \times (\text{N5N code}).$$9 bits (Barthelemy, 16 Dec 2025). Standard chess, position #518, is described as having typical overall complexity but above-average asymmetry, at the 47th percentile for total complexity and about the 91st percentile for asymmetry (Barthelemy, 16 Dec 2025).

As a dataset object, this work is unusual because its most important outputs are reconstructable numerical tables rather than an explicitly released benchmark. Its significance lies in making the full 960-start space quantitatively indexable by engine evaluation, per-color information cost, and balance criteria. The paper also defines a normalized distance in the $E$0 plane to locate the “most balanced” position, identifying #198 as the minimizer with $E$1 pawns and $E$2 bits (Barthelemy, 16 Dec 2025).

5. Standard-chess benchmarks as methodological templates for Chess960

Several recent datasets are explicitly not Chess960 datasets, yet are presented in the literature as highly relevant templates for building Chess960 resources (Zhu et al., 3 Jun 2026, Walker et al., 28 May 2026, Wen et al., 28 Oct 2025). Their importance lies in protocol design rather than variant coverage.

"DeliChess: A Multi-party Dialogue Dataset for Deliberation in Chess Puzzle Solving" is a standard-chess, puzzle-based dataset, not a Chess960 dataset (Zhu et al., 3 Jun 2026). It contains 107 multi-party dialogues with full transcripts, pre- and post-discussion choices, and puzzle metadata; each group solves 3 multiple-choice puzzles with 5 legal candidate moves per puzzle, and performance is measured with Simple, ARR, and Eval scores grounded in Stockfish (Zhu et al., 3 Jun 2026). The paper states that, although it covers only classical chess, it is “highly relevant as a model of how to structure such data” for Chess960 reasoning, dialogue, and decision-making tasks (Zhu et al., 3 Jun 2026).

"Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences" is likewise a standard chess benchmark rather than a Chess960 one (Walker et al., 28 May 2026). It is built from 10 million real games from the Lichess Open Database, with a held-out real test of 10,000 games and a random-uniform test of 10,000 games generated by uniformly sampling legal moves (Walker et al., 28 May 2026). States are represented as 75 categorical labels comprising 64 board squares, side to move, castling rights, en passant information, and move counters, and evaluation centers on ExactState, which is $E$3 only when all 75 labels are correct (Walker et al., 28 May 2026). The paper explicitly describes this as a blueprint for a Chess960 world-model benchmark: the move encoding, state alignment, and exact-state metric can be reused while replacing the data source and enabling Chess960 rules (Walker et al., 28 May 2026).

"ChessQA: Evaluating LLMs for Chess Understanding" is a 50-task, 3,500-item QA benchmark for standard chess (Wen et al., 28 Oct 2025). It spans Structural, Motifs, Short Tactics, Position Judgment, and Semantic tasks, using FEN, UCI, PGN, Stockfish centipawn values, and python-chess as the core machinery (Wen et al., 28 Oct 2025). The paper explicitly notes that Chess960 is not mentioned, but argues that the benchmark is “almost entirely representation-agnostic” and therefore a suitable template for a dedicated Chess960 QA benchmark once data sources and engine settings are changed to Chess960 mode (Wen et al., 28 Oct 2025).

Taken together, these standard-chess resources define three complementary transferable paradigms: dialogue and deliberation, exact state tracking, and broad QA-style evaluation. A plausible implication is that Chess960 dataset design is currently advancing less through direct large-scale releases than through adaptation of mature standard-chess benchmarking patterns.

6. Limitations, reproducibility, and extension paths

The present Chess960 dataset landscape remains fragmented. The concept benchmark is carefully curated but small, at 240 positions, and the paper does not report inter-annotator agreement or detailed annotation reliability statistics (Lomaso et al., 29 Oct 2025). The engine-game corpus is large, but it is based entirely on engine play, uses coarse region-count features, notes class imbalance, and does not describe train/test splits in detail (Deo et al., 2023). The full-space analytical study provides strong methodological detail but explicitly lacks a data-availability statement or repository link in the supplied text, so a usable dataset must be reconstructed from the published protocol (Barthelemy, 16 Dec 2025).

A second limitation is that much of the strongest infrastructure around chess datasets still targets standard chess. DeliChess, Chess-World-Model, and ChessQA all state that they are not Chess960 datasets (Zhu et al., 3 Jun 2026, Walker et al., 28 May 2026, Wen et al., 28 Oct 2025). Their relevance is therefore indirect: they supply schemas, metrics, and interfaces, but not Chess960-native content. This creates a separation between variant-specific data availability and variant-agnostic benchmarking methodology.

The literature nevertheless outlines concrete extension paths. DeliChess describes how a Chess960 deliberation dataset could be formed by sampling Chess960 positions, constructing multiple-choice puzzles, recording solo and group phases, and reusing the same three evaluation metrics (Zhu et al., 3 Jun 2026). Chess-World-Model describes how to preserve its UCI move encoding, FEN-like state targets, and ExactState evaluation while substituting Chess960 games and configuring the engine in Chess960 mode (Walker et al., 28 May 2026). ChessQA describes a similarly mechanical adaptation: retain task definitions and exact-match evaluation, but generate Chess960 FENs, engine evaluations, and commentary instead of standard-chess data (Wen et al., 28 Oct 2025). This suggests that the principal technical barrier is no longer representation or evaluation design, but the assembly and release of large, variant-specific corpora.

In that sense, “Chess960 dataset” now refers less to a settled benchmark than to an emerging family of resources. Position-level expert annotation, large engine-game archives, and fully enumerated engine analyses over all 960 starts are all already present in the literature (Lomaso et al., 29 Oct 2025, Deo et al., 2023, Barthelemy, 16 Dec 2025). The next stage, as implied by adjacent work, is likely to combine these strands into datasets that are simultaneously variant-native, engine-grounded, representation-complete, and task-diverse.