Tracing the Thought of a Grandmaster-level Chess-Playing Transformer

Abstract: While modern transformer neural networks achieve grandmaster-level performance in chess and other reasoning tasks, their internal computation process remains largely opaque. Focusing on Leela Chess Zero (LC0), we introduce a sparse decomposition framework to interpret its internal computation by decomposing its MLP and attention modules with sparse replacement layers, which capture the primary computation process of LC0. We conduct a detailed case study showing that these pathways expose rich, interpretable tactical considerations that are empirically verifiable. We further introduce three quantitative metrics and show that LC0 exhibits parallel reasoning behavior consistent with the inductive bias of its policy head architecture. To the best of our knowledge, this is the first work to decompose the internal computation of a transformer on both MLP and attention modules for interpretability. Combining sparse replacement layers and causal interventions in LC0 provides a comprehensive understanding of advanced tactical reasoning, offering critical insights into the underlying mechanisms of superhuman systems. Our code is available at https://github.com/JacklE0niden/Leela-SAEs.

• The paper introduces a sparse decomposition framework using Transcoders and Lorsas to isolate monosemantic features for tracing grandmaster-level chess decisions.
• It employs quantitative metrics and causal intervention methods to validate feature activations and construct interpretable reasoning pathways.
• Empirical results demonstrate that the approach effectively reveals parallel tactical circuits and enhances insight into transformer-based chess reasoning.

Tracing the Internal Computation of a Grandmaster-Level Chess Transformer

Introduction

This work proposes an interpretability framework for analyzing the internal reasoning of the transformer-based chess model Leela Chess Zero (LC0), which is able to attain grandmaster-level policy strength using only a forward pass of its policy network, without search. The primary contribution is a sparse decomposition technique that disentangles the MLP and attention computations into monosemantic, feature-level units using Transcoders and Low-Rank Sparse Attention modules (Lorsas), enabling end-to-end circuit-level tracing and intervention. By integrating these replacement layers with causal feature steering, the study introduces the concept of a "reasoning pathway"—a directed graph representing the sequence of semantically aligned feature activations corresponding to a specific model decision (move). The approach is the first to provide simultaneous circuit-level sparse decomposition for both MLP and attention modules in a transformer model applied to a real-world grandmaster reasoning domain.

Sparse Replacement Layers and Monosemantic Feature Extraction

Traditional interpretability in transformers is hindered by computational superposition and entangled neuron representations. This work applies recent advances in unsupervised dictionary learning—specifically, Transcoders for MLP layers and Lorsas for attention modules. Transcoders use sparse linear encoders/decoders to decompose MLP computations such that only the top $K$ most activated features (monosemantic sparse directions) are preserved per position, while Lorsas decompose multi-head self-attention (MHSA) into sparse, rank-1 OV heads, whose value activations and attentional focus (z-patterns) map monosemantic relational features on the board. These replacement layers were quantitatively validated, with $L_2$ norm reconstruction error ratios below 0.3 and explained variances above 0.6 for both component types.

The study introduces a systematic patching/steering protocol, in which decoder vectors of identified features are injected (or zeroed out) in the residual stream at arbitrary layers. The framework formally quantifies both feature-to-output and feature-to-feature causal effects, enabling mechanistic analysis at circuit level.

Interpreting and Validating Features

Feature interpretability is assessed via spatial activations (Transcoder: square-wise scalars, Lorsa: attention z-patterns) and rule-based validation grounded in chess semantics (e.g., detection of specific piece presence, tactical motifs, spatial relations). Human annotated evaluation (Elo 1000+, 2000+, 2100+) demonstrated that Transcoder and Lorsa features received high marks for consistency (up to 4.15/5) and moderate to low perceived complexity, indicating meaningful semantic alignment. Feature categories with rule-based labels include piece detection, source/target square, value estimation, tactical and spatial relations, capture, and piece movement.

On a large set of lichess standard games, feature activations covering rare tactical motifs were validated with high precision (frequently above 85%, reaching 100% for basic piece detection and movement).

Construction and Analysis of Reasoning Pathways

The methodology for extracting reasoning pathways for a specific move is as follows: (1) identify significant features with high influence on move output, (2) select nodes whose ablation most affects the move's predicted probability, (3) compute feature-to-feature effects to construct a directed graph, and (4) prune to retain only causally salient pathways. Feature steering is performed with a steering factor $a=-1$ (consistent with zero ablation), empirically shown to yield a strongly linear and interpretable response in both policy logits and probability space.

Three new quantitative metrics are introduced for pathway analysis: path overlap (Jaccard similarity of significant features across moves), path cohesion (intra-pathway average causal effect), and path coupling (inter-pathway causal effect).

Case Study: Mechanistic Insights

In a detailed critical tactical position, LC0 assigns significant but split probabilities between three plausible moves, with the optimal choice being one-move checkmate. Reasoning pathway analysis reveals:

1. Cross-Layer Superposition of Tactics: The model encodes and propagates piece control (bishop's coverage) over the target square via multiple distributed Lorsa features and layers. Zero ablation of these features suppresses activation and reduces the probability of the checkmate move.
2. Opponent Defensive Structure: Lorsa features encoding the rank-wise coverage of the black rook modulate whether a square is perceived as defended; copying this activation from an adjacent square to the checkmating square causes the model to suppress the attacking continuation, with downstream feature suppression validated.
3. Pathological Over-evaluation of Defensive Constraints: A false negative arises due to the model's detection of a pawn (feature with 100% validation as own-pawn-detection), which triggers defensive fixation and reduces tactical sensitivity. Causal intervention (removing the pawn or ablating the detection feature) recovers correct decisiveness.

Across all findings, causal manipulations of features directly modulate critical pathway activations and move selection, demonstrating robustness and semantic alignment of the extracted circuits.

Quantitative Results: Parallelism and Architectural Inductive Bias

Across a large suite of test positions stratified by model confidence, reasoning pathways for top moves demonstrate low path overlap (mean Jaccard < 15%), indicating a highly parallel and disjoint feature regime. Path cohesion (intra-path causal effects) is substantially higher than path coupling (cross-move effects), particularly in high-confidence settings (e.g., 67.1% vs. 1.8%). Same-source moves provide an internal baseline for representation overlap.

Further, both the entropy of significant feature spatial distribution and aggregate feature-to-output effect entropy decrease in deeper layers, while the Move-Square Contribution Ratio (MCR) increases. This demonstrates a progressive concentration of decision-critical information on source and target squares, explicitly matching the architectural computation of LC0's attention-based policy head.

Implications, Limitations, and Future Directions

This framework exposes the fine-grained, mechanistically interpretable internal mechanisms by which transformer-based chess agents compose and evaluate tactics, accounting for both local and global factors, cross-layer superposition, and subtle failures in decision-making. The approach subsumes but greatly extends prior local and attention-head-specific interpretability work, offering full-model tracing and unsupervised feature discovery.

Limitations include the reliance on human-in-the-loop validation (automatic auto-interpretation remains an open problem), potential feature granularity mismatches (feature splitting/absorption), and architectural-specific generality—though the method is broadly applicable to transformer variants and other structured domains. Larger-scale taxonomy and coverage studies of identified features are warranted to better understand "dark" latent features.

Future work should address generalization across architectures, improvements in automated semantic labeling via more agentic or generative approaches, systematic taxonomy of feature types, and application to domains beyond chess, such as mathematical reasoning or automated theorem proving. Circuit-level editing for safe or robust policy modification also presents a promising avenue for practical model control.

Conclusion

This study establishes a comprehensive sparse decomposition and causal intervention framework for transformer-based chess models, enabling the extraction and validation of monosemantic features and complete reasoning pathways encompassing both MLP and attention modules. The empirical evaluation on LC0 reveals parallelized but convergent tactical reasoning consistent with the model's architectural inductive biases. The methodology enables mechanistic interpretability, offering rigorous means to dissect, audit, and transfer knowledge from superhuman reasoning systems, with broader implications for transparency and robustness in advanced AI.