Chess-Pretrained Action-Value Network

• Chess-Pretrained Action-Value Network is a transformer-based model that predicts win probabilities from chess board states using expert-level data.
• It employs dense reward distillation by converting predicted win rates into nuanced feedback for reinforcing LLM tactical decisions.
• Despite outperforming sparse reward methods, its performance plateaus below expert levels, underscoring the need for domain-rich pretraining.

A chess-pretrained action-value network is a neural model—often a large transformer—that predicts the quality or win probability of actions (legal chess moves) from a given board state, with parameters and training derived from expert-level sources. These networks are commonly used as reward models or direct policy/value predictors in both chess-specific AI and recent research in strategic reasoning with LLMs.

1. Chess-Pretrained Action-Value Network: Architecture and Origins

Recent approaches utilize a transformer-based model, typically of moderate to large scale (e.g., 270M parameters), structured as a decoder-only network with 16 layers, 1024 embedding dimensions, and 8 attention heads. Input representations include FEN or algebraic-notation encodings of the board state and the UCI-encoded candidate move. The primary head outputs a scalar or binned estimate of the move's win probability—i.e., an action-value $Q_\theta(s, a) \in [0,1]$ indicating the estimated probability of winning when playing move $a$ from position $s$. This model is supervised on datasets exceeding 15 billion (state, action) pairs, with move values assigned via a superhuman engine such as Stockfish. The loss function is typically a variant of regression loss (such as HL-Gauss) over win probability, or a classification loss if win rates are binned.

The action-value network's role is to provide expert-derived, dense, and continuous reward signals for downstream learning systems, including both reinforcement learning (RL) and supervised fine-tuning (SFT) of LLMs.

2. Dense Reward Distillation to LLMs

Dense reward distillation leverages the chess-pretrained action-value network to supply a continuous and nuanced reward for every move generated by an LLM. For any candidate move, the LLM queries $Q_\theta(s, a)$, treating the resulting win probability as a reward signal. Two variants are used:

• Direct win-rate feedback: $r = Q_\theta(s, a)$. This captures how close the chosen move is to optimal in the expert model's judgment.
• Normalized rank-based feedback: For $L$ possible moves ranked by win-rate, the reward is normalized: $$r_{\text{dense}} = \frac{\text{rank}(a_t) - 1}{L - 1}$$. This makes feedback invariant to absolute win-rate ranges for different positions.

This stands in contrast to sparse binary rewards, where only the exact best move is considered correct: $r_{\text{sparse}} = \mathbf{1}[\hat{a}_t = a_t^*]$. Dense critic-based rewards are distributed across the move space, so LLMs receive informative feedback for sub-optimal but reasonable choices.

Within RL fine-tuning (e.g., using Generalized Reward Policy Optimization / GRPO), this reward guides LLM policy gradients, distilling positional and tactical evaluations of the expert model into the generative model's decision distribution.

3. Empirical Results: Performance and Plateau

Using chess-pretrained action-value networks as critique for RL tuning of LLMs demonstrably outperforms sparse binary rewards. Multiple LLMs (Qwen2.5-3B, Qwen2.5-7B, Llama3.1-8B) fine-tuned on chess puzzles with dense rewards derived from $Q_\theta$ achieve substantial short-term gains versus those using only sparse feedback, which often fail to learn except in larger models. Dense reward models also outperform comparable SFT baselines.

However, all approaches plateau at sub-expert performance, with held-out chess tactical puzzle accuracy stuck around 25–30%. By comparison, an "expert" 1800 Elo chess engine achieves $\sim$66.5% on the same puzzles. No combination of RL, dense reward, or SFT produces robust expertise (as measured by move quality, internal state tracking, or simulated board reasoning). Training with explicit reasoning traces (language explanations of move choice) improves linguistic quality but not chess performance.

Model	Board State Accuracy	MATE Puzzle Accuracy
Qwen2.5-3B	0.0%	35.8%
Qwen2.5-3B-It	0.0%	53.7%
Qwen2.5-7B	0.0%	42.7%
Qwen2.5-7B-It	0.0%	52.0%
Llama3.1-8B	0.0%	12.7%
Llama3.1-8B-It	0.0%	52.4%

Italics denote models trained with iterative reasoning SFT data (OpenAI o3 traces).

4. Limitations: Strategic Reasoning and the Knowledge Ceiling

A principal finding is that the limiting factor in LLM chess performance is not the form or density of reward, but the domain knowledge encoded in the pretrained model itself. All models fail diagnostic tasks involving internal state tracking (e.g., updating the board after a move) and struggle with basic tactics outside the most literal problem cases. RL fine-tuning and critic-guided distillation cannot compensate for deep knowledge gaps.

This deficit contrasts with RL successes in math and logic, which draw on greater overlap between LLM pretraining and the target domain. In chess, domain knowledge is largely absent from the LLM, so RL amplifies surface-level pattern matching rather than facilitating genuine strategic planning. The empirical upper bound for LLM performance (without chess-rich pretraining) thus remains well below human expert or strong engine levels, even with access to dense, expert-informed rewards.

5. Comparative and Broader Implications

The chess-pretrained action-value network approach demonstrates clear benefits over sparse RL for teaching chess-relevant selection to LLMs, enabling them to better leverage the chess knowledge encapsulated in their weights. However, the inability to achieve or approach true expert-level strategic reasoning highlights the necessity of domain-rich pretraining. In contrast, models explicitly pretrained on large chess corpora or designed for direct state evaluation (such as the transformer-based action-value networks themselves (Ruoss et al., 7 Feb 2024)), can achieve near-grandmaster play via direct inference or in combination with search, even in "searchless" settings.

The findings suggest that future research should prioritize:

• Domain-adaptive pretraining: Augmenting LLMs with large, annotated chess datasets during pretraining to bootstrap internal knowledge relevant for RL.
• Hybrid approaches: Pairing LLMs with explicit simulators to handle state-tracking and move legality, reserving the LLM for tactical and explanation tasks.
• Reward and architecture innovations: Further experiments with dense value-based distillation, hierarchical reward guidance, and neural-symbolic integrative models.

Approach	Chess Knowledge Needed	Reasoning Gain	Expert-Level Performance Possible?
RLVR (Dense Reward)	Yes	Modest	No (without chess exposure)
RLVR (Sparse Reward)	Yes	Little/None	No
RLVR + Rich Pretraining	Yes	???	Possibly (not yet tested)

6. Summary

Chess-pretrained action-value networks, derived from large-scale supervised learning on expert engine data, now serve as effective dense reward and knowledge distillation models for RL in chess and are key ingredients for LLM instruction in chess domains. While they enable structured, gradient-rich policy improvement, their efficacy is ultimately limited by the underlying chess expertise of the student model. Results in strategic reasoning highlight that RL and dense distillation cannot substitute for foundational domain knowledge, and that meaningful progress toward LLM-based strategic reasoning in chess will likely require chess-expert-level pretraining or direct integration with symbolic simulators.