Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning

Abstract: Learning and skill mastery require extensive and deliberate practice. In many learning settings, producing high-quality pedagogical materials can require a high level of domain expertise and be very time-consuming. Pedagogical materials often need to train students to engage in different thinking patterns. In some domains, such as chess, puzzles are used to help students practice their skills in calculating the next moves and recognizing known patterns on a board. Giving students a practice set of puzzles to help them learn different modes of thinking is challenging because the teacher needs to carefully balance between different motifs and how many look-ahead steps a student needs to perform. Popular online platforms like Chess.com and Lichess offer players millions of puzzles. Unlike chess tactics puzzles procured by human experts, where chess beginners can learn valuable insights, these puzzles are automatically generated and often regarded as having low pedagogical value. These platforms also rely on a heuristic to recommend puzzles to users for practice. Using the user history data over an entire year, a total of 1.5 billion puzzle-solving histories, we learn the pedagogical value of a puzzle and how to automatically choose a set of puzzles to better support chess learners using insights from offline reinforcement learning. We show that using offline policy evaluation, our trained policy has significant impact on beginners with puzzle-solving Elo range of 100--1000, particularly for the group of beginners whose learning growth was stagnant. We also performed a qualitative analysis of the puzzles discovered by our model by collecting annotation ratings from expert chess players. The success of our pipeline shows promise for a future where we can understand the pedagogical values of practice items given general user interaction data.

• The paper develops a transformer-based offline reinforcement-learning policy that uses user histories, board positions, puzzle features, and advantage-weighted actor-critic training to recommend among 441,113 puzzles.
• The paper reports estimated return gains for users below 1000 puzzle Elo, including an increase from 14.3 to 52.9 for stagnant users rated 100–600 and an overall improvement of approximately 110.1% over the existing heuristic.
• The paper finds modest expert and LLM-judged improvements in puzzle fun and rating appropriateness, but its offline, surrogate-reward evaluation cannot establish durable chess-skill gains without a prospective randomized trial.

Problem formulation and empirical setting

The paper formulates chess-puzzle recommendation as an offline reinforcement-learning problem rather than as a static difficulty-matching task. The central premise is that a puzzle’s pedagogical value depends jointly on its intrinsic properties, the learner’s current state, and the sequence of puzzles previously attempted. This reframes recommendation from selecting any puzzle within a rating interval to estimating the expected learning utility of candidate puzzles under a personalized policy.

The study uses a large-scale interaction dataset collected from Chess.com between March 2021 and March 2022. It contains 1,536,254,297 puzzle-solving interactions from 3,132,428 active users and 441,113 unique puzzles. Users solved, on average, 490.4 puzzles during the observation period. The data are highly temporally clustered: 96.9% of puzzle attempts occur within three minutes of another attempt, with an average burst containing 5.1 puzzles and bursts separated by approximately two and a half days. These statistics motivate a sequential formulation in which short-term puzzle histories provide meaningful context for subsequent recommendation.

Chess.com’s deployed policy is described as bucketed uniform sampling. It selects puzzles whose puzzle Elo lies within approximately $\pm 200$ points of the user’s Elo, then progressively shifts the interval toward easier puzzles after incorrect attempts. The policy therefore incorporates the user’s current rating and recent performance, but it does not explicitly learn which puzzles produce favorable subsequent outcomes or which motifs and structural properties are pedagogically valuable.

The paper also analyzes user-level rating trajectories. Players are partitioned into growth and stagnant groups according to their relative Elo changes within rating ranges. Growth users exhibit increasing puzzle Elo, whereas stagnant users show little change. The relationship is not uniform across ability levels: higher-rated growth users improve more rapidly during their first 50 attempts, while low-rated growth users initially decline before improving. This initial dip is consistent with an acclimatization effect, but the observational design cannot distinguish learning from selection, persistence, rating dynamics, or changes in the difficulty distribution.

Figure 1: User puzzle-Elo trajectories separated by initial rating and relative growth, including rapidly improving and stagnant groups.

The dataset also exposes substantial heterogeneity in puzzle content. Puzzle lengths range from one to 16 moves, with a mean of 2.6 moves. Puzzle ratings range from 100 to 4000, and puzzles contain an average of 5.6 annotated motifs. However, 26% of puzzles have no motifs. Thus, puzzle Elo and move count provide only partial descriptions of content quality; the representation must also encode board positions and other puzzle-specific features.

Offline RL objective

The method uses a causal transformer as a non-Markovian policy over user–puzzle histories. The state includes a fixed-length sequence of previous interactions together with user features such as puzzle Elo and recent correctness. The action space consists of the 441,113 individual puzzles. A direct softmax over this action space is computationally expensive, particularly when policy learning requires repeated evaluation of action probabilities and action-value estimates.

The policy is derived from a KL-regularized actor-critic objective. The policy is encouraged to select actions with high estimated value while remaining close to the behavior policy that generated the data. This regularization is important in offline RL because unconstrained policy improvement can select poorly supported actions, producing extrapolation error in the learned $Q$ function. Following advantage-weighted actor-critic and implicit-Q-learning-style constructions, the policy is trained by weighted maximum likelihood. Dataset actions receive weights proportional to the exponentiated estimated advantage, $Q(s,a)-V(s)$, scaled by a temperature parameter $\beta$. High-advantage historical actions therefore exert greater influence on the policy than low-advantage actions.

The value function is trained with expectile regression, while the action-value function uses a temporal-difference target involving the observed reward and the next-state value. This separation permits advantage estimation entirely from logged transitions, without learning an explicit transition model or interacting with the live platform.

The architecture combines three components. A user embedder maps user Elo and correctness features into normalized latent vectors. A puzzle embedder combines a learned puzzle identity representation, first-move information, and a CNN encoding of the board position. The transformer processes sequences of user and puzzle embeddings and produces a history-dependent representation used to score candidate puzzles through inner products.

Figure 2: Transformer-based recommendation architecture combining user histories, board-level puzzle representations, policy learning, and value estimation.

The resulting policy assigns scores through a temperature-weighted inner product between the transformer state representation and each candidate puzzle embedding. During training, the full denominator over all puzzles is approximated using in-batch negatives. The implementation uses batches containing 65,000 candidate puzzles, sequence length 256, batch size 16, discount factor 0.99, and 25,000 optimization steps. This approximation makes training feasible, but it changes the exact policy objective: the learned normalization is over sampled negatives rather than the full action space. Its statistical effect depends on the negative-sampling distribution and is not fully characterized in the paper.

Reward design and interpretation

The reward is intended to represent immediate pedagogical utility rather than long-term chess improvement. For a puzzle containing $N$ solution moves, with $c$ correctly executed moves, the reward is defined as

$$r(s,a)=\frac{c}{N}\exp\left(\alpha(\mathrm{Elo}_{\mathrm{puzzle}}-\mathrm{Elo}_{\mathrm{user}})\right),$$

with $\alpha=0.002$. Incorrect attempts receive zero reward, while partially correct solutions receive rewards proportional to the fraction of correct moves and adjusted upward when the puzzle is harder than the user’s current rating.

This reward encodes a specific pedagogical tradeoff: solving a difficult puzzle correctly is preferable to solving an easy puzzle correctly, and partial success on a difficult puzzle can be comparable to complete success on a substantially easier puzzle. The implication is that the learned policy is optimized for challenge-weighted correctness, not directly for retention, knowledge transfer, Elo growth, motif coverage, or calibrated progression. The choice of $\alpha$ is justified through internal testing rather than an external learning-theoretic criterion, so the resulting policy’s behavior is conditional on this hand-designed utility function.

The reward also treats incorrectness as having no verifiable learning value. That assumption is consequential. Failed attempts may contribute to learning through error diagnosis, feedback, retrieval difficulty, or subsequent adaptation, but those effects are not represented unless they influence later observed transitions and are captured indirectly by the value function.

Offline policy evaluation

Because online experimentation was not performed, the primary comparison uses one-step importance sampling on held-out users. Full trajectory importance sampling is avoided because sequence-level products of behavior-policy ratios would have high variance. Instead, the method applies per-step policy corrections across 32-step segments and clips each ratio to the interval $[0.1,10]$. This reduces variance but introduces bias, both through truncation and through removal of historical importance weights.

The reported returns show large estimated gains for beginners and negligible differences at higher ratings.

User puzzle Elo	Behavior policy	Learned policy	Relative pattern
100–600, stagnant	$Q$0	$Q$1	Large gain
100–600, growth	$Q$2	$Q$3	Large gain
600–1000, stagnant	$Q$4	$Q$5	Significant gain
600–1000, growth	$Q$6	$Q$7	Significant gain
1000–1500, average	$Q$8	$Q$9	Approximately neutral
1500+, average	$Q(s,a)-V(s)$0	$Q(s,a)-V(s)$1	Approximately neutral

For users below 1000 Elo, the learned policy consistently outperforms the deployed heuristic according to the paper’s return estimator. The largest absolute improvement appears in the 100–600 stagnant group, where the estimated return increases from 14.3 to 52.9. Averaged over the lower-rated population, the authors report an improvement of approximately 110.1% over the existing system, noting that more than 80% of users are below 1500 Elo.

These results support the claim that puzzle selection is most consequential during early skill acquisition, at least under the study’s reward definition. The stronger effect for low-rated users does not establish that the policy increases chess skill or long-term Elo: the evaluated quantity is the estimated probability of solving puzzles weighted by relative difficulty. The absence of a meaningful improvement above 1000–1500 Elo may indicate diminishing recommendation sensitivity, insufficient support in the data, or limitations in the reward and policy representations.

The policy’s top recommendations generally remain near the behavior policy’s puzzle-rating range and the user’s rating. Nevertheless, it tends to select harder puzzles, particularly for higher-rated users. This provides a useful sanity check against pathological extrapolation: the policy does not simply recommend extreme-difficulty items. At the same time, the tendency to recommend harder puzzles is partly induced by the reward itself, which rewards difficulty conditional on correctness.

Expert and LLM-based qualitative evaluation

The qualitative study evaluates puzzle characteristics using a rubric covering calculation, pattern recognition, informativeness, rating appropriateness, quality, and fun. Eight expert annotators participated, including five titled players: two USCF Experts, a National Master, a FIDE Master, an International Master, and two Grandmasters. Their ratings span USCF Elos from 1990 to 2576.

Only 30 puzzles were directly annotated, with 12 shared across all experts and 18 additional puzzles distributed across annotators. To scale evaluation, the authors calibrate eight LLM judges, one for each expert, using six shared examples for prompt optimization and six for validation. The LLM judges receive the rubric, target-user Elo, and a board representation derived from FEN. This procedure is useful for exploratory comparison but provides limited independent validation because the judges are calibrated on a very small number of expert examples and are evaluated on closely related judgments.

The learned policy receives higher mean scores on most dimensions:

Criterion	Chess.com policy	Learned policy	Difference
Calculation	70.17	72.39	+2.22
Pattern recognition	58.89	61.48	+2.59
Fun	65.57	70.34	+4.77
Rating appropriateness	67.95	72.73	+4.78
Quality	69.83	71.31	+1.48
Informativeness	75.11	75.11	0.00

The statistically significant differences are in fun ($Q(s,a)-V(s)$2) and rating appropriateness ($Q(s,a)-V(s)$3). The largest numerical gains are therefore not in informativeness or holistic quality, but in perceived enjoyment and difficulty matching. This is consistent with a policy that learns to select puzzles producing favorable solve outcomes at an appropriate challenge level, but it does not demonstrate that the recommended puzzles teach more transferable chess concepts.

The expert examples further illustrate that puzzle quality is multidimensional. A puzzle can receive a high holistic quality score while still being criticized for testing a banal motif or following an uninformative continuation. Conversely, a puzzle selected by the learned policy may be valued for unusual tactical combinations and greater calculation demands. These examples support the paper’s contention that rating-based recommendation alone does not capture pedagogical distinctions between automatically generated puzzles.

Figure 3: Example of a low-calculation puzzle in which the forced continuation is essentially determined at each move.

The qualitative findings should consequently be read as preliminary evidence that the learned policy changes the composition of recommended puzzles in expert-recognizable ways. They do not constitute a large-scale human preference study, nor do they establish agreement between LLM judgments and independent expert evaluation beyond the limited calibration set.

Limitations and open questions

The principal limitation is the absence of an online randomized controlled trial. All policy-performance claims are based on offline evaluation, and the chosen one-step importance-sampling estimator is explicitly biased. Clipping ratios to $Q(s,a)-V(s)$4 improves stability but prevents unbiased estimation of the target policy value. In addition, the behavior policy is described as bucketed uniform sampling, yet the effective logging distribution may include implementation details, filtering, availability constraints, repeated-puzzle effects, and unobserved user-state variables.

The dataset is observational and excludes player-versus-player game information. Puzzle Elo is therefore treated as the relevant skill measure, although it is a platform-specific rating that evolves through the same interaction process being modeled. This creates potential feedback and measurement confounding: users who persist may both solve more puzzles and receive different rating trajectories, while stagnant users may differ systematically in motivation, prior chess knowledge, or exposure to other learning resources.

The reward function is also a substantive assumption rather than a validated learning outcome. It privileges correct solutions and difficulty-weighted performance, but does not directly measure delayed retention, motif mastery, strategic transfer, enjoyment over repeated sessions, or player-versus-player performance. The reported 110.1% improvement is consequently an improvement in an estimated surrogate return, not a 110.1% improvement in chess learning.

Finally, the model’s scalability mechanism uses in-batch negative sampling rather than exact normalization over 441,113 actions. The consequences for calibration, ranking quality, and probability-ratio estimation are not reported in detail. The qualitative evaluation is small, and its LLM component is dependent on six-shot expert-specific calibration. The paper therefore leaves open whether the learned policy would improve measured chess ability in a prospective trial, whether its gains persist over longer horizons, and whether its advantage remains after controlling for user persistence and exposure.

Conclusion

The paper presents a coherent large-scale application of offline RL to educational recommendation. Its technical contribution is an advantage-weighted actor-critic objective combined with transformer histories and continuous puzzle embeddings, making policy learning tractable over hundreds of thousands of discrete actions. Empirically, the learned policy substantially improves the paper’s offline surrogate return for users between 100 and 1000 puzzle Elo, with especially large gains among low-rated and stagnant users, while producing modest qualitative improvements in perceived fun and rating appropriateness.

The evidence supports the narrower conclusion that historical interaction data can identify puzzle-selection policies that differ favorably from a rating-bucket heuristic under offline evaluation. Whether those differences produce durable chess learning remains unresolved and requires prospective experimentation with outcomes beyond immediate, difficulty-weighted puzzle correctness.