Evaluating In Silico Creativity: An Expert Review of AI Chess Compositions

Abstract: The rapid advancement of Generative AI has raised significant questions regarding its ability to produce creative and novel outputs. Our recent work investigates this question within the domain of chess puzzles and presents an AI system designed to generate puzzles characterized by aesthetic appeal, novelty, counter-intuitive and unique solutions. We briefly discuss our method below and refer the reader to the technical paper for more details. To assess our system's creativity, we presented a curated booklet of AI-generated puzzles to three world-renowned experts: International Master for chess compositions Amatzia Avni, Grandmaster Jonathan Levitt, and Grandmaster Matthew Sadler. All three are noted authors on chess aesthetics and the evolving role of computers in the game. They were asked to select their favorites and explain what made them appealing, considering qualities such as their creativity, level of challenge, or aesthetic design.

• The paper presents an AI pipeline that integrates large-scale generative models with a custom reinforcement learning reward to generate unique and counter-intuitive chess puzzles.
• It uses a hybrid filtering strategy combining theme detection and expert manual review to ensure both aesthetic appeal and tactical quality.
• Expert evaluations reveal varied perceptions of creativity, underscoring the promise and current limitations of AI in chess composition.

Evaluating In Silico Creativity: An Expert Review of AI Chess Compositions

Introduction

This work addresses the challenge of evaluating and eliciting creativity in generative AI systems, focusing on the domain of chess composition. The authors present a pipeline for generating chess puzzles using large-scale generative models, followed by a rigorous expert review to assess the creative merit of the generated content. The study is notable for its integration of modern generative modeling, reinforcement learning with custom reward functions, and a human-in-the-loop evaluation protocol involving world-class chess composition experts.

Generative Modeling and Reward Design

The core of the system is a suite of generative neural models—Auto-Regressive Transformers, Discrete Diffusion, and MaskGit—trained on a corpus of 4 million Lichess puzzles. Each chess position is encoded as a FEN string, and the models are trained to autoregressively predict the next character, effectively learning the distribution of plausible chess puzzles.

To move beyond mere imitation of the training data, the authors introduce a reinforcement learning (RL) phase. The RL reward is a composite of two criteria:

• Uniqueness: Ensures the puzzle has a single correct solution, as verified by a strong chess engine.
• Counter-intuitiveness: The solution must be solvable by a strong engine but not by a weak one, favoring positions that are non-trivial for humans and less likely to be found in standard play.

This reward is used both for sample selection and as a signal for further RL fine-tuning, iteratively biasing the generative model toward more creative outputs.

Filtering and Human-in-the-Loop Selection

After generating approximately 4 million candidate puzzles, the system applies a hybrid filtering strategy. First, positions are ranked by the RL reward. Then, aesthetic theme detectors—trained to recognize motifs such as sacrifice, underpromotion, or interference—are applied. The detectors alone are insufficiently precise, but their effectiveness is enhanced by the initial reward-based ranking. The top 50 puzzles per theme are then manually reviewed, with validation from FIDE-rated players (2200–2300 Elo) to ensure playability and challenge.

Expert Evaluation Protocol

A curated booklet of selected puzzles was sent to three leading experts: IM Amatzia Avni, GM Jonathan Levitt, and GM Matthew Sadler. Each expert was asked to select and comment on their favorite puzzles, focusing on creativity, challenge, and aesthetic value. The experts' feedback is integrated into the analysis, providing a nuanced, domain-specific assessment of the generated content.

Analysis of Expert Feedback

The experts' selections and commentary reveal several key findings:

• Subjectivity of Creativity: There was little consensus among the experts regarding which puzzles were most creative, underscoring the subjective nature of aesthetic and creative evaluation in chess.
• Positive Reception of Novelty: The experts highlighted the originality, paradoxical motifs, and counter-intuitive solutions in many puzzles. Notably, one puzzle (starting with 1.Rg6+!) received unanimous acclaim for its unorthodox double rook sacrifice and geometric queen maneuvering.
• Critique of Depth and Realism: Some puzzles were deemed too trivial or unrealistic, lacking the depth and complexity of traditional endgame studies. The experts recommended increasing the complexity, introducing more robust counterplay, and combining multiple themes in future iterations.

Representative Examples

The paper provides detailed analysis of several puzzles, including:

• Unanimous Favorite: A position where White sacrifices both rooks to open a diagonal for the queen, culminating in a geometric sequence that is difficult for humans to find.
• Underpromotion and Smothered Mate: Puzzles featuring underpromotion to a knight, combined with smothered mate motifs, which are rare even in human composition.
• Stalemate and Paradox: Positions where the only path to a draw or win involves a sequence of sacrifices leading to stalemate or a surprising reversal.

These examples demonstrate the system's ability to generate puzzles that are not only novel but also exhibit deep tactical and geometric themes.

Methodological Implications

The approach demonstrates that generative models, when combined with carefully designed reward functions and human-in-the-loop selection, can produce content that is competitive with human creativity in a highly formalized domain. The RL reward design is critical: by explicitly encoding uniqueness and counter-intuitiveness, the system avoids trivial or repetitive outputs and instead discovers positions that challenge both engines and human solvers.

The hybrid filtering strategy—combining automated ranking with theme detection and manual review—proves effective in surfacing high-quality, creative puzzles from a vast candidate pool. This pipeline is generalizable to other domains where creativity is valued but hard to formalize.

Limitations and Future Directions

The primary limitations identified are:

• Depth and Realism: Some generated puzzles lack the depth and naturalness of human-composed studies, occasionally featuring unrealistic piece placements or insufficiently complex sidelines.
• Theme Combination: The system could be improved by explicitly encouraging the combination of multiple creative motifs within a single puzzle.
• Subjectivity of Evaluation: The lack of consensus among experts highlights the need for more robust, possibly multi-dimensional, metrics for creativity.

Future work should focus on:

• Enhancing the RL reward to capture additional aspects of creativity, such as thematic richness and positional realism.
• Incorporating adversarial or co-creative human feedback loops to further refine the generative process.
• Extending the methodology to other games and problem-solving domains, testing the generality of the approach.

Implications for AI Creativity

This study provides evidence that generative AI, when guided by domain-specific reward functions and expert evaluation, can produce outputs that are recognized as creative by human experts. The findings have broader implications for computational creativity, suggesting that similar pipelines could be applied to other structured domains (e.g., Go, Shogi, mathematical problem composition) and, with appropriate adaptation, to less formalized creative tasks.

The work also raises important questions about the nature of creativity in AI: to what extent can formal reward functions capture the richness of human aesthetic judgment, and how can subjective human feedback be systematically integrated into the training loop?

Conclusion

The paper presents a comprehensive framework for generating and evaluating creative chess puzzles using generative models, RL-based reward shaping, and expert human review. The system is capable of producing puzzles that are original, challenging, and aesthetically valued by leading experts, though further work is needed to match the depth and complexity of the best human compositions. The methodology offers a promising template for computational creativity in other domains, and the expert review protocol provides a valuable model for rigorous, domain-specific evaluation of AI-generated content.