Pencil Puzzle Bench

• Pencil Puzzle Bench is a benchmark suite that rigorously evaluates AI systems' reasoning using deterministic, unique-solution puzzles such as Sudoku, Nurikabe, and Slitherlink.
• It utilizes both single-shot and iterative, agentic protocols that provide granular per-move verification and dense feedback for modeling constraint-based logic.
• The framework supports multimodal inputs and process supervision, driving research into verifiable reasoning, hybrid search, and overcoming data leakage in AI evaluations.

A pencil puzzle bench is a rigorously defined benchmark suite for evaluating reasoning, perception, and multi-step problem-solving capabilities, predominantly in LLMs and large vision-LLMs (LVLMs), using the domain of deterministic constraint-satisfaction puzzles commonly known as pencil puzzles. These puzzles—exemplified by Sudoku, Kakuro, Slitherlink, and numerous derivatives—require structured, verifiable, and creative sequential reasoning. Recent benchmark initiatives systematically formalize the evaluation of both textual and multimodal models on extensive, diverse, and precisely curated puzzle corpora, establishing controlled measurement protocols, verifiability at intermediate steps, and pathways for dense process supervision and reinforcement learning.

1. Purpose and Scope

Pencil puzzle bench frameworks address fundamental limitations in existing reasoning benchmarks by leveraging the intrinsic properties of pencil puzzles:

• Uniqueness and determinism: Each instance is guaranteed to have a unique solution, typically verified via SAT solvers, enabling unambiguous assessment.
• Step-wise, process-driven reasoning: Solutions require extended sequences of logic operations and support granular verification at the move level, unlike static benchmarks that permit only end-point checking.
• Expressivity and diversity: Benchmarks span dozens of puzzle varieties—Sudoku, Nurikabe, Slitherlink, Nonogram, and others—with widely varying rule systems (arithmetic, connectivity, matching, region constraint) and difficulty regimes.
• Resistance to memorization and contamination: Many puzzles are specifically chosen to avoid overlap with online solutions, reducing data leakage risks in model evaluation.

Pencil puzzle benches thus provide demanding, yet reproducible, settings for evaluating both short-horizon manipulation and long-form strategic reasoning, across pure text and multimodal input modalities (Waugh, 2 Mar 2026, Ren et al., 29 Mar 2025, Seely et al., 22 May 2025).

2. Dataset Construction and Curation

Contemporary pencil puzzle benches are constructed from large databases of verified puzzles, with stratified sampling to control for both variety and difficulty:

• Pencil Puzzle Bench (2026): Draws 300 test puzzles from a SAT-verified corpus of 62,231 puzzles spanning 94 varieties, selecting 20 representative varieties. For each, 15 puzzles are chosen (5 each for "short," "medium," and "long" solution-path quantiles). Key dataset properties are solution uniqueness, intermediate state verifiability, and low contamination risk (Waugh, 2 Mar 2026).
• VGRP-Bench: Defines a suite of 20 grid-based pencil puzzles (Sudoku, Killer Sudoku, Nonogram, Trees-and-Tents, etc.), grouping instances by grid size ($N \times N$), clue density $h\in[0,1]$, and rule complexity, enabling controlled difficulty scaling (Ren et al., 29 Mar 2025).
• Sudoku-Bench: Specializes in creative Sudoku variants, curating 100 puzzles across $4\times4$, $6\times6$, and $9\times9$ grids, with an emphasis on constraint interactions that defy memorized solution templates. Each puzzle is specified via natural-language rules, standardized text grid, and visual element annotations; all solutions are unique (Seely et al., 22 May 2025).

Dataset extension is supported through standardized APIs (e.g., SudokuPad integration) and open-source harnesses that parse text or graphical puzzle representations.

3. Formalization and Constraint Verification

Every pencil puzzle variety is specified by a formal constraint-satisfaction model. For example:

• Sudoku (standard): Assign $x_{i,j} \in \{1,\dots,N\}$ to each cell. Constrain rows, columns, blocks to have unique digits; enforce solution uniqueness.
• Nurikabe: Use binary shading variables $s_{i,j} \in \{0,1\}$ and region-size constraints per clue; require global connectivity and exclusion of $2\times2$ fully shaded blocks.
• Slitherlink: Binary edge variables $e_{v\to w} \in \{0,1\}$ enforce degree constraints and the formation of a single-cycle loop.

Every move by a solver—model or human—can be validated by checking that all rule-specific constraints $C^V_i$ are satisfied in the new board state. This enables per-move feedback such as precise localization of rule violations, immediate detection of invalid actions, and fine-grained reward shaping for learned algorithms (Waugh, 2 Mar 2026).

4. Evaluation Protocols and Metrics

Multiple protocols have been adopted to probe distinct solver competencies:

• Direct ask (single-shot): The model is provided with a full puzzle description (ASCII or image, plus rules), and returns a complete solution in a single inference. The solution is then fully validated.
• Agentic (multi-turn with iterative verification): The model interacts with a tool API, making sequential moves and receiving automated, rule-specific feedback after each action. Iteration continues until solution, error, or move/turn timeout.
• Vision-language protocols: Puzzles are rendered as images, paired with a rules prompt; models must infer both perception (cell content extraction) and reasoning.

Metrics are systematically decomposed:

• Puzzle-solve rate: Ratio of puzzles solved completely.
• Cell-level accuracy: For perception tasks, fraction of correctly read or assigned cells.
• Step-level rule adherence: Rate of valid action judgments.
• Agentic metrics: Median number of turns per solve (29), duration (median 17 min), and maximum session lengths (1,221 turns, 14.3h) have been observed (Waugh, 2 Mar 2026).
• Effort scaling: Performance as a function of model "thinking depth" or iteration count, with factors up to $h\in[0,1]$0 improvement between minimal and maximal inference effort.

5. Empirical Findings and Model Capabilities

Experimental analyses across benchmarks consistently demonstrate acute limitations in current LLMs and LVLMs:

• Generalization: Models fine-tuned on a narrow puzzle class (e.g., Sudoku) perform poorly on structurally divergent variants (e.g., Trees-and-Tents, Aquarium) (Ren et al., 29 Mar 2025).
• Single-shot versus agentic gap: Iterative, feedback-driven solving provides 4–30 percentage point boosts, even for advanced models (e.g., GPT-5.2@xhigh, 27% direct-ask → 56% agentic). Some models move from near-zero to substantial accuracy via iteration (Waugh, 2 Mar 2026).
• Difficulty scaling: Larger grids ($h\in[0,1]$1), lower clue density ($h\in[0,1]$2), and greater rule complexity each sharply degrade performance, often driving puzzle-solve rates below 1% for nontrivial instances (Ren et al., 29 Mar 2025, Seely et al., 22 May 2025).
• Error modes: Typical failures include confidently producing incorrect or contradictory full solutions, premature surrender, and missing subtle constraint interactions.
• Effort scaling: Increasing model inference steps or promoting explicit chain-of-thought reasoning multiplies accuracy (e.g., GPT-5.2 rises $h\in[0,1]$3 from no effort to maximal agentic iteration) (Waugh, 2 Mar 2026).
• Baselines: On curated hard variants (e.g., Sudoku-Bench), no leading model exceeds 15% full-solve rate unaided on $h\in[0,1]$4 puzzles.

A selection of top-performing models and their regime-specific performance is shown:

Model	Direct‐Ask	Agentic
GPT-5.2@xhigh	27.0 %	56.0 %
Claude Opus 4.6-1m	0.0 %	36.7 %
Gemini 3.1 Pro	20.0 %	33.3 %

6. Implications, Challenges, and Research Vectors

The pencil puzzle bench paradigm reveals fundamental obstacles and opportunities:

• State-tracking: Most LLM architectures lack persistent, interpretable memory for puzzle state; advances may require differentiable memory structures or symbolic-integrated hybrids.
• Local-global reasoning: Modular "constraint networks" capable of handling arbitrary rule sets and dynamically composing new constraint systems are needed (Ren et al., 29 Mar 2025).
• Verifiability and RL: Step-level verification enables dense reward signals for process-supervised reinforcement learning, supporting curriculum learning by solution compressibility and localized feedback (Waugh, 2 Mar 2026).
• Benchmark extensibility: APIs and dataset tools permit rapid addition of new varieties and conversion between text, SVG, and raster; this supports both discrimination between reasoning architectures and research into multimodal integration.
• Memorization resistance and real creativity: Puzzles that demand "break-ins"—non-template logical leaps—expose the limits of both shallow heuristic search and massive memorization, incentivizing models that learn genuine abstraction.

A plausible implication is that robust progress on pencil puzzle benches may transfer to broader, tool-augmented, and grounded problem-solving domains, where flexible plan adaptation and verifiable intermediate state tracking are critical.

7. Future Directions and Extensions

Several trends and ongoing research areas are emerging:

• Process supervision: Development of process reward models and curricula targeting not only outcome but quality and trajectory of reasoning steps.
• Tool-use and hybrid search: Incorporating external solvers (e.g., constraint propagation, SAT, SMT) as callable modules within agentic solution strategies.
• Multimodal expansion: Integrating vision-based input with textual logic (e.g., PuzzleBench, VGRP-Bench), thereby evaluating joint perception-reasoning pipelines under rigorous, controlled settings (Zhang et al., 15 Apr 2025, Ren et al., 29 Mar 2025).
• Community benchmarks and human comparison: Leveraging extensive annotated transcripts (e.g., Cracking-the-Cryptic action logs) for imitation learning and fine-grained human-model comparator studies (Seely et al., 22 May 2025).
• Extension beyond puzzles: Applying the principles of granular, per-step verifiability and agentic, feedback-driven tool use to other domains of scientific and mathematical discovery.

Thus, the pencil puzzle bench serves as a critical infrastructure layer for rigorous, nuanced evaluation of long-horizon, transparent, and creative reasoning in AI systems, paralleling how human solvers “think on paper” through sustained, explicit constraint tracking and adaptive logic.