Unpuzzles Dataset Benchmark

• The Unpuzzles Dataset is a benchmark that trivializes well-known logic puzzles via minimal edits, challenging LLMs to rely on inference over memorization.
• It includes context-shifted variants that modify vocabulary and domains while preserving logical structure, aiding in diagnosing reasoning versus pattern matching.
• Evaluation shows that LLMs perform well on original puzzles but significantly falter on unpuzzles, highlighting gaps in out-of-distribution generalization.

The Unpuzzles Dataset refers to a benchmark comprising trivialized versions of well-known logic and mathematics puzzles, designed specifically to assess the genuine reasoning and generalization capabilities of LLMs independent of simple statistical pattern matching. The dataset was introduced in the paper "Frontier LLMs Still Struggle with Simple Reasoning Tasks" (Malek et al., 9 Jul 2025), where it is used to reveal that, despite their strong performance on traditional and competitive benchmarks, LLMs systematically fail on tasks that have been rendered “easy” by minimal logical or textual modifications that should make the solutions obvious to humans.

1. Dataset Construction and Structure

The Unpuzzles Dataset consists of two main components. The first section includes 97 original logic puzzles and brainteasers—widely distributed on public platforms—and for each such puzzle, a corresponding “unpuzzle” is crafted. The trivialization is performed by human annotators via targeted minimal edits (often only a few characters), which remove key inferential challenges in the original problem. Such edits might involve reducing a parameter, changing a critical number, or introducing a statement that makes the solution immediate.

The second subsection comprises 64 unpuzzles with categorical or numerical answers suitable for automatic evaluation. For these, the authors also generate context-shifted variants where, by automatic prompting, the puzzle content (i.e., characters, settings, vocabulary) is replaced while the logical structure and trivial answer remain unchanged. The context-shifted subset is specifically designed to diagnose whether model failures stem from memorization and pattern dependence or broader reasoning deficiencies.

Component	Quantity	Example Modification
Trivialized Puzzles	97	Changing key numbers to render the answer trivial
Context-Shifted Unpuzzles	64	Retelling in different domain (e.g. sports fans instead of animals)

This dataset structure facilitates rigorous differentiation between performance due to memorization and out-of-distribution logical reasoning.

2. Methodological Principles

Trivialization in the Unpuzzles Dataset is guided by the instruction to remove the core difficulty while altering the original problem as little as possible. Annotators are advised to edit, for example, parameter values or phrasing, to produce a puzzle where the logical answer can be derived immediately without the original complex reasoning.

Context-shifting is achieved via automated prompting, instructing the model to rewrite each unpuzzle in new domains with altered vocabulary or settings, yet keeping the logical relationships and structure intact. For example, replacing “purple chameleons” with “Spurs fans” but maintaining the same rules for color change and pairing.

The careful curation aims for maximum surface similarity between original and unpuzzle, with change localized to the logical fulcrum, ensuring that the dataset evaluates adaptation and reasoning rather than rote recall.

3. Representative Examples and Analytical Features

A canonical example is the chameleon puzzle. In its original form, the puzzle employs an invariant modulo 3 (e.g., $S = P + 2Y \pmod{3}$ for purple, yellow, and maroon chameleons) and demands sophisticated deduction. In the trivialized unpuzzle, a single parameter adjustment (e.g., changing "13 purple" to "15 purple") decimates the invariance and renders the answer ("yes") obvious. Surprisingly, LLMs continue to output invariant-driven reasoning in response to the unpuzzle, manifesting a phenomenon described in the paper as "reasoning delirium" or "context corruption."

A plausible implication is that memorization of sophisticated solution chains for well-known puzzles overrides the model’s recognition of trivially modified, out-of-distribution instances, even when direct logical inference should suffice.

4. Evaluation Outcomes and Observed Failure Patterns

Empirical results indicate a sharp dichotomy in LLM performance on original puzzles versus unpuzzles. While models such as Gemini 1.5, Claude, and GPT-4 show 79–87% accuracy on original problems, their accuracy on trivialized versions generally collapses, sometimes to as low as 17–62%, depending on the model (see summary table in the paper).

Key failure patterns documented include:

• Context corruption: Model outputs often include irrelevant or incorrect intermediate steps derived from the original, non-trivial puzzle, despite the trivial solution.
• Overthinking (reasoning delirium): Models produce elaborate proofs or invariants for unpuzzles where a basic answer and short deduction suffice.
• Improved performance on context-shifted unpuzzles: Changing the problem context boosts model accuracy, suggesting that failures are substantially due to surface-pattern memorization rather than logical incapacity.

This suggests that models are not inherently unable to solve trivialized logic problems but are strongly biased toward memorized solution forms.

5. Implications for Out-of-Distribution Generalization

The findings from the Unpuzzles Dataset challenge the assumption that simplifying a task (i.e., trivializing it) ensures improved LLM performance. Instead, prominent LLMs remain susceptible to out-of-distribution generalization gaps, particularly when the surface form matches a previously memorized hard problem but the logical substance no longer does.

A plausible implication is that superficial task ease does not guarantee robust logical reasoning from frontier LLMs; statistical shortcuts and training distribution artifacts play a dominant role in driving model behavior on "easy" reasoning problems.

6. Relevant Formulas and Procedural Details

The dataset’s analytical backbone includes both explicit logical formulas and illustrative pseudocode for problem generation and evaluation. For example:

• The chameleon puzzle invariant: $S = P + 2Y \pmod{3}$, with $P$, $Y$, and $M$ denoting counts of distinct color entities.
• Procedural logic formula generation pseudocode (abbreviated):

  Algorithm: SampleLogicFormula(max_depth, p_d, …) If depth = max_depth or random() ≥ p_d: return atomic proposition Else: select operator (∧, ∨, ¬, …), recursively apply

These serve both as generative templates and diagnostic tools for assessing where model reasoning diverges from expected algorithmic processes.

7. Significance and Future Directions

The introduction of the Unpuzzles Dataset marks a methodological advance in LLM evaluation, foregrounding the critical issue of reasoning under triviality and distributional shift. The demonstrated performance collapse on trivialized logic tasks exposes current model deficiencies in systematic reasoning, abstraction, and context adaptation.

A plausible implication is that future model designs must address not only scaling and data diversity but also mechanisms for flexible, context-sensitive reasoning. Robust generalization across minor perturbations in surface form remains a pressing challenge for both the current and next generation of LLMs.

In conclusion, the Unpuzzles Dataset provides a rigorous testbed for probing memorization-driven reasoning, adaptability, and failure patterns in advanced LLMs. It reveals substantial gaps in out-of-distribution robustness even for tasks that should, by construction, be easy for both humans and machines, and highlights generalization as a key unresolved problem in machine reasoning research (Malek et al., 9 Jul 2025).