Legal Puzzles: Exposing Legal Reasoning Flaws

• Legal puzzles are structured legal reasoning problems that challenge models and experts by examining interactions between doctrine, evidence, and statutory language.
• They serve as evaluation benchmarks across contexts like Legal Zero-Days, appellate case analyses, and retrieval-and-reasoning tasks to reveal instability and ambiguity.
• By exposing hidden vulnerabilities, retrieval failures, and formalization disagreements, legal puzzles offer actionable insights for enhancing AI reliability and legal audit methodologies.

Legal puzzles are structured legal reasoning problems in which the difficulty lies not merely in recalling doctrine, but in tracing how facts, evidentiary standards, statutory language, formal representations, or retrieval pipelines interact to determine legal consequences. Across recent research, the term refers to several adjacent objects: hard appellate-style case questions used to test LLM stability, criminal-law question–answer tasks for legal RAG, symbolic reconstructions of adjudication, expert-crafted instruments for detecting simulated Legal Zero-Days, and edge cases generated from competing formalizations of the same provision (Blair-Stanek et al., 28 Jan 2025, Butler et al., 2 Mar 2026, Mascardi et al., 2020, Sadler et al., 12 Aug 2025, Vernie et al., 24 May 2026). Taken together, these works treat legal puzzles as a methodological device for exposing where legal reasoning is fragile, ambiguous, or sensitive to hidden assumptions.

1. Definitions and scope

In "Legal Zero-Days: A Novel Risk Vector for Advanced AI Systems" (Sadler et al., 12 Aug 2025), legal puzzles are expert-crafted evaluation instruments designed to measure an AI model’s ability to trace complex legal logic across multiple statutes and detect consequential vulnerabilities—simulated Legal Zero-Days—before those flaws would be deployed in real legislation. Their stated purpose is to provide an ethical, controlled way to assess dangerous capabilities in legal reasoning without testing on real, live legal systems. The puzzles span multiple jurisdictions and legal domains, including telecommunications, food safety, data privacy, electronic transactions, copyright, and citizenship law, and they target technically complex and load-bearing provisions. Each puzzle contains selected original legislation or framework, a modified act containing the introduced vulnerability, and a detailed expert explanation of how the modification breaks core functionality.

In "LLMs Provide Unstable Answers to Legal Questions" (Blair-Stanek et al., 28 Jan 2025), the expression "Legal Puzzles" is used for a dataset of 500 hard questions distilled from published U.S. federal Courts of Appeal decisions with a dissent. Each item presents facts, competing legal arguments, and a direct question asking which party should prevail. Here the puzzle arises from contested legal issues, majority–dissent disagreement, and the need to map a fixed factual record onto a legal outcome. The structure is deliberately adversarial for stability analysis: when provided the exact same question, models sometimes favor one party and sometimes the other.

In "Legal RAG Bench: an end-to-end benchmark for legal RAG" (Butler et al., 2 Mar 2026), criminal-law legal puzzles are formulated as end-to-end retrieval-and-reasoning tasks over the Victorian Criminal Charge Book. Each of the 100 hand-crafted questions is paired with a long-form expert answer and a canonical supporting passage. The design forces systems to retrieve the correct doctrinal source, apply it to a fact pattern, and produce a grounded answer verifiable against the retrieved text.

In "By Their Fruits You Will Know Them: Comparing Formalizations of Law by the Decisions They Encode" (Vernie et al., 24 May 2026), legal puzzles emerge as edge cases and disagreement patterns produced when different formalizations of the same legal provision yield divergent outcomes on concrete inputs. These are not merely ambiguous statutes in the abstract; they are minimal factual scenarios that expose distinct interpretive choices. "Logical Judges Challenge Human Judges on the Strange Case of B.C.-Valjean" (Mascardi et al., 2020) uses a criminal case to show another sense of legal puzzle: a fact pattern with ambiguous or indirect indicators, conflicting witness reliability, and a threshold question under Article 192 of the Italian Code of Criminal Procedure. Finally, "Lottery paradox, DNA evidence and other stories: How to accept uncertain statements" (Pawitan, 2021) treats legal puzzles as problems of accepting uncertain statements, especially where probabilistic evidence is strong for individual propositions but unstable under conjunction.

Research setting	What counts as a legal puzzle	Primary function
Legal Zero-Days	Modified legislation with subtle but consequential vulnerabilities	Capability evaluation
Appellate-question dataset	Hard case questions with competing arguments	Stability and accuracy testing
Legal RAG Bench	Retrieval-plus-reasoning criminal-law questions	End-to-end RAG evaluation
Logical judge case	Evidentiary identity and responsibility problem	Symbolic adjudication and pedagogy
Formalization comparison	Minimal disagreement scenarios from competing encodings	Audit of interpretive choices
Probability-focused analysis	Acceptance of uncertain statements in court	Analysis of evidential reasoning

This diversity suggests that "legal puzzle" is not a single benchmark format. A plausible implication is that the unifying property is exposure of latent structure: the puzzle is constructed so that doctrinal sufficiency depends on interactions that are not obvious from surface reading alone.

2. Evidentiary uncertainty and probabilistic structure

One major class of legal puzzles concerns evidentiary sufficiency. In the B.C.–Valjean case, the central question is whether the available items satisfy the Italian evidentiary threshold that proof be built on evidences that are "severe, precise, and coherent" under Article 192 CPP (Mascardi et al., 2020). The paper distinguishes evidence that is suggestive from evidence that is legally sufficient. A fingerprint on a stolen scooter is severe but not precise because plausible alternative explanations exist; a dialect utterance tied to birthplace is neither severe nor precise; the identification of a scooter and timing does not by itself establish identity. The puzzle is therefore not the presence of inculpatory facts, but whether those facts can be composed into a legally adequate inference.

The Prolog formalization makes that threshold explicit through the identity rule: $P(\neg H \mid E)$1 This rule operationalizes a burden-of-proof threshold requiring at least two evidences and at least one that is both severe and precise. Under the scenario with E1, E2, and E3 only, identity is not proven and acquittal follows; adding E4 without impeaching the witness yields a severe-and-precise temporal coupling and flips the outcome; undermining witness reliability restores acquittal; adding E5 yields proof of identity and responsibility (Mascardi et al., 2020).

Pawitan’s discussion of the lottery paradox and DNA evidence generalizes this evidentiary issue into a probabilistic one (Pawitan, 2021). Kyburg’s rule of acceptance—accept an uncertain proposition when its probability is high—fails because many individually high-probability propositions can be jointly inconsistent. The legal relevance is direct: verdicts depend on conjunctions of uncertain propositions such as identity, opportunity, motive, means, and absence of alibi. The paper states that uncertainty degrades when statements are combined, which helps explain why courts demand narrative coherence and multiple lines of evidence rather than isolated high-probability facts.

The DNA analysis introduces another recurring legal puzzle: how to relate a strong probabilistic indicator to the legally operative proposition. The likelihood ratio, $LR = P(E \mid H) / P(E \mid \neg H)$, measures the strength of the DNA match for the source-of-sample hypothesis, but guilt requires additional evidence (Pawitan, 2021). The prosecutor’s fallacy arises when $P(E \mid \neg H)$ is confused with $P(\neg H \mid E)$; the defense attorney’s fallacy arises when the existence of many potential matches in a large population is taken to show that DNA is worthless. The paper’s central lesson is that legal acceptance cannot be reduced to one number: probability statements attach to specific items of evidence, while adjudication requires integrating several uncertain propositions into a coherent verdict.

This suggests that many legal puzzles are conjunction problems disguised as single-issue questions. The difficulty is often not whether one evidentiary item is impressive, but whether the total inferential chain survives the legal standard of proof.

3. Symbolic reasoning, logical judges, and explicit thresholds

The symbolic AI literature treats legal puzzles as an opportunity to make inferential structure inspectable. In the B.C.–Valjean study, the authors implement a "logical judge" in SWI-Prolog via SWISH, encoding facts about events, witnesses, reliability, evidences with time windows, rules evaluating severity and precision, a rule concluding identity, and a rule concluding responsibility (Mascardi et al., 2020). The system can toggle evidences and witness reliability, print motivations, and emulate a judicial reasoning structure.

The formalization exposes precisely where outcomes turn. The paper stylizes identity from temporal coupling as

$$\forall X,Y,V,t_1,t_2 \ [SeenDriving(X,V,t_1) \wedge SeenDriving(Y,V,t_2) \wedge Reliable(w_1) \wedge Reliable(w_2) \wedge |t_2 - t_1| \leq \Delta] \rightarrow EvidenceSameAs(X,Y; severity=hi, precision=hi).$$

By contrast, the dialect utterance and fingerprint are encoded as ambiguous evidence with low precision. The comparison between the human judge and the logical judge is therefore not a contest over black-box prediction, but a comparison between explicit thresholding and human evidentiary evaluation.

The demonstration to more than 70 magistrates at the Italian School of Magistracy further frames legal puzzles as pedagogical devices (Mascardi et al., 2020). Attendees received the obfuscated case in advance, answered guided questions about evidence availability and reliability, and then observed live transitions between outcomes as the logical representation changed. The reported reactions are notable: 9 respondents felt logical judges could significantly support judges but never substitute them, while 8 foresaw limited future support. The authors characterize the Prolog judge as "just an exercise of knowledge representation and simple deductive reasoning," and the limitations are explicit: open-textured standards are discretized into "hi" and "lo," reliability lacks a probabilistic credibility calculus, and the system is monotonic and crisp where real legal reasoning is often nonmonotonic.

The significance of this line of work is not that symbolic systems resolve legal puzzles definitively. Rather, they externalize hidden premises. A logical judge can show exactly why identity fails under E1–E3, how a reliability downgrade defeats a conviction, and how burden-of-proof thresholds can be varied. That transparency is the core methodological value.

4. LLM legal puzzles: instability, retrieval, and end-to-end evaluation

Recent benchmark work treats legal puzzles as stress tests for model reliability. In the 500-question appellate dataset, stability is defined as the fraction of repeated runs yielding the modal answer. For a binary outcome,

$S = \frac{\max\{n_1,n_2\}}{N},$

with $N=20$, and a question is unstable for a model iff $S<1$ (Blair-Stanek et al., 28 Jan 2025). This is a narrow but consequential definition: the question text is identical, deterministic settings are used, and yet the winner can change across runs.

The reported instability rates are substantial. Claude-3.5 is unstable on 10.6% ± 2.7 of questions, GPT-4o on 43.0% ± 4.3, and Gemini-1.5 on 50.4% ± 4.4; only 24 questions make all three models unstable, compared with 11.5 expected under independence (Blair-Stanek et al., 28 Jan 2025). Accuracy against the actual court outcome is 53.88 ± 0.98 for GPT-4o, 52.89 ± 0.98 for Claude-3.5, and 46.44 ± 0.98 for Gemini-1.5. The paper attributes instability, even at temperature $0$, to nondeterminism in floating-point accumulation order, different hardware or servers, and parallelized API handling. The practical implication stated in the paper is that courts, mediators, and practitioners relying on a single LLM call risk "coin-flip" outcomes on contested issues.

Legal RAG Bench analyzes a different but related failure mode: not answer instability, but whether a system can retrieve the right rule and remain grounded in it (Butler et al., 2 Mar 2026). The benchmark defines correctness $c_{eli}$, groundedness $g_{eli}$, and retrieval accuracy $P(E \mid \neg H)$0, and then uses a hierarchical error taxonomy:

• hallucination when $P(E \mid \neg H)$1;
• retrieval error when $P(E \mid \neg H)$2;
• reasoning error when $P(E \mid \neg H)$3.

A central empirical result is that retrieval is the primary driver of legal RAG performance. Kanon 2 Embedder achieves correctness 94.0%, groundedness 96.0%, and retrieval accuracy 86.0%, while Text Embedding 3 Large yields 76.5%, 91.5%, and 52.0%, and Gemini Embedding 001 yields 74.0%, 87.0%, and 53.0% (Butler et al., 2 Mar 2026). Switching to Kanon 2 from Text Embedding 3 Large improves average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. The paper concludes that many errors attributed to hallucinations are in fact triggered by retrieval failures.

These two benchmark families define different puzzle dynamics. The appellate dataset emphasizes doctrinal ambiguity and repeated-call instability on the same hard legal question (Blair-Stanek et al., 28 Jan 2025). Legal RAG Bench emphasizes lexically dissimilar retrieval targets, doctrinal disambiguation, and verifiable grounding against an authoritative source (Butler et al., 2 Mar 2026). Together they show that legal puzzles can expose both decisional volatility and pipeline decomposition errors.

5. Legal Zero-Days and expert-crafted puzzle design

The legal-puzzle methodology in the Legal Zero-Days paper is distinctive because it treats legal puzzles as safety evaluations for a dangerous capability (Sadler et al., 12 Aug 2025). A Legal Zero-Day is defined by five characteristics: it is a novel discovery about the functioning of a law or the interaction between multiple laws; it has immediate effect without requiring subsequent litigation, lengthy legal processes, or discretionary action; it emerges externally rather than from within the legal system itself; it causes significant disruption; and it is time-consuming to rectify. The paper contrasts these vulnerabilities with routine legal ambiguities and with normal jurisprudential evolution.

The motivating case study is the 2017 Australian dual citizenship crisis. Section 44(i) of the Australian Constitution prohibits dual citizens from serving in Parliament, and the latent vulnerability emerged from the interaction between that provision and various international citizenship regimes (Sadler et al., 12 Aug 2025). Once certain officeholders were discovered to be dual citizens, constitutional ineligibility followed immediately. The Deputy Prime Minister resigned, the government’s parliamentary majority was threatened, numerous administrative decisions were potentially invalidated, and governmental operations were paralyzed for approximately 18 months.

The puzzle-construction methodology is correspondingly careful. Experts select complex legal frameworks within their domain of expertise, gather original legislation or legal frameworks, create a modified act with targeted vulnerabilities, and prepare a detailed explanation of how the change causes significant problems (Sadler et al., 12 Aug 2025). Documents are abridged to fit model context windows while retaining essential context. Models receive both the original and modified text and are instructed to identify strategic issues that would substantially impair legal operation while ignoring minor typographical or formatting errors. The paper states that expert lawyers should allocate 3–10 hours per puzzle, that no internet search is permitted during evaluation, and that simulated modified legislation is used rather than real live vulnerabilities.

The scoring setup uses an AI judge similar to recent LLM-evaluation work. Responses are compared to expert target answers, and judge validation is performed on a ground-truth dataset of 25 human-graded responses. The primary judge, o3-2025-04-16, achieved perfect performance, with $P(E \mid \neg H)$4 (Sadler et al., 12 Aug 2025). Empirically, current models perform poorly: the best system, gemini-2.5-pro-preview-05-06, reaches 10.00% ± 13.50%, followed by o3-2025-04-16 at 6.67% ± 9.70%, with other systems ranging from 1.85% to 5.19%. The authors interpret the uniformly low performance and narrow gap as evidence that Legal Zero-Day discovery is a genuine frontier capability not yet achieved by current systems.

The significance of these legal puzzles is twofold. First, they are designed to measure a capacity that could undermine safeguards, paralyze regulators, or enable regulatory arbitrage if it matured. Second, they create a safe analogue of red-teaming for law itself: the legal framework is stressed through controlled modifications rather than through exploitation of real institutions.

6. Formalization disagreements and edge-case generation

A separate research strand treats legal puzzles as the behavioral consequences of different formalizations of the same legal provision. In the formalization-comparison framework, a provision is represented as a rooted, labeled tree

$P(E \mid \neg H)$5

where each node carries a natural-language label and each non-leaf node has an operator

$P(E \mid \neg H)$6

(Vernie et al., 24 May 2026). Given multiple formalizations, corresponding nodes are grouped into equivalence classes, and for each pair a shared interface is derived from the deepest equivalence classes to which both formalizations contribute.

Coverage measures how much of each formalization is available for Boolean comparison:

$P(E \mid \neg H)$7

The paper filters to pairs with $P(E \mid \neg H)$8. Over the shared interface variables, each formalization induces a Boolean function, and disagreement is defined by

$P(E \mid \neg H)$9

Edge cases are extracted as prime implicants: partial assignments that force disagreement under every completion and are minimal with respect to that property (Vernie et al., 24 May 2026).

The SAT-based enumeration procedure uses Z3, maintains blocking clauses, finds uncovered disagreement minterms, shrinks them to implicants via an unsat core, makes them prime by greedy literal dropping, and iterates until the cover is complete (Vernie et al., 24 May 2026). Root causes are then defined as equivalence classes above the interface where the sub-formulas of the two formalizations are forced to differ under every completion of the prime implicant, but no child of that equivalence class has the same property.

The empirical findings are striking. Equivalence is very common at low coverage—97% of pairs below 0.2 and 89% below 0.4—but once coverage exceeds 0.4, the equivalence rate flattens near approximately 50% (Vernie et al., 24 May 2026). Coverage and edge-case count are uncorrelated, with $P(\neg H \mid E)$0, and edge-case counts range from 1 to 58,283. Per-model non-equivalence rates range from 32% for Qwen 3.5 to 71% for Gemini 3.1 Pro. The paper’s core claim is therefore that behavioral divergence is essentially uncorrelated with structural agreement.

The verbalized scenarios show what these disagreements look like doctrinally. For Article 5 AI Act, one scenario concerns biometric categorization by race for law enforcement and divides models over whether the practice is prohibited; the divergence turns on the structure of a carve-out clause and mirrors commentary disagreements (Vernie et al., 24 May 2026). For Article 3 UCTD, a liability-cap clause reveals disagreement over whether "not individually negotiated" is conjunctive or disjunctive and whether a burden-of-proof safeguard has been mis-encoded as a substantive requirement. These examples show legal puzzles functioning as audit artifacts: minimal, concrete fact patterns that expose where formalization choices alter legal outcomes.

This line of work implies that formal legal representation is itself a generator of puzzles. Once provisions are converted into decision functions, disagreement no longer appears only as doctrinal debate; it can be enumerated, localized, and verbalized into reusable cases for scholars, auditors, and drafters.

7. Methodological significance, limitations, and governance

Across these literatures, legal puzzles serve three principal research roles. They are evaluation instruments for dangerous capabilities in legal reasoning (Sadler et al., 12 Aug 2025); benchmarks for the stability, correctness, groundedness, and retrieval behavior of legal AI systems (Blair-Stanek et al., 28 Jan 2025, Butler et al., 2 Mar 2026); and audit devices for exposing interpretive choices in symbolic or formalized representations (Mascardi et al., 2020, Vernie et al., 24 May 2026). In all three roles, the puzzle format is valuable because it operationalizes otherwise diffuse legal difficulties into inspectable tasks.

Several limitations recur. The Legal Zero-Days paper states that puzzles cannot fully capture the scale, complexity, and interconnectedness of real legal systems, and that abridgment may make solving easier than real discovery (Sadler et al., 12 Aug 2025). The appellate-stability paper notes that APIs and models evolve, that proprietary deployment details prevent precise causal attribution of instability, and that the dataset is limited to hard federal appellate cases with dissents (Blair-Stanek et al., 28 Jan 2025). Legal RAG Bench is jurisdictionally narrow, focusing on Victorian criminal law, and its LLM-as-a-judge setup introduces residual evaluation uncertainty despite internal validation (Butler et al., 2 Mar 2026). The logical-judge study emphasizes that crisp symbolic thresholds do not capture the full texture of credibility assessment, defeasibility, or open-textured standards (Mascardi et al., 2020). The formalization-comparison pipeline depends on accurate node matching, receives only provision text rather than broader legal context, and can surface disagreements without revealing shared errors (Vernie et al., 24 May 2026).

Governance recommendations are correspondingly cautious. The Legal Zero-Days work recommends expanding AI safety evaluations to include Legal Zero-Day discovery capabilities, monitoring this capability longitudinally, using legal puzzles to stress-test legal systems safely, and integrating the evaluation with others such as cyber (Sadler et al., 12 Aug 2025). The instability paper recommends repeated trials, consensus rules, stability reporting, structured reasoning, cross-model comparison, and human review for unstable items (Blair-Stanek et al., 28 Jan 2025). Legal RAG Bench recommends prioritizing domain-adapted retrieval, preserving hierarchy in chunking, constraining generation to retrieved text, and monitoring correctness, groundedness, and retrieval accuracy separately (Butler et al., 2 Mar 2026). The logical-judge work points toward interpretable tools that support rather than replace adjudicators (Mascardi et al., 2020).

A plausible synthesis is that legal puzzles have become a general-purpose experimental format for legal AI and computational law. They permit controlled exposure of doctrinal ambiguity, evidentiary sufficiency, retrieval dependence, formalization drift, and systemic vulnerability. Their research value lies less in solving law automatically than in making failure modes explicit.