---
title: Knights-and-Knaves Reasoning Benchmark
url: https://www.emergentmind.com/topics/knights-and-knaves-logical-reasoning-benchmark
type: topic
---

# Knights-and-Knaves Reasoning Benchmark

Knights-and-Knaves Logical Reasoning Benchmark

A Knights-and-Knaves Logical Reasoning Benchmark is a class of formal and natural-language datasets used to assess the reasoning ability of humans and machines when confronted with combinatorial puzzles involving agents who always tell the truth (knights) or always lie (knaves). Such puzzles require models to resolve the truth status of each agent based strictly on a set of logically entangled public statements, with a unique solution enforceable by propositional or first-order logic encoding. Knights-and-knaves benchmarks have been established as canonical tests for symbolic model-checking, natural language inference, and—recently—large language model (LLM) reasoning and their propensity to memorize training data versus achieving robust generalization.

## 1. Formal Foundation and Problem Specification

Knights-and-knaves puzzles are defined over $N$ inhabitants, each an element of $\{knight, knave\}$, yielding a $2^N$-sized search space. Each character $i$ issues a single statement $S_i$, composed of logical atoms (“Person $j$ is a knight/knave”) and connectives (and, or, not, $\rightarrow$, $\leftrightarrow$). The benchmark instance is the tuple $(N, \mathcal{S} = \{S_1,...,S_N\})$. The solution is an assignment $B = (B_1,...,B_N)$, where $B_i=1$ iff $i$ is a knight, such that the biconditional $B_i \leftrightarrow S_i$ holds for all $i$.

The benchmark requires generating instances such that the propositional formula
$$
(B_1 \leftrightarrow S_1) \land \dots \land (B_N \leftrightarrow S_N)
$$
is satisfied by exactly one assignment, enforcing uniqueness. Some variants extend the agent roster to include “Normals” (truth-indeterminate), requiring “exactly-one-role” constraints per agent [2309.13044].

## 2. Dataset Construction and Generative Methodologies

A canonical methodology, as formalized by Xie et al. [2410.23123], specifies instances dynamically via the triple $(N, D, W)$: $N$ (number of characters), $D$ (max statement tree depth), and $W$ (max fan-out for “and/or”). For each individual $i$:
- Sample $S_i$ recursively up to depth $D$, choosing randomly among logical connectives and leaf atoms.
- Prune trivial/self-referential statements.
- Validate by brute-force search that the system yields a unique solution; otherwise, reject.

Difficulty is controlled by increasing $N, D, W$. In recent experiments, $D=2$, $W=2$, with $N$ ranging 2–8. For $N$ ≥ 3, 1,000 training and 100 test puzzles are used; for $N=2$, 200 training and 100 test puzzles [2410.23123]. Other prominent datasets include:
- 382 hand-collected natural language puzzles [2110.00558]
- 2,400 synthetic propositional puzzles with complexity gradations [2406.12546]
- 450 puzzles (300 determinate, 150 ambiguous) for NLI, each paired with atomic entailment/contradiction questions [2112.05742]

TruthQuest [2406.12546] systemically varies complexity via statement types: simple (self-reference, accusation), implicative (material implication), and equivalence (biconditional), for $n=3..6$.

## 3. Modeling Approaches and Experimental Protocols

Early systems encode each inhabitant and role as separate propositional (or first-order) variables, strictly formulating the mutual-exclusivity and role/utterance dependencies as logic clauses. Automated methods include:
- Brute-force model-checking using Python DSLs and full enumeration for $N\leq$4 [2309.13044]
- FOL translation (using custom CFG/NER/coref pipelines) followed by model search in Mace4 [2110.00558]
- Proof-theoretic evaluation (Prover9) for entailment labeling [2112.05742]

Contemporary LLM evaluation proceeds under several regimes:
- Zero- or few-shot instruction prompting.
- Chain-of-thought (CoT) prompting.
- Supervised fine-tuning (Direct FT: supervision on assignment; CoT FT: supervision on stepwise trace+assignment) [2410.23123].
- Reinforcement learning with verifiable rewards (RLVR) and motivation-enhanced setups (MeRF), where explicit scoring rules are injected into the prompt [2506.18485].

Key experimental protocols entail train/test splits (e.g., 1,000/100 per $N$ in [2410.23123]), held-out generalization, and distribution shift via synthetic perturbations at both mathematical (altering statements) and linguistic (renaming, statement order, role inversion) levels.

## 4. Evaluation Metrics and Diagnostic Tools

Benchmarks use stringent accuracy: an instance is scored correct iff all $N$ roles are assigned correctly. For more granular NLI settings, each “Who is a knight?” query is labeled as entailment, contradiction, or unknown (“ambiguous”) using automated theorem provers [2112.05742].

Perturbation-based memorization quantification is central in recent work [2410.23123]:
- Consistency Ratio: $CR(f; \mathcal{D}) = \frac{\# \text{puzzles solved before \& after perturbation}}{\# \text{puzzles solved before perturbation}}$
- LiMem score: $LiMem(f; \mathcal{D}) = Acc(f; \mathcal{D}) \cdot [1 - CR(f; \mathcal{D})]$
A high LiMem under minimal perturbation indicates pattern memorization rather than reasoning.

Further, per-sample analysis predicts “memorized” vs. “reasoned” decisions using both puzzle-derived features (TF-IDF, token count) and model internal activations.

## 5. Empirical Findings and Error Taxonomies

Off-the-shelf LLMs, including Llama and GPT variants, rarely exceed 70% accuracy for $N=2$, rapidly decaying to near chance for larger $N$ (e.g., $\leq 10\%$ for $N=8$) [2410.23123, 2406.12546]. Fine-tuning enables near-perfect interpolation but high LiMem scores, indicating that generalization is only partial until explicitly probed [2410.23123]. Motivation-enhanced RL (MeRF) on K&K offers $\sim$10–15 percentage point gains over RLVR baselines, provided the model is sufficiently large [2506.18485].

Error analyses (using TruthQuest [2406.12546]) expose distinctive LLM failure patterns, including:
- Confusing truth vs. lies (TL)
- Assuming statements are true without checking (TS)
- Misinterpreting logical operators (LO)
- False reproduction, unjustified conclusions, and unfaithfulness to deduction

CoT generally aids smaller $N$, but sensitivity to problem depth and operator complexity persists across architectures.

## 6. Probing Internal Representations and Robustness

Statement verification probing reveals that fine-tuned LLMs can reliably encode support for atomic truth judgments internally, achieving >95% accuracy in layerwise probes (vs. <80% pre-finetuning) [2410.23123]. Logistic classifiers over hidden states yield AUCs of 0.63–0.79 for “memorized” vs. “reasoned” instances, demonstrating explicit encodability of reasoning mode in activations.

Systematic “wrong-answer” or corrupted CoT fine-tuning confirms partial constraint learning even in misaligned supervision, though convergence is slower for highly corrupted signals [2410.23123].

## 7. Theoretical and Practical Implications

A principal conclusion is the demonstration that high LLM performance on logic benchmarks may result from superficial memorization unless explicitly stress-tested with perturbation and out-of-distribution settings [2410.23123]. Perturbation-based diagnostic metrics (e.g., LiMem) provide practical, quantitative audits for reasoning versus rote learning.

Recommendations for benchmark and system design include:
- Dynamic on-the-fly puzzle generation with controllable complexity (parameterized by $N, D, W$)
- Integrated structural and linguistic perturbations during evaluation
- Incorporation of memorization-based metrics
- Exploration of more sophisticated in-context learning and neuro-symbolic architectures capable of explicit hypothetical (“suppositional”) reasoning [2406.12546]

Recent advances, such as MeRF’s in-context injection of the ground truth reward, highlight the potential for prompt-based reward alignment, but expose limitations—such as inability to generalize motivational scaffolding to small models and performance degradation under misaligned motivation [2506.18485]. Both symbolic and neural approaches benefit from interface with theorem-proving infrastructure for both data generation and verification [2110.00558, 2112.05742].

The Knights-and-Knaves Logical Reasoning Benchmark thus serves as both a crucible for studying the boundaries of neural deductive behavior and a driver for the development of metrics and methodology that precisely distinguish pattern-matching from genuine reasoning.

Source: https://www.emergentmind.com/topics/knights-and-knaves-logical-reasoning-benchmark