Watson & Holmes Detective Game
- Watson & Holmes is a detective tabletop game featuring narrative evidence, location-based clue gathering, and exact-match resolution.
- It combines competitive gameplay with a naturalistic reasoning benchmark to evaluate incremental evidence integration and hypothesis revision.
- Researchers use the game to compare human and AI performance in sequential narrative inference and graded, open-ended responses.
Searching arXiv for the specified paper and any directly relevant supporting work on the Watson & Holmes benchmark. Watson & Holmes is a competitive whodunnit game for 3–7 players that stages Holmesian detection through incrementally acquired narrative evidence, location-based clue gathering, and exact-match solution checking. In research usage, it has also been adapted into a naturalistic benchmark for comparing human and LLM reasoning while preserving the original game’s narrative structure and open-ended inference and removing competitive strategy (Leelawat et al., 23 Feb 2026). The resulting dual significance of Watson & Holmes—both as a commercial detective tabletop game and as the substrate for a controlled reasoning benchmark—rests on its combination of story-fragment evidence, multi-step hypothesis revision, and concise, canonical case resolution.
1. Original game structure and objective
Watson & Holmes is organized around classic Holmesian detection. Players begin from a case introduction, such as a new client visiting 221B Baker Street, then visit locations around the case via location cards, collect narrative clues, and deduce answers to case questions (Leelawat et al., 23 Feb 2026). The objective is to be the first to visit 221B Baker Street, reveal and answer all case-specific questions, and match the model answers exactly.
The game is explicitly competitive. It supports 3–7 independent detectives competing in parallel. Access to locations is mediated by Carriage Tokens, which are used to bid for entry during visiting phases when a location is already taken that turn. This mechanic makes evidence acquisition not merely informational but also strategic, because clue access depends on contested movement through the case space rather than on a fixed reveal schedule (Leelawat et al., 23 Feb 2026).
Case materials are standardized around several narrative components. Each play includes a case introduction, location cards, 2–4 questions per case, model answers, and a narrated “Holmes explains to Watson” case solution. The reported mean lengths are approximately 600 words for the introduction, a mean of 15 locations per case, approximately 160 words per location, and approximately 600 words for the solution. Victory depends on exact alignment with the canonical answers when making a final accusation, rather than on partial-credit reasoning or probabilistic confidence (Leelawat et al., 23 Feb 2026).
A central design property is that evidence is strictly narrative. Players do not receive structured facts but must infer key elements from story fragments and subtle clues at locations. This makes the game substantively different from deduction systems based on explicit symbolic constraints. A plausible implication is that Watson & Holmes occupies an intermediate regime between literary mystery reconstruction and formal puzzle solving: the evidence space is segmented and finite, but inference remains linguistically mediated rather than rule-tabulated.
2. Mechanics of investigation and evidence presentation
Play alternates between clue acquisition and attempted resolution. During the visiting phase, players choose locations and may need to outbid others with Carriage Tokens to enter if a location is already taken that turn. During the investigation phase, each player privately reads the narrative on the reverse side of the chosen location card and notes clues; some cases feature special rules such as locking locations (Leelawat et al., 23 Feb 2026). The evidence model is therefore incremental, private, and path-dependent.
The final accusation system encodes the game’s strict epistemic threshold. When a player believes the case is solvable, that player visits 221B Baker Street to unlock the questions, writes answers, and compares them to the model answers. If all are correct, the player wins immediately; otherwise, only the number of correct answers is revealed, and play continues for others (Leelawat et al., 23 Feb 2026). This mechanism discourages premature commitment while preserving the possibility of strategic early resolution.
Typical case lengths are substantial for a tabletop deduction game. Although the physical products are not distributed digitally, the adapted benchmark subset is reported to fall in the 1,900–4,000 word range, consistent with the per-component means derived from introductions, locations, and solutions (Leelawat et al., 23 Feb 2026). This suggests that the game’s reasoning load depends not only on clue subtlety but also on long-form narrative integration over many evidence fragments.
A common misconception is to treat Watson & Holmes as a conventional board game whose primary complexity lies in turn order or resource management. The evidence indicates otherwise: strategic multiplayer mechanics are present, but the core challenge is inference from distributed narrative clues. The bidding layer modulates access to information; it does not replace the underlying abductive and deductive workload.
3. Adaptation into a naturalistic reasoning benchmark
The 2026 benchmark adaptation preserves the game’s core narrative-evidence structure and open-ended questions while removing strategic multiplayer mechanics (Leelawat et al., 23 Feb 2026). In this single-player protocol, there is no bidding, locking, or token strategy; the case questions are revealed up front with the introduction; the solver provides answers after the introduction and after each location visited; and play continues until all locations are visited.
Under this benchmark formulation, a case consists of an introduction plus the set of locations for that case, with a mean of 15 locations and standard deviation . Evidence segmentation follows the original design, in which each location is a narrative unit. For each case, the solver gives an initial answer set after the introduction, then repeatedly chooses a next location, reads it, and updates answers after each visit until all locations have been seen (Leelawat et al., 23 Feb 2026). This converts the tabletop game’s clue topology into a longitudinal reasoning trace.
The question format remains open-ended. Examples include “Who is the culprit?”, “What was the murder weapon?”, and “How did the perpetrator gain entry?” Answers are unconstrained natural language but must be concise. To standardize parsing and logging, models return answers in JSON keyed by question number, and no explanation is requested (Leelawat et al., 23 Feb 2026). The benchmark therefore evaluates answer content rather than stylistic elaboration.
Three prompt templates define the AI protocol: an Introduction prompt containing the case introduction and full question list and requesting initial answers in JSON; a Choose-location prompt that recaps the introduction and all previously visited locations, lists remaining locations, and requests the next location in JSON; and a Question-answering prompt that recaps the introduction and all visited locations so far and requests updated answers in JSON (Leelawat et al., 23 Feb 2026). A fresh prompt is used each time when choosing the next location so that models do not see their previous answers during location selection.
This structure operationalizes an incremental-evidence reasoning problem that is neither multiple-choice nor single-shot question answering. A plausible implication is that the benchmark isolates a form of sequential narrative inference in which hypothesis maintenance, evidence search, and answer revision are all first-class components.
4. Scoring, autograding, and validation
Per-answer grading uses a 4-point rubric: $0$ for nothing correct or misleading, $1$ for some correct but mostly incorrect or missing, $2$ for mostly correct with minor omissions or errors, and $3$ for fully correct (Leelawat et al., 23 Feb 2026). The benchmark defines several temporally resolved metrics. The instantaneous score is the $0$–$3$ grade at a given stage of the case; the progressive score is the average of instantaneous scores across all stages; the final score is the instantaneous score at the end of the case; and the overall per-question score is the average of progressive and final.
The formal aggregation is specified as follows for question with stages and stage grades :
$0$0
$0$1
$0$2
and the overall model score is
$0$3
where $0$4 is the number of questions (Leelawat et al., 23 Feb 2026).
Automated grading uses GPT-4.1 with five prompt modes. Modes 1–3 include combinations of case solution, marking scheme, and human-written examples; Modes 4–5 first generate a detailed, case-specific rubric as JSON and then grade with that rubric. Mode 5 omits the case solution when creating the rubric to reduce overfitting and leakage from the gold solution into grading. Rubrics specify definitions, examples, and edge-case guidance per score level, and explicitly emphasize that minor misspellings should not be penalized if the intended answer is clear (Leelawat et al., 23 Feb 2026).
Validation was conducted against five independent human markers on a test set of sample answers authored by the experimenters. Each question came with 2–3 example answers at each level $0$5–$0$6 to anchor the rubric. Agreement was measured by percentage agreement and root-mean-square difference between numeric grades, with
$0$7
where $0$8 is the human grade and $0$9 the autograder grade (Leelawat et al., 23 Feb 2026).
The best-performing mode was Mode 5. Human–human agreement was 88.6% with $1$0, while Mode 5–human agreement was 87.4% with $1$1. Since Mode 5’s agreement is close to human–human, all benchmark results use Mode 5 (Leelawat et al., 23 Feb 2026). This addresses, though does not eliminate, the common concern that open-ended reasoning benchmarks cannot be scaled without introducing grading unreliability.
5. Dataset composition, human study, and empirical findings
The benchmark uses 11 evaluation cases from the commercial Watson & Holmes game. Cases with significant special rules or visual clues were excluded, and the first case was used only for practice and omitted from analysis. Because the game is commercial and not distributed online or digitally, the authors argue that it is unlikely the cases appear in model pretraining corpora (Leelawat et al., 23 Feb 2026). This is a key leakage-resistant property of the dataset.
The cases have the following reported characteristics.
| Property | Reported value |
|---|---|
| Evaluation cases | 11 |
| Locations per case | mean 15, SD = 2 |
| Total narrative length | mean 2,900 words, SD = 600 |
| Narrative length range | ~1,900–4,000 words |
| Questions per case | 2–5, mean 3.4 |
| Total questions | 37 |
The cases span typical detective themes including means, motive, opportunity, misdirection, alibis, and hidden relationships, though raw texts are not reproduced because Watson & Holmes is proprietary (Leelawat et al., 23 Feb 2026). The tutorial case “A damsel in distress” is referenced in the paper, including an example location card “Bont’s Surgery,” but only the format rather than the full content is reproduced.
Human performance data were collected from $1$2 UCL Computer Science undergraduates, aged 18+ and proficient in English, under university ethics approval. In one-on-one online sessions, an experimenter revealed the introduction, questions, and incrementally selected location texts via a slide deck. Participants could revisit any prior evidence, take notes, and answer after the introduction and after each location visit. They were paid at London Living Wage for time, and the top scorer on each case received double payment for that case (Leelawat et al., 23 Feb 2026).
Median case-solving time was 59 minutes with interquartile range $1$3. Longer time correlated with higher scores, with Pearson $1$4, significant at 95% confidence (Leelawat et al., 23 Feb 2026). This suggests that the task rewards sustained deliberation rather than merely fast pattern completion.
Across nine months of 2025, AI performance rose from roughly the lower quartile of the human comparison group to approximately the top 5% of that group. About half of the improvement came from steady advances across successive model releases, and the remainder came from a step change associated with reasoning-oriented model architectures (Leelawat et al., 23 Feb 2026). Among specific results, the best completion-oriented model, GPT-4.1, achieved an overall score of 1.45, approximately at the average human level of 1.43, while the best reasoning-oriented models, o3-pro and GPT-5, scored 2.14 and 2.10 respectively, estimated near the top 5% of humans (Leelawat et al., 23 Feb 2026).
6. Comparative benchmark properties, systematic effects, and limitations
The benchmark is distinguished by naturalistic narrative with incremental evidence, open-ended responses in unconstrained language, and direct human calibration under matched conditions (Leelawat et al., 23 Feb 2026). Unlike multiple-choice datasets such as CommonsenseQA or ARC, solvers must integrate messy story-based clues over time. Partial credit captures answer quality beyond exact-match accuracy, and the cases’ commercial, nondigital provenance is presented as a protection against contamination from pretraining corpora.
Pairwise correlations reported in the paper indicate that Watson & Holmes as benchmark material correlates moderately with ARC-AGI-1/2 at approximately $1$5–$1$6, Humanity’s Last Exam at approximately $1$7, and Chatbot Arena at approximately $1$8, suggesting overlapping but distinct capability measurement relative to abstract visual reasoning and broad question answering or chat performance (Leelawat et al., 23 Feb 2026). This supports its role as a complementary naturalistic reasoning probe.
Systematic model–human differences were mostly absent, but two effects are specifically highlighted. First, there is a significant negative coefficient for word count when regressing model-minus-human overall score on puzzle features. For GPT-5 versus average human, the coefficient is $1$9 with 95% confidence interval $2$0; for GPT-4.1 versus average human, it is $2$1 with 95% confidence interval $2$2 (Leelawat et al., 23 Feb 2026). Relative to humans, model performance drops on longer cases even though total lengths remain far below nominal context windows. This suggests length harms LLM reasoning quality before hard context limits are reached.
Second, immediately after the introduction, GPT-5 shows an early-stage inductive advantage relative to average human on induction/deduction/abduction ratings. The reported coefficients are $2$3 with 90% confidence interval $2$4, $2$5 with $2$6, and $2$7 with $2$8 (Leelawat et al., 23 Feb 2026). The paper interprets this as reasoning-oriented models making bold, plausible guesses from faint, distributed hints at scant-evidence stages, whereas humans tend to be more conservative.
Several limitations are explicitly identified. Domain specificity constrains generalization because detective narratives emphasize multi-hop narrative integration and abductive inference. Models degrade on longer cases in the 1,900–4,000 word range. Grading ambiguity remains possible despite close autograder–human agreement, and Cohen’s $2$9 was not reported. IDA ratings are difficult to classify cleanly and were generated by an LLM. The human sample is small at $3$0. Finally, top reasoning-oriented models are already near top-5% human performance, so the benchmark may saturate for frontier systems by late 2026, though it remains useful for smaller or deployable models (Leelawat et al., 23 Feb 2026).
These limitations do not negate the game’s research value. Rather, they delimit what Watson & Holmes can validly establish: it is a strong probe of incremental natural-language detective reasoning, not a universal measure of intelligence or reasoning competence across domains.
7. Practical and scholarly significance
Watson & Holmes has value in at least three overlapping senses: as a commercial tabletop design centered on evidence sequencing, as a reusable source of naturalistic reasoning tasks, and as a methodological template for open-ended benchmark construction. In research, it can be used to track longitudinal changes in reasoning models, analyze divergences between models and humans such as length sensitivity and early-stage inductive behavior, and study the effect of prompting and inference-time reasoning on incremental evidence integration (Leelawat et al., 23 Feb 2026).
The evaluation pipeline is fully specified at the level of prompting templates, grading procedure, and aggregation formulas. Data ingestion transcribes the case introduction and all locations for internal use; questions and model answers are recorded; cases with special rules are filtered; responses are collected after the introduction and after each location visit; and grading is performed by the GPT-4.1 Mode 5 autograder using a generated rubric without access to the case solution (Leelawat et al., 23 Feb 2026). Because raw case texts are not released, full reproduction requires access to new, licensed cases, but the benchmark mechanics are designed to be reproducible in principle.
Prompt variants were also tested. Appendix C.1 compares a Simple Prompt, a Discussion Prompt encouraging rationale before answers, and a Revision Prompt using reflect-then-revise. No clear, statistically robust benefit was found for the more complex prompting, and differences below 0.17 are within uncertainty; the Simple Prompt was therefore used as default (Leelawat et al., 23 Feb 2026). This result is methodologically relevant because it suggests that benchmark difficulty is not trivially overcome by more verbose response scaffolding.
Educationally, the cases can be used as classroom exercises in logic, cognitive science, or AI, emphasizing hypothesis formation, updating, and evidence synthesis. They can also be used to train rubric-based grading systems for open-ended reasoning tasks. In game design and HCI, they offer a testbed for calibrating difficulty through early-stage versus late-stage performance curves and for exploring human–AI cooperative play in which models propose next locations or hypotheses (Leelawat et al., 23 Feb 2026).
Future improvements proposed in the paper include expanding the case pool and domains, increasing human sample size and diversity, adding optional rationale evaluation while preserving concise answer scoring, and calibrating difficulty and length to probe long-context reasoning without undermining grading reliability (Leelawat et al., 23 Feb 2026). A plausible implication is that Watson & Holmes may prove most durable not as a fixed benchmark alone, but as an extensible benchmark design pattern for narrative, incrementally revealed, open-ended reasoning evaluation.