HOLMES: Higher-Order Logic Benchmark

• HOLMES is a benchmark for higher-order symbolic reasoning, requiring models to infer meta-rules, resolve conflicts, and compose decision procedures.
• It pairs natural language cases with formal HOL representations using Isabelle/HOL to produce verifiable ground-truth answers and detailed reasoning traces.
• The benchmark controls reasoning difficulty via explicit factors and metrics, exposing weaknesses in rule composition and scope-conditioned decision-making.

HOLMES, short for Higher-Order Logic Meets real-world Explainable Symbolic reasoning, is a benchmark for evaluating higher-order logical reasoning in LLMs. Introduced as the first real-world benchmark for higher-order symbolic reasoning in LLMs, it contains 1379 instances spanning law and finance, and pairs each natural-language problem with a higher-order logic (HOL) formalization, a solver-verified ground-truth answer, and a verifiable reasoning trace (Wu et al., 22 Jun 2026). Its central claim is that prevailing evaluation practice is too first-order-logic-centric: many benchmarks test object-level deduction under fixed predicates, whereas realistic rule systems require reasoning over rules, predicates, functions, constraints, and decision procedures themselves. In this setting, HOLMES presents higher-order symbolic reasoning as a key bottleneck for reliable and verifiable LLMs (Wu et al., 22 Jun 2026).

1. Conceptual scope and motivation

HOLMES is motivated by structural limitations in earlier reasoning benchmarks. Datasets such as RuleTaker, ProofWriter, FOLIO, ProverQA, LogicBench, Multi-LogiEval, LogiConBench, CLUTRR, LogicNLI, and LogicalDeduction are described as highly useful, but mostly grounded in first-order logic (FOL), where rules concern objects and fixed predicates and evaluation asks whether a proposition such as $P(a)$ follows from a fixed rule set (Wu et al., 22 Jun 2026). In that paradigm, predicates and relations are given in advance, and models rarely need to reason about the rule system itself.

HOLMES instead targets what its authors characterize as HOL-style reasoning. In this setting, the model must determine which rules apply, which conflicts must be resolved by meta-rules such as priority or exception, which decision procedure is appropriate for a particular scope, and how several rule chains should be composed. The distinction is explicit: FOL-style evaluation asks whether an object satisfies a fixed predicate, whereas HOL-style evaluation can ask whether one policy is more lenient than another, or whether one rule defeats another (Wu et al., 22 Jun 2026).

The choice of domains follows directly from this goal. The benchmark covers law and finance / reimbursement, both of which are rule-intensive and naturally exhibit higher-order structure. Legal cases involve rule priority, exceptions, and article-level conflicts. The finance domain, grounded in a real reimbursement policy from Fudan University, involves scope-conditioned policy selection, numerical constraints, and composition across several expenses or scenarios (Wu et al., 22 Jun 2026). This suggests that HOLMES is not primarily a logic puzzle dataset, but a benchmark organized around realistic rule systems whose complexity is carried by the rules themselves.

2. Formal foundations and representation

Each HOLMES instance is represented as

$x = (\mathcal{R}, \mathcal{F}, q),$

where $\mathcal{R}$ is a set of rules, $\mathcal{F}$ a set of case facts, and $q$ a target question. The natural-language instance is paired with a HOL formalization $\hat{x}$ in a HOL language, and the model is asked to produce

$y = (a, \tau),$

where $a$ is the final answer and $\tau = (s_1,\dots,s_T)$ is a reasoning trace. Ground truth includes both a formally verified answer $a^\star$ and a gold trace derived from Isabelle/HOL (Wu et al., 22 Jun 2026).

The benchmark is built directly on Isabelle/HOL, in the Church–Isabelle/HOL tradition. Predicates are encoded as typed Boolean functions, intermediate concepts as functions over predicates, and meta-relations such as priority and exception as relations over rules. In the legal domain, a rule bundle is defined as

$x = (\mathcal{R}, \mathcal{F}, q),$0

with $x = (\mathcal{R}, \mathcal{F}, q),$1 for atomic case predicates, $x = (\mathcal{R}, \mathcal{F}, q),$2 for intermediate concepts, $x = (\mathcal{R}, \mathcal{F}, q),$3 for articles of the form $x = (\mathcal{R}, \mathcal{F}, q),$4, $x = (\mathcal{R}, \mathcal{F}, q),$5 for a priority order between articles, and $x = (\mathcal{R}, \mathcal{F}, q),$6 for exceptions (Wu et al., 22 Jun 2026). The reasoning hierarchy is explicit: atomic facts, intermediate concepts, article triggering, conflict resolution, and final judgment.

The finance side encodes a reimbursement policy as a HOL rule base with 20 clauses, formalized as rules over personnel attributes $x = (\mathcal{R}, \mathcal{F}, q),$7 and case attributes $x = (\mathcal{R}, \mathcal{F}, q),$8. Queries are instantiated from HOL templates for approval, completeness, invoice compliance, loan eligibility, and amount computation (Wu et al., 22 Jun 2026). Isabelle’s proof kernel is used to derive both answers and traces, making correctness formally grounded rather than annotator-imposed.

3. Benchmark construction and controllable structure

HOLMES contains 1379 instances. Of these, 300 belong to law and are built from 285 rules; 1079 belong to finance and are built from 58 rules (Wu et al., 22 Jun 2026). The construction pipeline begins with natural-language documents, then proceeds through manual or LLM-assisted formalization in HOL, case instantiation, theorem-prover derivation in Isabelle/HOL, and natural-language rendering of the symbolic structures back into rules and scenarios.

A notable design choice is that HOLMES controls reasoning difficulty through explicit factors rather than only through aggregate dataset size. For law, each instance has a difficulty profile

$x = (\mathcal{R}, \mathcal{F}, q),$9

where $\mathcal{R}$0 is reasoning-chain length, $\mathcal{R}$1 the triggered intermediate concepts, $\mathcal{R}$2 the triggered articles, and $\mathcal{R}$3 the triggered exceptions or conflicts (Wu et al., 22 Jun 2026). For finance, the benchmark controls the number of personnel scopes in parallel and the number of composed scenarios, each ranging from 2 to 5.

This construction gives HOLMES a dual status. It is both a benchmark and a symbolic evaluation framework: each example comes with solver-generated traces, theorem-prover-backed answers, and controllable higher-order reasoning phenomena such as rule conflicts, scope-conditioned rules, and composition of decision procedures (Wu et al., 22 Jun 2026). A plausible implication is that the benchmark is designed less for leaderboard-only comparison than for failure analysis under specific reasoning regimes.

4. Tasks and evaluation methodology

Models receive the natural-language rules and case description and must answer a case question while also producing a reasoning trace. In law, traces are expected to cite rule numbers and reflect the progression from facts to intermediate concepts, articles, conflict resolution, and conclusion. In finance, traces take an audit-style form, explaining which rules applied and why (Wu et al., 22 Jun 2026).

Evaluation has two principal axes. The first is final-answer accuracy, comparing $\mathcal{R}$4 to $\mathcal{R}$5. The second is reasoning-trace quality, measured with ROUGE-L, BERTScore-F1, ROSCOE, key-rule recall and precision for conflict analysis, and the newly introduced SRMR F1. SRMR is defined by greedily matching model steps to gold steps under an embedding-based similarity threshold and then checking whether step results are correct; if correct is the number of matched pairs with both semantic alignment and correct result, then

$\mathcal{R}$6

$\mathcal{R}$7

For Boolean results, correctness requires exact match; for numeric results, the metric allows 1% relative tolerance (Wu et al., 22 Jun 2026).

The benchmark’s notion of explainability is therefore procedural rather than rhetorical. Traces are expected to cite rules, express intermediate conclusions, and mirror the structure of the formal derivation. Because gold traces come from Isabelle/HOL proofs, trace evaluation is tied to which rules were actually used and in what order, rather than to fluency alone (Wu et al., 22 Jun 2026).

5. Experimental findings and error patterns

HOLMES was evaluated on 11 LLMs, including open-source and proprietary systems such as DeepSeek-V3.2, DeepSeek-R1, Qwen3-30B-Instruct, Qwen3-30B-Thinking, Qwen3.6-Flash, GPT-5.4-Mini, and Gemini-3.1-Flash. The benchmark’s headline result is that the overall average accuracy across all models is 50.64%, and the best overall model is Qwen3.6-Flash with 59.54% (Wu et al., 22 Jun 2026).

Performance is uneven across domains. In law, accuracy ranges from about 63.33% to 86.33%, with an average of 75.33%. In finance, performance is materially lower: the average is 43.77%, and the best score is 52.09%, again by Qwen3.6-Flash (Wu et al., 22 Jun 2026). In finance trace evaluation, ROUGE-L is around 0.21–0.25, BERTScore-F1 around 0.82–0.85, and ROSCOE around 0.64–0.69, indicating that generated traces are semantically related to the gold traces but not closely aligned step by step.

The law analysis shows a specific failure mode: shortcut reasoning. As the number of conflict rules increases, final-answer accuracy is described as almost flat, but recall of conflict-resolution rules in traces drops sharply, while ROUGE-L and SRMR F1 decline (Wu et al., 22 Jun 2026). A case study in the appendix shows a model outputting the correct verdict of “no criminal liability” while citing only two rules and skipping the liability structure and conflict layer entirely. This suggests that high answer accuracy can conceal failure to reconstruct the intended higher-order reasoning process.

The finance analysis highlights breakdown under scope-conditioned and compositional reasoning. On atomic tasks, Qwen3-30B-Thinking outperforms Qwen3-30B-Instruct on average, with 0.488 vs 0.384, and especially on reimbursable amount computation, with 0.57 vs 0.15 (Wu et al., 22 Jun 2026). Both models perform worse on parallel multi-scope settings, and both perform poorly on composition across rules, with average accuracies of 0.200 and 0.225. Accuracy falls from about 0.52 at 2 scenarios to about 0.03 at 5 scenarios. The qualitative pattern is that models often paraphrase individual clauses correctly but fail to compose rules across entities, scopes, and items, with errors biased toward over-lenient interpretations (Wu et al., 22 Jun 2026).

6. Significance, extensions, and disambiguation

HOLMES is presented as a diagnostic benchmark for reliable and verifiable AI in high-stakes rule-governed domains. Its specific contribution is to show that current LLMs can sometimes guess correct answers, particularly on simpler legal tasks, while remaining weak at faithful higher-order symbolic reasoning when multiple rule variants interact, scope-conditioned chains proliferate, or compositions across rules are required (Wu et al., 22 Jun 2026). The benchmark therefore supports evaluation of stepwise logical supervision, RL over reasoning traces, hybrid neuro-symbolic systems, solver-integrated prompting, planning modules, and reasoning-faithfulness metrics.

The project code and dataset are publicly available at https://github.com/wuyucheng2002/HOLMES (Wu et al., 22 Jun 2026). The authors also describe potential extensions to medicine, scientific reasoning, program verification, and policy compliance, as well as broader higher-order patterns involving functions over functions, constraints over decision procedures, and more complex policy hierarchies (Wu et al., 22 Jun 2026).

The term HOLMES is also used by several unrelated research systems and experiments. These include an automated fact-checking framework (Ou et al., 6 May 2025), a direct neutrino-mass experiment based on $\mathcal{R}$8 calorimetry (Nucciotti et al., 2018, Giachero et al., 2016), an evidence-grounded LLM agent for cloud DDoS investigation (Chen et al., 21 Jan 2026), an anomalous email detector (Wu et al., 2021), a benchmark of linguistic competence in LLMs (Waldis et al., 2024), an adversarial-example detector (Wen, 2024), a semantic inspection method for CNN image classifiers (Dibitonto et al., 2024), a video anomaly detection framework (Zhang et al., 2024), and a hierarchical evidential learning framework for partially relevant video retrieval (Li et al., 7 May 2026). In current LLM reasoning research, however, HOLMES most specifically denotes the higher-order symbolic reasoning benchmark introduced in 2026 (Wu et al., 22 Jun 2026).