---
title: 'HOLMES: Higher-Order Logic Benchmark'
url: https://www.emergentmind.com/topics/holmes
type: topic
---

# HOLMES: Higher-Order Logic Benchmark

HOLMES, short for **Higher-Order Logic Meets real-world Explainable Symbolic reasoning**, is a benchmark for evaluating **higher-order logical reasoning** in large language models. Introduced as the first real-world benchmark for higher-order symbolic reasoning in LLMs, it contains **1379 instances** spanning **law** and **finance**, and pairs each natural-language problem with a **higher-order logic (HOL)** formalization, a solver-verified ground-truth answer, and a verifiable reasoning trace [2606.23238]. Its central claim is that prevailing evaluation practice is too **first-order-logic-centric**: many benchmarks test object-level deduction under fixed predicates, whereas realistic rule systems require reasoning over **rules, predicates, functions, constraints, and decision procedures themselves**. In this setting, HOLMES presents higher-order symbolic reasoning as a key bottleneck for reliable and verifiable LLMs [2606.23238].

## 1. Conceptual scope and motivation

HOLMES is motivated by structural limitations in earlier reasoning benchmarks. Datasets such as RuleTaker, ProofWriter, FOLIO, ProverQA, LogicBench, Multi-LogiEval, LogiConBench, CLUTRR, LogicNLI, and LogicalDeduction are described as highly useful, but mostly grounded in **first-order logic (FOL)**, where rules concern objects and fixed predicates and evaluation asks whether a proposition such as \(P(a)\) follows from a fixed rule set [2606.23238]. In that paradigm, predicates and relations are given in advance, and models rarely need to reason *about* the rule system itself.

HOLMES instead targets what its authors characterize as **HOL-style reasoning**. In this setting, the model must determine which rules apply, which conflicts must be resolved by meta-rules such as priority or exception, which decision procedure is appropriate for a particular scope, and how several rule chains should be composed. The distinction is explicit: FOL-style evaluation asks whether an object satisfies a fixed predicate, whereas HOL-style evaluation can ask whether one policy is more lenient than another, or whether one rule defeats another [2606.23238].

The choice of domains follows directly from this goal. The benchmark covers **law** and **finance / reimbursement**, both of which are rule-intensive and naturally exhibit higher-order structure. Legal cases involve rule priority, exceptions, and article-level conflicts. The finance domain, grounded in a real reimbursement policy from Fudan University, involves scope-conditioned policy selection, numerical constraints, and composition across several expenses or scenarios [2606.23238]. This suggests that HOLMES is not primarily a logic puzzle dataset, but a benchmark organized around realistic rule systems whose complexity is carried by the rules themselves.

## 2. Formal foundations and representation

Each HOLMES instance is represented as
\[
x = (\mathcal{R}, \mathcal{F}, q),
\]
where \(\mathcal{R}\) is a set of rules, \(\mathcal{F}\) a set of case facts, and \(q\) a target question. The natural-language instance is paired with a HOL formalization \(\hat{x}\) in a HOL language, and the model is asked to produce
\[
y = (a, \tau),
\]
where \(a\) is the final answer and \(\tau = (s_1,\dots,s_T)\) is a reasoning trace. Ground truth includes both a formally verified answer \(a^\star\) and a gold trace derived from Isabelle/HOL [2606.23238].

The benchmark is built directly on **Isabelle/HOL**, in the Church–Isabelle/HOL tradition. Predicates are encoded as typed Boolean functions, intermediate concepts as functions over predicates, and meta-relations such as priority and exception as relations over rules. In the legal domain, a rule bundle is defined as
\[
\mathcal{B} = (\mathcal{A}, \mathcal{C}, \mathcal{P}, \prec, \mathcal{E}),
\]
with \(\mathcal{A}\) for atomic case predicates, \(\mathcal{C}\) for intermediate concepts, \(\mathcal{P}\) for articles of the form \(\varphi \Rightarrow \psi\), \(\prec\) for a priority order between articles, and \(\mathcal{E}\) for exceptions [2606.23238]. The reasoning hierarchy is explicit: atomic facts, intermediate concepts, article triggering, conflict resolution, and final judgment.

The finance side encodes a reimbursement policy as a HOL rule base with **20 clauses**, formalized as rules over personnel attributes \(\mathbf{u}\) and case attributes \(\mathbf{c}\). Queries are instantiated from HOL templates for approval, completeness, invoice compliance, loan eligibility, and amount computation [2606.23238]. Isabelle’s proof kernel is used to derive both answers and traces, making correctness formally grounded rather than annotator-imposed.

## 3. Benchmark construction and controllable structure

HOLMES contains **1379 instances**. Of these, **300** belong to law and are built from **285 rules**; **1079** belong to finance and are built from **58 rules** [2606.23238]. The construction pipeline begins with natural-language documents, then proceeds through manual or LLM-assisted formalization in HOL, case instantiation, theorem-prover derivation in Isabelle/HOL, and natural-language rendering of the symbolic structures back into rules and scenarios.

A notable design choice is that HOLMES controls reasoning difficulty through explicit factors rather than only through aggregate dataset size. For law, each instance has a difficulty profile
\[
d(x) = (\ell,\;|\mathcal{C}^\star|,\;|\mathcal{P}^\star|,\;|\mathcal{E}^\star|),
\]
where \(\ell\) is reasoning-chain length, \(\mathcal{C}^\star\) the triggered intermediate concepts, \(\mathcal{P}^\star\) the triggered articles, and \(\mathcal{E}^\star\) the triggered exceptions or conflicts [2606.23238]. For finance, the benchmark controls the number of personnel scopes in parallel and the number of composed scenarios, each ranging from **2 to 5**.

This construction gives HOLMES a dual status. It is both a benchmark and a symbolic evaluation framework: each example comes with solver-generated traces, theorem-prover-backed answers, and controllable higher-order reasoning phenomena such as rule conflicts, scope-conditioned rules, and composition of decision procedures [2606.23238]. A plausible implication is that the benchmark is designed less for leaderboard-only comparison than for failure analysis under specific reasoning regimes.

## 4. Tasks and evaluation methodology

Models receive the **natural-language** rules and case description and must answer a case question while also producing a reasoning trace. In law, traces are expected to cite rule numbers and reflect the progression from facts to intermediate concepts, articles, conflict resolution, and conclusion. In finance, traces take an audit-style form, explaining which rules applied and why [2606.23238].

Evaluation has two principal axes. The first is **final-answer accuracy**, comparing \(a\) to \(a^\star\). The second is **reasoning-trace quality**, measured with **ROUGE-L**, **BERTScore-F1**, **ROSCOE**, key-rule recall and precision for conflict analysis, and the newly introduced **SRMR F1**. SRMR is defined by greedily matching model steps to gold steps under an embedding-based similarity threshold and then checking whether step results are correct; if `correct` is the number of matched pairs with both semantic alignment and correct result, then
\[
P = \frac{correct}{|\mathcal{M}|}, \qquad R = \frac{correct}{|\mathcal{G}|},
\]
\[
\text{SRMR-}F_1 = \frac{2PR}{P + R}.
\]
For Boolean results, correctness requires exact match; for numeric results, the metric allows **1% relative tolerance** [2606.23238].

The benchmark’s notion of explainability is therefore procedural rather than rhetorical. Traces are expected to cite rules, express intermediate conclusions, and mirror the structure of the formal derivation. Because gold traces come from Isabelle/HOL proofs, trace evaluation is tied to which rules were actually used and in what order, rather than to fluency alone [2606.23238].

## 5. Experimental findings and error patterns

HOLMES was evaluated on **11 LLMs**, including open-source and proprietary systems such as DeepSeek-V3.2, DeepSeek-R1, Qwen3-30B-Instruct, Qwen3-30B-Thinking, Qwen3.6-Flash, GPT-5.4-Mini, and Gemini-3.1-Flash. The benchmark’s headline result is that the **overall average accuracy across all models is 50.64%**, and the **best overall model is Qwen3.6-Flash with 59.54%** [2606.23238].

Performance is uneven across domains. In **law**, accuracy ranges from about **63.33%** to **86.33%**, with an average of **75.33%**. In **finance**, performance is materially lower: the average is **43.77%**, and the best score is **52.09%**, again by Qwen3.6-Flash [2606.23238]. In finance trace evaluation, ROUGE-L is around **0.21–0.25**, BERTScore-F1 around **0.82–0.85**, and ROSCOE around **0.64–0.69**, indicating that generated traces are semantically related to the gold traces but not closely aligned step by step.

The law analysis shows a specific failure mode: **shortcut reasoning**. As the number of conflict rules increases, final-answer accuracy is described as almost flat, but recall of conflict-resolution rules in traces drops sharply, while ROUGE-L and SRMR F1 decline [2606.23238]. A case study in the appendix shows a model outputting the correct verdict of “no criminal liability” while citing only two rules and skipping the liability structure and conflict layer entirely. This suggests that high answer accuracy can conceal failure to reconstruct the intended higher-order reasoning process.

The finance analysis highlights breakdown under **scope-conditioned** and **compositional** reasoning. On atomic tasks, Qwen3-30B-Thinking outperforms Qwen3-30B-Instruct on average, with **0.488 vs 0.384**, and especially on reimbursable amount computation, with **0.57 vs 0.15** [2606.23238]. Both models perform worse on parallel multi-scope settings, and both perform poorly on composition across rules, with average accuracies of **0.200** and **0.225**. Accuracy falls from about **0.52** at **2 scenarios** to about **0.03** at **5 scenarios**. The qualitative pattern is that models often paraphrase individual clauses correctly but fail to compose rules across entities, scopes, and items, with errors biased toward over-lenient interpretations [2606.23238].

## 6. Significance, extensions, and disambiguation

HOLMES is presented as a diagnostic benchmark for **reliable and verifiable** AI in high-stakes rule-governed domains. Its specific contribution is to show that current LLMs can sometimes guess correct answers, particularly on simpler legal tasks, while remaining weak at faithful higher-order symbolic reasoning when multiple rule variants interact, scope-conditioned chains proliferate, or compositions across rules are required [2606.23238]. The benchmark therefore supports evaluation of stepwise logical supervision, RL over reasoning traces, hybrid neuro-symbolic systems, solver-integrated prompting, planning modules, and reasoning-faithfulness metrics.

The project code and dataset are publicly available at **https://github.com/wuyucheng2002/HOLMES** [2606.23238]. The authors also describe potential extensions to **medicine**, **scientific reasoning**, **program verification**, and **policy compliance**, as well as broader higher-order patterns involving functions over functions, constraints over decision procedures, and more complex policy hierarchies [2606.23238].

The term **HOLMES** is also used by several unrelated research systems and experiments. These include an automated fact-checking framework [2505.03135], a direct neutrino-mass experiment based on \({}^{163}\mathrm{Ho}\) calorimetry [1807.09269; 1612.03947], an evidence-grounded LLM agent for cloud DDoS investigation [2601.14601], an anomalous email detector [2104.08044], a benchmark of linguistic competence in language models [2404.18923], an adversarial-example detector [2405.19956], a semantic inspection method for CNN image classifiers [2403.08536], a video anomaly detection framework [2406.12235], and a hierarchical evidential learning framework for partially relevant video retrieval [2605.06083]. In current LLM reasoning research, however, **HOLMES** most specifically denotes the higher-order symbolic reasoning benchmark introduced in 2026 [2606.23238].

Source: https://www.emergentmind.com/topics/holmes