---
title: 'HiToM: Higher-Order ToM Benchmark'
url: https://www.emergentmind.com/topics/hitom
type: topic
---

# HiToM: Higher-Order ToM Benchmark

Searching arXiv for the benchmark paper and closely related ToM work.
HI-TOM, also written HiToM, is a multiple-choice benchmark for evaluating higher-order Theory of Mind (ToM) reasoning in large language models. It extends Sally-Anne-style false-belief tasks from first- and second-order belief attribution to recursively nested beliefs up to the fourth order, and was introduced to test whether contemporary LLMs can sustain belief tracking under long event chains, deceptive communication, and increasing recursion depth [2310.16755]. In the benchmark, models answer questions of the form “Where does \(A_{n+1}\) think \(A_n\) thinks … \(A_2\) thinks \(A_1\) thinks \(O\) is?”, making HI-TOM both a diagnostic dataset for nested-belief inference and a stress test for the limits of current prompting-based reasoning [2310.16755].

## 1. Higher-order Theory of Mind in HI-TOM

HI-TOM formalizes Theory of Mind of order \(n\) (\(n\)-ToM) as the ability to reason about nested beliefs \(n\) levels deep. Given one target object \(O\) and a sequence of agents \(A_1,\dots,A_{n+1}\), an \(n\)-ToM query asks where \(A_{n+1}\) thinks \(A_n\) thinks … \(A_1\) thinks the object is [2310.16755]. The benchmark therefore covers a spectrum from 0th-order factual reality questions to 4th-order recursively embedded false-belief questions.

Two metrics are central. Standard per-order accuracy is defined as

$$
\mathrm{accuracy}_n=\frac{1}{N_n}\sum_{i=1}^{N_n} C_n(i),
$$

where \(N_n\) is the number of \(n\)-ToM questions and \(C_n(i)\in\{0,1\}\) is the correctness indicator for question \(i\) at order \(n\) [2310.16755]. Because correct higher-order reasoning presupposes consistency with lower-order reasoning, HI-TOM also defines joint accuracy up to order \(n\),

$$
J_n=\frac{1}{N_n}\sum_{i=1}^{N_n}\Bigl[\prod_{k=0}^{n} C_k(i)\Bigr],
$$

with \(C_0(i)\) denoting correctness on the 0th-order reality question for the same story [2310.16755]. This joint metric is diagnostically stricter than marginal order-wise accuracy: a model that intermittently answers high-order questions correctly while failing lower-order premises is penalized.

A plausible implication is that HI-TOM is designed less as a surface QA benchmark than as a compositional consistency benchmark. The joint metric explicitly encodes the requirement that recursive belief attribution be grounded in a correct epistemic base.

## 2. Benchmark construction and scenario design

HI-TOM is built on false-belief stories with one target object and five agents per story. Each story contains 1–3 “chapters,” where a chapter corresponds to one round of an object-finding game, and each story yields exactly five questions spanning 0th through 4th order [2310.16755]. The story world is composed from four component types: rooms, objects, containers, and agents.

The public release contains thousands of stories, although the exact total is omitted in the paper; because each story contributes one question per order \(n\in\{0,1,2,3,4\}\), the number of questions at each order equals the number of stories [2310.16755]. This yields a balanced multi-order evaluation regime in which lower- and higher-order queries are structurally paired.

Story generation proceeds in several stages. Chapter templates, adapted from Nematzadeh et al. 2018, are filled automatically: agents enter rooms and observe object placements; agents may sequentially move an object among containers; agents exit; and optional public or private communications may then occur outside [2310.16755]. Deceptive statements such as “publicly claim …” or “privately tell …” are inserted in 50% of stories, and random distractor lines such as “Anne likes the red_box” appear at roughly 1–2 per story [2310.16755]. A question generator \(Q_n\) instantiates the nested-belief template with randomly chosen agents and target object, while an answer generator \(A_n\) tracks each agent’s belief state up to four levels and emits the correct container together with three distractors [2310.16755]. All stories, questions, and answers are manually reviewed for logical consistency.

The benchmark also controls simple positional artifacts. Correct containers appear roughly uniformly across the text: quarters 1–4 of story lines contain the correct answer about 28.7%, 27.2%, 18.8%, and 25.3% of the time, respectively [2310.16755]. Exactly half of all stories include at least one deceptive communication chapter, while the other half contain none [2310.16755]. This balancing is important because it reduces the extent to which models can exploit answer-position bias or dataset skew instead of epistemic reasoning.

## 3. Evaluation protocol and measurement regime

HI-TOM evaluates models under two prompting styles. In Vanilla Prompting (VP), the model selects one answer from the multiple-choice options without explanation. In Chain-of-Thought Prompting (CoTP), the model is asked to “think step-by-step” before answering [2310.16755]. These two settings were chosen to test whether explicit intermediate reasoning improves recursive belief tracking.

The benchmark reports standard per-order accuracy and joint accuracy up to order \(n\), using the formal definitions above [2310.16755]. Notably, the paper does not report confidence intervals or formal significance tests such as McNemar’s test [2310.16755]. This absence matters methodologically: HI-TOM offers strong diagnostic patterns, but exact between-model comparisons should be interpreted without inferential statistics.

A common misconception is that chain-of-thought prompting automatically yields deeper Theory of Mind reasoning. The evaluation protocol was explicitly designed to test that assumption, and the reported results show that CoT provides only marginal gains and can even hurt performance under deception [2310.16755].

## 4. Empirical performance and degradation with order

The benchmark’s central empirical result is monotonic degradation as belief nesting deepens. For GPT-4, joint accuracy \(J_n\) in the non-deceptive setting, averaged across story lengths, is reported as 100% at \(n=0\), approximately 95% at \(n=1\), approximately 65% at \(n=2\), approximately 40% at \(n=3\), and approximately 10% at \(n=4\) [2310.16755]. Similar monotonic declines are reported for GPT-3.5, Claude, and Guanaco [2310.16755].

The paper’s summary reports the following overall standard accuracies, aggregated over stories with and without deception [2310.16755]:

| Model | VP | CoT Prompting |
|---|---:|---:|
| GPT-\(4_{32\mathrm{k}}\) | 59.9% | 58.1% |
| GPT-3.5 | 59.9% | 58.1% |
| Claude-instant | 32.2% | 27.5% |
| Guanaco 65B | 45.7% | 31.0% |

These aggregate scores are less informative than the joint-accuracy curves, because higher-order ToM requires preservation of lower-order premises. The joint analysis shows that from 0th to 4th order the drop in GPT-4 joint accuracy is about 90 points [2310.16755]. Increasing story length through additional chapters further depresses performance, and deceptive communications impose an additional 10–30 point cost in joint accuracy [2310.16755].

The principal significance of these findings is diagnostic rather than leaderboard-oriented. HI-TOM isolates a failure regime in which models that remain competent at factual tracking and shallow false-belief reasoning degrade rapidly when nested epistemic state updates must be composed over time, particularly when deceptive utterances alter only some agents’ belief states.

## 5. Failure modes and interpretive significance

The HI-TOM study manually inspected 300 CoT explanations, including 150 from GPT-4 and 150 from GPT-3.5, and organized errors into five categories [2310.16755]. The first is **insufficient reasoning-depth**, where the model skips recursive steps and effectively substitutes a lower-order inference for the required higher-order one. The second is **temporal ignorance**, where entry and exit order is ignored or misordered, leading the model to attribute observations to agents who were absent. The third is **commonsense errors**, which include unjustified trust in deceptive statements. The fourth is **spurious causal inference**, where belief updates are inferred from events that should be epistemically irrelevant. The fifth is **hallucinations**, where models introduce ungrounded details not present in the story [2310.16755].

At 4th order, the summary reports that insufficient reasoning-depth and commonsense errors each appear in approximately 30–35% of GPT-4 explanations, while hallucinations and spurious inferences account for approximately 10–15% [2310.16755]. This suggests that failure is not reducible to a single mechanism such as hallucination. Instead, the error profile is distributed across recursion management, temporal bookkeeping, and inappropriate default assumptions about communication and trust.

The paper interprets these patterns as evidence of shallow “System 1” behavior, poor recursion, and vulnerability to deception in current LLMs [2310.16755]. It recommends hybrid neural-symbolic architectures, explicit belief-tracking modules with structured memory, curriculum training that gradually increases ToM depth, RLHF targeting deception handling, richer real-dialogue corpora for higher-order ToM, and more rigorous statistical evaluation in future comparisons [2310.16755]. These proposals are consistent with the benchmark’s diagnostic structure: the errors emerge where latent pattern matching is insufficient and explicit epistemic state management becomes necessary.

## 6. Subsequent use of HI-TOM in neuro-symbolic ToM research

HI-TOM has been used as an evaluation target for later Theory of Mind systems. EnigmaToM introduces a neuro-symbolic framework combining a Neural Knowledge Base of entity states with an Iterative Masking mechanism for perspective-taking, and reports results on ToMi, HiToM, and FANToM [2503.03340]. In the HiToM setting, the task is described as reasoning over a narrated event sequence \(\mathcal{E}=\{\epsilon_1,\dots,\epsilon_n\}\) with a \(k\)th-order false-belief query such as “Where does \(c_k\) think that \(c_{k-1}\) … thinks that \(c_1\) thinks the object is?” [2503.03340].

The motivation for EnigmaToM on HiToM is threefold: long event chains, with an average of 26.5 events in HiToM; deep recursive perspective-taking; and reporting bias, in which narration omits fine-grained entity states [2503.03340]. EnigmaToM addresses these with structured state retrieval and recursive graph masking. Formally, it builds an omniscient augmented graph \(G^{(0)}\) and then applies a sequence of masked graph updates \(G^{(t)}=G^{(t-1)}\otimes G_{c_t}\), where unobserved events are replaced by a null node \(\varnothing\), after which the \(k\)th-order query is reformulated as a first-order question on the surviving event set [2503.03340].

On Qwen2.5-7B, the reported overall HiToM accuracies are 0.378 for Vanilla, 0.441 for CoT, 0.402 for SimToM, 0.316 for TimeToM, 0.444 for DWM, 0.393 for PerceptToM, and 0.508 for EnigmaToM, corresponding to a \(+0.064\) improvement over the best baseline [2503.03340]. The same summary reports order-wise gains for EnigmaToM over Vanilla, including \(+0.130\) at 0th order and \(+0.036\) at 1st order, with estimated gains of about \(+0.090\), \(+0.160\), and \(+0.148\) at 2nd, 3rd, and 4th order, respectively [2503.03340]. An ablation with a Llama3.3-70B backbone reports 0.696 for the full system, 0.726 without Knowledge Injection, and 0.460 without Iterative Masking, indicating that Iterative Masking is the critical component for event-based ToM on HiToM [2503.03340].

This later usage clarifies one of HI-TOM’s research functions: it is not merely a standalone benchmark, but a diagnostic substrate for testing whether explicit scene-graph construction, belief-state masking, and structured entity-state injection can mitigate the recursive failures exposed by the original benchmark.

## 7. Terminological ambiguity and disambiguation

The string “HiToM” is not unique in the recent arXiv literature. “HoliTom: Holistic Token Merging for Fast Video Large Language Models” describes a training-free token-merging framework for video LLMs and is unrelated to higher-order Theory of Mind reasoning [2505.21334]. In that work, the method combines outer-LLM pruning through global redundancy-aware temporal segmentation and spatial-temporal merging with inner-LLM token similarity-based merging, reducing visual tokens by over 90% and lowering prefilling FLOPs on LLaVA-OneVision-7B to 6.9% of baseline while maintaining 99.1% of original performance [2505.21334].

The two uses of the name occupy different subfields. HI-TOM in the 2023 benchmark literature denotes a Higher Order Theory of Mind benchmark for nested-belief reasoning [2310.16755], whereas HoliTom in the 2025 video-LLM literature denotes an efficiency method for spatio-temporal token compression [2505.21334]. Because both forms may appear as “HiToM,” disambiguation is often necessary in bibliographic search and citation practice.

In the Theory of Mind context, however, HI-TOM refers specifically to the benchmark centered on recursive false-belief reasoning, joint accuracy across epistemic orders, and the analysis of how LLM performance deteriorates under deeper nesting, longer stories, and deceptive communication [2310.16755].

Source: https://www.emergentmind.com/topics/hitom