Cognitive Hierarchy Benchmark (CHBench)

Updated 8 July 2026

CHBench is a benchmark framework that evaluates hierarchical reasoning by inferring cognitive levels rather than using flat utility measures.
In large language models, it collects data from various games and employs Level-K and Poisson models to assess strategic reasoning and the effects of memory versus chat prompts.
For speech-based cognitive assessment, CHBench predicts task-, domain-, and global-level scores using advanced audio processing and self-supervised embeddings to detect mild cognitive impairment.

Searching arXiv for the cited CHBench-related papers and adjacent work. Cognitive Hierarchy Benchmark (CHBench) denotes a family of benchmark formulations grounded in cognitive hierarchy models, with two distinct but conceptually related usages in recent arXiv literature. In one usage, CHBench is explicitly introduced as a benchmark for evaluating the strategic reasoning capability of LLMs in normal-form games by inferring cognitive levels rather than relying on utility alone (Liu et al., 16 Aug 2025). In another, the term is not used in the original paper, but the underlying design is described as constituting the core elements of a hierarchy-aware benchmark for speech-based cognitive assessment that spans task-, domain-, and global-level targets in mild cognitive impairment (Kopar et al., 26 May 2026). Related work further contributes experimental benchmark designs for discriminating cognitive hierarchy and level- $k$ models from equilibrium behavior in auctions (Rasooly, 2021), and provides an epistemic-theoretic foundation for cognitive hierarchy and dynamic cognitive hierarchy via directed rationalizability (Liu, 2024).

1. Terminological scope and conceptual identity

CHBench is used explicitly in "CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs" as a benchmark and evaluation framework that measures strategic reasoning capability through cognitive hierarchy models from behavioral economics rather than utility performance alone (Liu et al., 16 Aug 2025). Its central object is inferred reasoning depth: agents are modeled as operating at different cognitive levels, with level-0 non-strategic behavior and higher levels defined through iterative best response.

A second strand uses the label more implicitly. "Beyond Binary: Speech Representations Across the Cognitive Score Hierarchy" does not introduce the term “Cognitive Hierarchy Benchmark (CHBench)” in the paper itself, but its design is described as providing the components, methodology, and initial results of a benchmark for hierarchy-aware cognitive score prediction from speech (Kopar et al., 26 May 2026). Here, “hierarchy” refers not to levels of strategic reasoning, but to the standardized score hierarchy of neuropsychological assessment: task, domain, and global cognition.

These two usages share a formal motif: both benchmark behavior or prediction across an explicitly structured hierarchy rather than through a single flat target. This suggests that “CHBench” functions as an umbrella designation for benchmark designs in which hierarchical latent structure is itself part of what is evaluated, though the concrete meaning of “hierarchy” differs across domains.

2. CHBench for strategic reasoning in LLMs

The explicit LLM benchmark version of CHBench evaluates strategic reasoning in two-player finite normal-form games using cognitive hierarchy models as the inferential backbone (Liu et al., 16 Aug 2025). The motivation is that utility-based evaluations are opponent-dependent and game-structure-dependent: the same agent can obtain high or low payoffs depending on the opponent’s policy and the topology of the payoff matrix, so utility is not treated as a robust proxy for reasoning depth.

The benchmark comprises three phases: data collection, level inference, and evaluation/generalization. Data are collected from six state-of-the-art LLMs—DeepSeek-chat-V3, Llama-3.1-70B-Instruct, Qwen-max, GPT-4o, GPT-4o-mini, and Gemini-1.5 Pro—across fifteen carefully selected normal-form games (Liu et al., 16 Aug 2025). The game suite contains four classic families—Prisoner’s Dilemma, Coordination Game, Hawk–Dove, and Snowdrift—and eleven randomly generated games. Games $\#1$ – $\#10$ are $2\times2$ , while games $\#11$ – $\#15$ are $3\times3$ ; some are symmetric and some asymmetric.

For each LLM–LLM pair and each game, repeated play with $T=30$ rounds is conducted, and for asymmetric games roles are swapped and the experiment repeated (Liu et al., 16 Aug 2025). Across all pairs and all games, the dataset totals 14,400 action choices. Prompting is organized into four mechanisms: Baseline, Chat, Memory, and Chat + Memory. In the Baseline condition, the model receives a one-shot decision prompt with the payoff matrix explicitly listed and must output a single action ID. In the Chat mechanism, each agent engages in two structured Q&A rounds with an advisor LLM of the same type before choosing. In the Memory mechanism, prompts include historical records of past rounds, with Partial Memory retaining the last 10 rounds and Full Memory retaining all history. Chat + Memory combines both procedures.

The benchmark fits two CH-style models to observed choices by maximum likelihood (Liu et al., 16 Aug 2025). The Level-K CH model uses myopic best response to level- $(k-1)$ with an imperfect-execution parameter $\epsilon_i^k$ and mixture weights $\#1$ 0, while the Poisson CH model uses a Poisson distribution over levels with parameter $\#1$ 1 and best response to the truncated mixture of lower levels. Estimation is performed by constrained nonlinear optimization via SLSQP with 10 random restarts, minimizing the negative log-likelihood

$\#1$ 2

A central empirical result is cross-opponent stability. For each LLM, the variance of inferred levels across opponents is consistently below 0.1 in both CH variants and all mechanisms (Liu et al., 16 Aug 2025). The benchmark also studies the maximum reasoning level $\#1$ 3. Likelihood improvement ratios show substantial gains from $\#1$ 4 but minimal gains from $\#1$ 5, and the KL divergence between $\#1$ 6 and $\#1$ 7 is near zero; accordingly, CHBench sets $\#1$ 8 throughout (Liu et al., 16 Aug 2025).

The mechanism ablations are among the most distinctive findings. Memory consistently improves inferred reasoning levels, whereas Chat generally degrades them (Liu et al., 16 Aug 2025). For example, under Poisson CH, GPT-4o-mini rises from 0.6061 in Baseline to 1.1510 with Memory, while dropping to 0.0513 with Chat; Qwen-max rises from 0.4362 to 1.2288 with Memory and drops to 0.1963 with Chat. DeepSeek-chat is noted as an exception in Poisson CH, where Chat raises the mean level from 0.7139 to 1.0506. Chat + Memory is typically intermediate between chat-only and memory-only. The paper interprets these results as indicating that memory aids planning and integration of opponent patterns, while chat may induce over-cooperation or suggestion-following (Liu et al., 16 Aug 2025).

3. Formal cognitive hierarchy models underlying CHBench

The explicit LLM benchmark adopts the cognitive hierarchy framework from behavioral economics (Liu et al., 16 Aug 2025). In a two-player normal-form game with action sets $\#1$ 9 and payoff function $\#10$ 0, mixed strategies $\#10$ 1 induce expected utility

$\#10$ 2

For a pure action $\#10$ 3, expected utility against an opponent mixture $\#10$ 4 is

$\#10$ 5

Level-0 is assumed to play uniformly at random:

$\#10$ 6

A standard CH assumption is that population reasoning levels follow a Poisson distribution

$\#10$ 7

and a level- $\#10$ 8 player believes opponents are distributed over lower levels $\#10$ 9 with truncated-renormalized weights

$2\times2$ 0

The believed opponent mixture is then

$2\times2$ 1

and the level- $2\times2$ 2 best-response set is

$2\times2$ 3

CHBench distinguishes this Poisson CH formulation from a Level-K variant in which a level- $2\times2$ 4 agent best responds only to level- $2\times2$ 5 rather than to the truncated mixture of all lower levels (Liu et al., 16 Aug 2025).

The theoretical paper "Level- $2\times2$ 6 Reasoning, Cognitive Hierarchy, and Rationalizability" reframes this picture epistemically (Liu, 2024). It interprets level- $2\times2$ 7 not as exogenous strategic sophistication but as an information type $2\times2$ 8, and defines a belief restriction $2\times2$ 9 under which a $\#11$ 0-type can only deem possible opponents of levels $\#11$ 1, treats level-0 as uniformly random, and assigns normalized prior weight over lower levels. In static games, $\#11$ 2-rationalizability is defined through an iterative elimination procedure on type-action pairs, with fixed point $\#11$ 3; in dynamic games, the analogous operator is defined on type-strategy pairs using conditional probability systems and sequential rationality (Liu, 2024).

The key theoretical result is generic coincidence. In static games, the CH solution generically coincides with $\#11$ 4-rationalizability; in dynamic games, dynamic cognitive hierarchy generically coincides with the behavioral consequences of rationality, common strong belief in rationality, and transparency of dynamic $\#11$ 5 (Liu, 2024). The source of non-equivalence is non-generic tie cases, because CH imposes equal probability among multiple best responses at lower levels, while $\#11$ 6-rationalizability allows more flexible beliefs among tied responses.

4. Benchmark design choices, metrics, and implementation in the LLM setting

The LLM version of CHBench is explicitly positioned against alternative evaluation paradigms such as simple utility, Nash-compliance, regret minimization, and QRE-based metrics (Liu et al., 16 Aug 2025). Its claim is not that these alternatives are useless, but that they do not isolate reasoning depth robustly under opponent heterogeneity and bounded rationality.

Several implementation details are standardized. Prompt templates explicitly enumerate actions and payoffs, and the output is restricted to a parseable “Your Choice” field (Liu et al., 16 Aug 2025). In the Chat mechanism, agents and advisors are assigned distinct names to reduce pronoun ambiguity. In asymmetric games, roles are swapped to neutralize role-specific confounds. Decoding settings are not disclosed.

The main benchmark outputs are the mean and variance of inferred cognitive levels per LLM under each mechanism (Liu et al., 16 Aug 2025). The following table summarizes the reported mean levels for Baseline, Memory, Chat, and Chat + Memory.

Model	Level-K CH means	Poisson CH means
DeepSeek-chat	0.6639 / 1.4607 / 0.9490 / 1.0765	0.7139 / 1.1446 / 1.0506 / 0.9832
Gemini-1.5 Pro	1.0530 / 1.0919 / 0.5713 / 0.6140	1.2152 / 1.3107 / 0.4603 / 0.4157
GPT-4o-mini	0.7381 / 1.0313 / 0.3016 / 0.7608	0.6061 / 1.1510 / 0.0513 / 0.6935
Llama-3.1-70B	1.1265 / 1.5417 / 0.5392 / 0.9580	0.7465 / 1.0651 / 0.6626 / 0.6772
Qwen-max	0.6761 / 1.1528 / 0.2707 / 0.5641	0.4362 / 1.2288 / 0.1963 / 0.6294
GPT-4o	1.1022 / 1.1754 / 0.6025 / 0.9706	1.2462 / 1.3192 / 0.6872 / 1.2352

In each row, the ordering is Baseline / Memory / Chat / Chat+Memory (Liu et al., 16 Aug 2025).

The paper also reports that Partial Memory improves performance, with examples including Level-K partial-memory means of 1.2060 for DeepSeek, 1.4977 for Llama, and 1.2588 for GPT-4o, and Poisson partial-memory means of 1.3619 for Gemini and 1.3633 for GPT-4o (Liu et al., 16 Aug 2025). Since code and dataset release are not specified, reproducibility depends on the appendix materials, the full set of payoff matrices, the prompt templates, and access to the relevant APIs.

5. CHBench as hierarchy-aware speech-based cognitive assessment

In the clinical speech-analysis literature, CHBench is not an official name in the original article, but the study’s structure is described as instantiating a benchmark for hierarchy-aware cognitive score prediction from speech (Kopar et al., 26 May 2026). The benchmark logic is to move beyond binary Alzheimer’s-versus-healthy classification and instead evaluate performance across the inherent three-level hierarchy used in neuropsychological testing: task-level continuous scores, domain-level composite scores, and global cognition modeled both continuously and binarily.

The dataset consists of 5,754 German neuropsychological assessment recordings collected within the TREND study (Kopar et al., 26 May 2026). After quality and eligibility filtering, the final analytical set included 959 sessions from 593 participants, with 698 healthy control sessions and 261 MCI sessions. Subject-disjoint development and hold-out splits were used and matched demographically and clinically, with no significant differences reported at $\#11$ 7 by chi-square or $\#11$ 8-tests (Kopar et al., 26 May 2026).

The six cognitive tasks are MMSE, Word List Recognition (RW), Boston Naming Test (BNT), Word List Recall (RL), Verbal Fluency (VF), and Phonemic Fluency (PF) (Kopar et al., 26 May 2026). Domains follow CERAD+ conventions: Language (LAN), Memory (MEM), Executive Function (EXE), and Visuospatial Ability (VIS). Global cognition is represented by the CERAD+ total score, modeled continuously and binarily with threshold 85, and by binary clinical MCI status defined as performance more than $\#11$ 9 standard deviations below demographically adjusted norms, i.e. “MCI” if $\#15$ 0 SD (Kopar et al., 26 May 2026).

Preprocessing is unusually detailed. Non-native speakers, incomplete profiles, and MCI-to-HC reverters were excluded. Acoustic integrity required minimum duration 15 s, RMS $\#15$ 1 dBFS, clipping $\#15$ 2, and SNR $\#15$ 3 dB, with contradictions manually reviewed (Kopar et al., 26 May 2026). A hyperparameter grid search of more than 2,500 combinations on a ground-truth subset of $\#15$ 4 selected a 6th-order Butterworth high-pass filter with cutoff $\#15$ 5 Hz, spectral-gating noise suppression with $\#15$ 6, and loudness normalization to $\#15$ 7 LUFS. Participant-disjoint validation yielded DER 0.20, JER 0.33, PUR 94%, and COV 97% (Kopar et al., 26 May 2026).

Two audio streams were created: a Prosody-Preserved stream with the examiner masked and temporal prosody retained, and a Concatenated stream in which participant-only segments were merged with 10 ms linear cross-fades (Kopar et al., 26 May 2026). Features comprised hand-crafted eGeMAPS variants—EG Prosody, EG V-Qual, and EG All—and frozen self-supervised representations from wav2vec 2.0 base and HuBERT large, with final hidden layer features, global mean pooling, and no fine-tuning on the clinical data (Kopar et al., 26 May 2026). Prediction models were Ridge regression, SVR, SVM, and XGBoost, embedded in a nested $\#15$ 8 cross-validation pipeline with z-score normalization, PCA variance thresholding, and subject-disjoint folds.

The principal results are organized by hierarchical level (Kopar et al., 26 May 2026). At Level 1, SSL embeddings outperform eGeMAPS across all tasks, with HuBERT strongest overall; open-ended tasks such as VF and PF attain higher Pearson $\#15$ 9 than constrained tasks such as MMSE, RW, and BNT. PF with HuBERT achieves approximately $3\times3$ 0 on development and 0.80 on hold-out (Kopar et al., 26 May 2026). At Level 2, HuBERT remains best overall, though predicting drawing-heavy EXE and VIS from speech is harder than predicting LAN and MEM; PF and VF are best for LAN, RL is strongest for MEM, and MMSE predicts EXE and LAN equally with $3\times3$ 1 (Kopar et al., 26 May 2026). At Level 3, RL + HuBERT yields approximately $3\times3$ 2 on development and 0.49 on hold-out for continuous CERAD+, and balanced accuracy approximately $3\times3$ 3 and 0.65 for binary CERAD+ thresholding at 85 (Kopar et al., 26 May 2026). For binary clinical MCI, however, the pattern reverses: the best model is MMSE + eGeMAPS All with balanced accuracy approximately $3\times3$ 4 on development and 0.63 on hold-out (Kopar et al., 26 May 2026).

The paper interprets these patterns through a “specialist” versus “generalist” distinction (Kopar et al., 26 May 2026). Open-ended tasks such as PF and VF are said to yield specialist representations: they are highly predictive at the task level, but their signal is diluted as targets aggregate toward global cognition. Constrained tasks such as MMSE and RW are said to yield generalist representations: despite ceiling effects at item level, aggregation improves their predictive value for broader domain and global targets. The authors also report clinically interpretable feature importance for the best Level 3 MCI SVM model, highlighting increased low-frequency spectral slope variability $3\times3$ 5 and elevated $3\times3$ 6 instability $3\times3$ 7 in MCI, while HC shows wider $3\times3$ 8 bandwidths (Kopar et al., 26 May 2026).

A third line of work is not itself titled CHBench, but explicitly discusses implications for a cognitive hierarchy benchmark in auction environments (Rasooly, 2021). "Going... going... wrong: a test of the level- $3\times3$ 9 (and cognitive hierarchy) models of bidding behaviour" was designed to test whether level- $T=30$ 0 and cognitive hierarchy models explain observed bids better than equilibrium in auctions. It identifies two environments that best disentangle these models from Bayes–Nash equilibrium: an all-pay auction with uniformly distributed values, and a first-price auction with the possibility of cancelled bids (Rasooly, 2021).

In that framework, values and bids are discrete and identical sets $T=30$ 1, ties are broken by no-win, and two standard L0 specifications are considered: truthful bidding $T=30$ 2 and random bidding uniform on $T=30$ 3. Under uniform values on $T=30$ 4, these coincide (Rasooly, 2021). Cognitive hierarchy uses a Poisson distribution

$T=30$ 5

with normalized beliefs over lower levels

$T=30$ 6

The paper estimates $T=30$ 7 from a modified 11–20 game and uses this for CH calibrations (Rasooly, 2021).

The diagnostic force of the chosen auction environments comes from sharp divergence between equilibrium and low-level CH or level- $T=30$ 8 predictions (Rasooly, 2021). In all-pay auctions with uniform values and L0 uniform bids, the level-1 objective is

$T=30$ 9

which yields the corner solution $(k-1)$ 0 for all $(k-1)$ 1. With low-on-ties tie-breaking, higher levels minimally overcut, producing step-like strategies such as $(k-1)$ 2 if $(k-1)$ 3, else 0, and $(k-1)$ 4 if $(k-1)$ 5, else 0 (Rasooly, 2021). By contrast, the equilibrium bid function is increasing and well approximated by

$(k-1)$ 6

In the first-price auction with cancellation, level-1 also bids zero when $(k-1)$ 7, while equilibrium bids remain positive and increasing (Rasooly, 2021). The experiment sets $(k-1)$ 8 and $(k-1)$ 9, close to the cancellation probability maximizing separation while remaining simple. Across the experiment, equilibrium clearly outperforms level- $\epsilon_i^k$ 0 and CH in quantitative fit. For example, pooling all data and assigning each datapoint the best-fitting level among $\epsilon_i^k$ 1, RMSE in T1 first-price is 15.1 for BNE versus 21.3 for level- $\epsilon_i^k$ 2, and in T1 all-pay is 13.3 versus 19.8 (Rasooly, 2021). CH with $\epsilon_i^k$ 3 performs similarly poorly. When many levels are fitted, likelihood can be improved only by assigning most mass to implausibly high $\epsilon_i^k$ 4, such as $\epsilon_i^k$ 5, which the paper treats as a reductio ad absurdum (Rasooly, 2021).

This work proposes benchmark components rather than a completed shared benchmark artifact (Rasooly, 2021). Suggested components include the two auction environments, at least two discretization treatments, RMSE and likelihood/BIC evaluation, plausibility constraints on inferred $\epsilon_i^k$ 6, cross-task consistency with the 11–20 game, and qualitative checks such as the near-absence of iterated reasoning in self-reports. In this sense it complements the LLM CHBench by identifying environments that are especially discriminative for CH-style reasoning models.

7. Limitations, interpretive cautions, and future directions

Across its uses, CHBench inherits limitations from the particular domain instantiation. In the LLM benchmark, level-0 is fixed as uniform random, Poisson parameterization is assumed for one variant, decoding settings are not reported, and the framework is restricted to repeated normal-form interactions rather than extensive-form or information-set-rich settings (Liu et al., 16 Aug 2025). The paper notes that mapping text outputs into action indices may introduce occasional parsing errors, and that the Level-K noise model and Poisson CH truncation may not fully capture LLM stochasticity.

In the speech-based hierarchy benchmark, limitations include a single German-speaking TREND cohort, lack of multimodal inputs such as drawing kinematics or transcripts, omission of socio-demographic and lifestyle covariates, and the fact that models are trained independently per target level without explicit hierarchical loss coupling (Kopar et al., 26 May 2026). Future directions proposed there include generalization across languages and cultural contexts, joint hierarchical modeling with explicit hierarchical losses or regularization, multimodal inputs, and layer-wise SSL analysis (Kopar et al., 26 May 2026).

The epistemic-theoretic account sharpens a further caution: coincidence between CH and $\epsilon_i^k$ 7-rationalizability is generic, not universal (Liu, 2024). Non-generic ties matter because CH’s uniform tie-breaking is stronger than the underlying epistemic restriction. This suggests that benchmark tasks should either screen out tie-sensitive cases or annotate them explicitly, since benchmark conclusions may otherwise depend on conventions that are not implied by the broader theory.

Taken together, the recent literature presents CHBench not as a single fixed benchmark, but as a benchmark design pattern centered on hierarchical evaluation and bounded-rationality structure. In strategic reasoning, it provides an opponent-robust alternative to utility-based evaluation by inferring cognitive levels directly (Liu et al., 16 Aug 2025). In clinical speech analysis, it motivates hierarchy-aware prediction across task, domain, and global cognition rather than flat binary diagnosis (Kopar et al., 26 May 2026). In auction experiments and epistemic game theory, it supplies diagnostic environments and formal foundations that clarify what a cognitive hierarchy benchmark can validly measure and under what assumptions (Rasooly, 2021, Liu, 2024).