--- title: Cognitive Hierarchy Benchmark (CHBench) url: https://www.emergentmind.com/topics/cognitive-hierarchy-benchmark-chbench type: topic --- # Cognitive Hierarchy Benchmark (CHBench) Searching arXiv for the cited CHBench-related papers and adjacent work. Cognitive Hierarchy Benchmark (CHBench) denotes a family of benchmark formulations grounded in cognitive hierarchy models, with two distinct but conceptually related usages in recent arXiv literature. In one usage, CHBench is explicitly introduced as a benchmark for evaluating the strategic reasoning capability of large language models in normal-form games by inferring cognitive levels rather than relying on utility alone [2508.11944]. In another, the term is not used in the original paper, but the underlying design is described as constituting the core elements of a hierarchy-aware benchmark for speech-based cognitive assessment that spans task-, domain-, and global-level targets in mild cognitive impairment [2605.27189]. Related work further contributes experimental benchmark designs for discriminating cognitive hierarchy and level-\(k\) models from equilibrium behavior in auctions [2111.05686], and provides an epistemic-theoretic foundation for cognitive hierarchy and dynamic cognitive hierarchy via directed rationalizability [2404.19623]. ## 1. Terminological scope and conceptual identity CHBench is used explicitly in "CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs" as a benchmark and evaluation framework that measures strategic reasoning capability through cognitive hierarchy models from behavioral economics rather than utility performance alone [2508.11944]. Its central object is inferred reasoning depth: agents are modeled as operating at different cognitive levels, with level-0 non-strategic behavior and higher levels defined through iterative best response. A second strand uses the label more implicitly. "Beyond Binary: Speech Representations Across the Cognitive Score Hierarchy" does not introduce the term “Cognitive Hierarchy Benchmark (CHBench)” in the paper itself, but its design is described as providing the components, methodology, and initial results of a benchmark for hierarchy-aware cognitive score prediction from speech [2605.27189]. Here, “hierarchy” refers not to levels of strategic reasoning, but to the standardized score hierarchy of neuropsychological assessment: task, domain, and global cognition. These two usages share a formal motif: both benchmark behavior or prediction across an explicitly structured hierarchy rather than through a single flat target. This suggests that “CHBench” functions as an umbrella designation for benchmark designs in which hierarchical latent structure is itself part of what is evaluated, though the concrete meaning of “hierarchy” differs across domains. ## 2. CHBench for strategic reasoning in large language models The explicit LLM benchmark version of CHBench evaluates strategic reasoning in two-player finite normal-form games using cognitive hierarchy models as the inferential backbone [2508.11944]. The motivation is that utility-based evaluations are opponent-dependent and game-structure-dependent: the same agent can obtain high or low payoffs depending on the opponent’s policy and the topology of the payoff matrix, so utility is not treated as a robust proxy for reasoning depth. The benchmark comprises three phases: data collection, level inference, and evaluation/generalization. Data are collected from six state-of-the-art LLMs—DeepSeek-chat-V3, Llama-3.1-70B-Instruct, Qwen-max, GPT-4o, GPT-4o-mini, and Gemini-1.5 Pro—across fifteen carefully selected normal-form games [2508.11944]. The game suite contains four classic families—Prisoner’s Dilemma, Coordination Game, Hawk–Dove, and Snowdrift—and eleven randomly generated games. Games \(\#1\)–\(\#10\) are \(2\times2\), while games \(\#11\)–\(\#15\) are \(3\times3\); some are symmetric and some asymmetric. For each LLM–LLM pair and each game, repeated play with \(T=30\) rounds is conducted, and for asymmetric games roles are swapped and the experiment repeated [2508.11944]. Across all pairs and all games, the dataset totals 14,400 action choices. Prompting is organized into four mechanisms: Baseline, Chat, Memory, and Chat + Memory. In the Baseline condition, the model receives a one-shot decision prompt with the payoff matrix explicitly listed and must output a single action ID. In the Chat mechanism, each agent engages in two structured Q&A rounds with an advisor LLM of the same type before choosing. In the Memory mechanism, prompts include historical records of past rounds, with Partial Memory retaining the last 10 rounds and Full Memory retaining all history. Chat + Memory combines both procedures. The benchmark fits two CH-style models to observed choices by maximum likelihood [2508.11944]. The Level-K CH model uses myopic best response to level-\((k-1)\) with an imperfect-execution parameter \(\epsilon_i^k\) and mixture weights \(\alpha_i^k\), while the Poisson CH model uses a Poisson distribution over levels with parameter \(\lambda_i\) and best response to the truncated mixture of lower levels. Estimation is performed by constrained nonlinear optimization via SLSQP with 10 random restarts, minimizing the negative log-likelihood $$ \mathcal{L} = \sum_{(a_1^t, a_2^t) \in D_G} [ -\log s_1(a_1^t) - \log s_2(a_2^t) ]. $$ A central empirical result is cross-opponent stability. For each LLM, the variance of inferred levels across opponents is consistently below 0.1 in both CH variants and all mechanisms [2508.11944]. The benchmark also studies the maximum reasoning level \(\hat{k}\). Likelihood improvement ratios show substantial gains from \(\hat{k}=3 \to 4\) but minimal gains from \(4 \to 5\), and the KL divergence between \(s_i^3\) and \(s_i^4\) is near zero; accordingly, CHBench sets \(\hat{k}=4\) throughout [2508.11944]. The mechanism ablations are among the most distinctive findings. Memory consistently improves inferred reasoning levels, whereas Chat generally degrades them [2508.11944]. For example, under Poisson CH, GPT-4o-mini rises from 0.6061 in Baseline to 1.1510 with Memory, while dropping to 0.0513 with Chat; Qwen-max rises from 0.4362 to 1.2288 with Memory and drops to 0.1963 with Chat. DeepSeek-chat is noted as an exception in Poisson CH, where Chat raises the mean level from 0.7139 to 1.0506. Chat + Memory is typically intermediate between chat-only and memory-only. The paper interprets these results as indicating that memory aids planning and integration of opponent patterns, while chat may induce over-cooperation or suggestion-following [2508.11944]. ## 3. Formal cognitive hierarchy models underlying CHBench The explicit LLM benchmark adopts the cognitive hierarchy framework from behavioral economics [2508.11944]. In a two-player normal-form game with action sets \(A_i\) and payoff function \(u_i(a_i,a_{-i})\), mixed strategies \(s_i \in \Delta(A_i)\) induce expected utility $$ U_i(s_i, s_{-i}) = \sum_{a_i\in A_i} \sum_{a_{-i}\in A_{-i}} s_i(a_i) s_{-i}(a_{-i}) u_i(a_i, a_{-i}). $$ For a pure action \(a_i\), expected utility against an opponent mixture \(s_{-i}\) is $$ E[U_i(a_i \mid s_{-i})] = \sum_{a_{-i}\in A_{-i}} s_{-i}(a_{-i}) u_i(a_i, a_{-i}). $$ Level-0 is assumed to play uniformly at random: $$ s_i^0(a_i) = 1 / |A_i|, \quad \text{for all } a_i \in A_i. $$ A standard CH assumption is that population reasoning levels follow a Poisson distribution $$ f(k) = e^{-\tau} \tau^k / k!, $$ and a level-\(k\) player believes opponents are distributed over lower levels \(j 0.05\) by chi-square or \(t\)-tests [2605.27189]. The six cognitive tasks are MMSE, Word List Recognition (RW), Boston Naming Test (BNT), Word List Recall (RL), Verbal Fluency (VF), and Phonemic Fluency (PF) [2605.27189]. Domains follow CERAD+ conventions: Language (LAN), Memory (MEM), Executive Function (EXE), and Visuospatial Ability (VIS). Global cognition is represented by the CERAD+ total score, modeled continuously and binarily with threshold 85, and by binary clinical MCI status defined as performance more than \(1.5\) standard deviations below demographically adjusted norms, i.e. “MCI” if \(z < -1.5\) SD [2605.27189]. Preprocessing is unusually detailed. Non-native speakers, incomplete profiles, and MCI-to-HC reverters were excluded. Acoustic integrity required minimum duration 15 s, RMS \(> -55\) dBFS, clipping \(< 1.5\%\), and SNR \(> 10\) dB, with contradictions manually reviewed [2605.27189]. A hyperparameter grid search of more than 2,500 combinations on a ground-truth subset of \(N=89\) selected a 6th-order Butterworth high-pass filter with cutoff \(f_c = 100\) Hz, spectral-gating noise suppression with \(\alpha = 0.3\), and loudness normalization to \(-23\) LUFS. Participant-disjoint validation yielded DER 0.20, JER 0.33, PUR 94%, and COV 97% [2605.27189]. Two audio streams were created: a Prosody-Preserved stream with the examiner masked and temporal prosody retained, and a Concatenated stream in which participant-only segments were merged with 10 ms linear cross-fades [2605.27189]. Features comprised hand-crafted eGeMAPS variants—EG Prosody, EG V-Qual, and EG All—and frozen self-supervised representations from wav2vec 2.0 base and HuBERT large, with final hidden layer features, global mean pooling, and no fine-tuning on the clinical data [2605.27189]. Prediction models were Ridge regression, SVR, SVM, and XGBoost, embedded in a nested \(5 \times 3\) cross-validation pipeline with z-score normalization, PCA variance thresholding, and subject-disjoint folds. The principal results are organized by hierarchical level [2605.27189]. At Level 1, SSL embeddings outperform eGeMAPS across all tasks, with HuBERT strongest overall; open-ended tasks such as VF and PF attain higher Pearson \(r\) than constrained tasks such as MMSE, RW, and BNT. PF with HuBERT achieves approximately \(r = 0.85 \pm 0.02\) on development and 0.80 on hold-out [2605.27189]. At Level 2, HuBERT remains best overall, though predicting drawing-heavy EXE and VIS from speech is harder than predicting LAN and MEM; PF and VF are best for LAN, RL is strongest for MEM, and MMSE predicts EXE and LAN equally with \(r = 0.38\) [2605.27189]. At Level 3, RL + HuBERT yields approximately \(r = 0.58 \pm 0.07\) on development and 0.49 on hold-out for continuous CERAD+, and balanced accuracy approximately \(0.70 \pm 0.01\) and 0.65 for binary CERAD+ thresholding at 85 [2605.27189]. For binary clinical MCI, however, the pattern reverses: the best model is MMSE + eGeMAPS All with balanced accuracy approximately \(0.62 \pm 0.07\) on development and 0.63 on hold-out [2605.27189]. The paper interprets these patterns through a “specialist” versus “generalist” distinction [2605.27189]. Open-ended tasks such as PF and VF are said to yield specialist representations: they are highly predictive at the task level, but their signal is diluted as targets aggregate toward global cognition. Constrained tasks such as MMSE and RW are said to yield generalist representations: despite ceiling effects at item level, aggregation improves their predictive value for broader domain and global targets. The authors also report clinically interpretable feature importance for the best Level 3 MCI SVM model, highlighting increased low-frequency spectral slope variability \((+0.22)\) and elevated \(F0\) instability \((+0.18)\) in MCI, while HC shows wider \(F1/F2\) bandwidths [2605.27189]. ## 6. Related benchmark constructions in auctions and epistemic game theory A third line of work is not itself titled CHBench, but explicitly discusses implications for a cognitive hierarchy benchmark in auction environments [2111.05686]. "Going... going... wrong: a test of the level-\(k\) (and cognitive hierarchy) models of bidding behaviour" was designed to test whether level-\(k\) and cognitive hierarchy models explain observed bids better than equilibrium in auctions. It identifies two environments that best disentangle these models from Bayes–Nash equilibrium: an all-pay auction with uniformly distributed values, and a first-price auction with the possibility of cancelled bids [2111.05686]. In that framework, values and bids are discrete and identical sets \(X=\{0,1,\ldots,x\}\), ties are broken by no-win, and two standard L0 specifications are considered: truthful bidding \(b=v\) and random bidding uniform on \(X\). Under uniform values on \(X\), these coincide [2111.05686]. Cognitive hierarchy uses a Poisson distribution $$ P(K = k) = e^{-\tau} \frac{\tau^k}{k!}, $$ with normalized beliefs over lower levels $$ w_k(j) = \frac{g(j)}{\sum_{h=0}^{k-1} g(h)}. $$ The paper estimates \(\tau \approx 1.9\) from a modified 11–20 game and uses this for CH calibrations [2111.05686]. The diagnostic force of the chosen auction environments comes from sharp divergence between equilibrium and low-level CH or level-\(k\) predictions [2111.05686]. In all-pay auctions with uniform values and L0 uniform bids, the level-1 objective is $$ \pi(v,b) = v\left(\frac{b}{x+1}\right)^{n-1} - b, $$ which yields the corner solution \(\beta^1(v)=0\) for all \(v\). With low-on-ties tie-breaking, higher levels minimally overcut, producing step-like strategies such as \(\beta^2(v)=1\) if \(v \ge 2\), else 0, and \(\beta^3(v)=2\) if \(v \ge 3\), else 0 [2111.05686]. By contrast, the equilibrium bid function is increasing and well approximated by $$ \beta(v) = \left( \frac{n-1}{n} \right)\frac{v^n}{x^{n-1}}. $$ In the first-price auction with cancellation, level-1 also bids zero when \(p \le 1/n\), while equilibrium bids remain positive and increasing [2111.05686]. The experiment sets \(n=2\) and \(p=1/2\), close to the cancellation probability maximizing separation while remaining simple. Across the experiment, equilibrium clearly outperforms level-\(k\) and CH in quantitative fit. For example, pooling all data and assigning each datapoint the best-fitting level among \(\{1,2,3\}\), RMSE in T1 first-price is 15.1 for BNE versus 21.3 for level-\(k\), and in T1 all-pay is 13.3 versus 19.8 [2111.05686]. CH with \(\tau \approx 1.9\) performs similarly poorly. When many levels are fitted, likelihood can be improved only by assigning most mass to implausibly high \(k\), such as \(k \approx 32\), which the paper treats as a reductio ad absurdum [2111.05686]. This work proposes benchmark components rather than a completed shared benchmark artifact [2111.05686]. Suggested components include the two auction environments, at least two discretization treatments, RMSE and likelihood/BIC evaluation, plausibility constraints on inferred \(k\), cross-task consistency with the 11–20 game, and qualitative checks such as the near-absence of iterated reasoning in self-reports. In this sense it complements the LLM CHBench by identifying environments that are especially discriminative for CH-style reasoning models. ## 7. Limitations, interpretive cautions, and future directions Across its uses, CHBench inherits limitations from the particular domain instantiation. In the LLM benchmark, level-0 is fixed as uniform random, Poisson parameterization is assumed for one variant, decoding settings are not reported, and the framework is restricted to repeated normal-form interactions rather than extensive-form or information-set-rich settings [2508.11944]. The paper notes that mapping text outputs into action indices may introduce occasional parsing errors, and that the Level-K noise model and Poisson CH truncation may not fully capture LLM stochasticity. In the speech-based hierarchy benchmark, limitations include a single German-speaking TREND cohort, lack of multimodal inputs such as drawing kinematics or transcripts, omission of socio-demographic and lifestyle covariates, and the fact that models are trained independently per target level without explicit hierarchical loss coupling [2605.27189]. Future directions proposed there include generalization across languages and cultural contexts, joint hierarchical modeling with explicit hierarchical losses or regularization, multimodal inputs, and layer-wise SSL analysis [2605.27189]. The epistemic-theoretic account sharpens a further caution: coincidence between CH and \(\Delta^\kappa\)-rationalizability is generic, not universal [2404.19623]. Non-generic ties matter because CH’s uniform tie-breaking is stronger than the underlying epistemic restriction. This suggests that benchmark tasks should either screen out tie-sensitive cases or annotate them explicitly, since benchmark conclusions may otherwise depend on conventions that are not implied by the broader theory. Taken together, the recent literature presents CHBench not as a single fixed benchmark, but as a benchmark design pattern centered on hierarchical evaluation and bounded-rationality structure. In strategic reasoning, it provides an opponent-robust alternative to utility-based evaluation by inferring cognitive levels directly [2508.11944]. In clinical speech analysis, it motivates hierarchy-aware prediction across task, domain, and global cognition rather than flat binary diagnosis [2605.27189]. In auction experiments and epistemic game theory, it supplies diagnostic environments and formal foundations that clarify what a cognitive hierarchy benchmark can validly measure and under what assumptions [2111.05686] [2404.19623]. Source: https://www.emergentmind.com/topics/cognitive-hierarchy-benchmark-chbench