Big Bench Extra Hard (BBEH)

• Big Bench Extra Hard (BBEH) is a benchmark that hardens BBH by replacing each task with a more difficult analogue to test advanced reasoning in language models.
• It features 23 tasks with increased context length, distractor density, and multi-hop reasoning, covering skills like spatial, temporal, and inductive logic.
• Empirical results show significant performance drops compared to BBH, indicating room for improvement in both general-purpose and reasoning-specialized systems.

BIG-Bench Extra Hard (BBEH) is a 2025 benchmark for evaluating general reasoning in LLMs after saturation on BIG-Bench and BIG-Bench Hard (BBH). It contains 23 tasks, each replacing one BBH task with a novel task in a similar reasoning domain but with substantially increased difficulty. The benchmark was introduced to preserve BBH’s breadth while increasing context length, reasoning depth, distractor density, and resistance to shortcuts. On BBEH, the best general-purpose model reported in the benchmark paper reaches 9.8% harmonic mean accuracy and 23.9% micro average accuracy, while the best reasoning-specialized model reaches 44.8% harmonic mean accuracy and 54.2% micro average accuracy, indicating substantial remaining headroom for robust cross-domain reasoning (Kazemi et al., 26 Feb 2025).

1. Historical position and motivation

BBEH is best understood as a successor to BBH. BBH was introduced as a suite of 23 challenging BIG-Bench tasks selected from tasks on which prior LLM evaluations did not outperform the average human-rater; that work also showed that chain-of-thought prompting could substantially change measured difficulty, with PaLM surpassing the average human-rater on 10 of 23 tasks and Codex on 17 of 23 tasks under CoT prompting (Suzgun et al., 2022). BBEH begins from the claim that this earlier hard subset had itself become too easy for frontier systems.

The BBEH paper identifies several reasons why BBH was no longer sufficient. It states that state-of-the-art models now score in the 90%+ range on BBH, that 8/23 BBH tasks are binary, that another 5/23 have at most 5 options, that some tasks admit shortcuts, that BBH task inputs average about 700 characters macro-averaged across tasks, that many BBH problems require only a small number of reasoning steps, and that BBH does not sufficiently test capabilities such as long-context stitching, many-hop reasoning, strong-prior override, finding errors in reasoning traces, and induction from examples. BBEH was therefore designed to restore a benchmark that is difficult for frontier models, diverse across reasoning types, automatically scorable, lower in chance performance, more robust against shortcuts, and more reflective of real-world reasoning demands (Kazemi et al., 26 Feb 2025).

This design goal is explicitly benchmark-level rather than task-local. BBEH aims to preserve the broad capability coverage that made BIG-Bench and BBH useful while replacing each task with a harder analogue in the same reasoning domain. The result is not a subset of BIG-Bench, but a new benchmark inspired by BBH’s structure and intended to measure broad, robust, cross-domain reasoning under more difficult conditions (Kazemi et al., 26 Feb 2025).

2. Scope, composition, and targeted capabilities

BBEH contains 23 tasks and, for most tasks, 200 questions. The only exception is Disambiguation QA, which has 120 questions, yielding a total of 4520 examples via $22 \times 200 + 120 = 4520$. Each BBEH task replaces one BBH task, which preserves BBH’s diversity profile while changing the actual task instances and often the task format itself (Kazemi et al., 26 Feb 2025).

The benchmark targets both the broad reasoning skills already associated with BBH-like evaluation and an expanded set of harder capabilities. The paper first identifies the following skills as necessary for BBH-like reasoning: temporal understanding; spatial and geometric understanding; commonsense understanding; humour understanding; causal understanding; reasoning about world entities and events; deductive logical reasoning; reasoning through linguistic knowledge; counting and filtering; data structures and algorithms; and performing arithmetic operations. BBEH then adds explicit emphasis on many-hop reasoning; very long-range dependency; going against strong prior; learning on the fly; dealing with distractors; long-context reasoning; needle in a haystack; finding errors in reasoning traces; inductive reasoning; constraint satisfaction; compositional understanding; and knowledge-intense reasoning (Kazemi et al., 26 Feb 2025).

The benchmark paper quantifies the increase in difficulty in two benchmark-wide ways. It states that the macro average context length in BBEH is about 6× that of BBH, and that the macro average output length of Gemini 2.0 Flash responses, used as a proxy for required thinking, is about 7× larger than on BBH. This scaling of context and response length is central to the benchmark’s design: BBEH is intended not merely to be “harder” in the sense of lower accuracy, but harder because it forces longer-range state maintenance, denser compositional reasoning, and more brittle interaction between instructions, priors, and distractors (Kazemi et al., 26 Feb 2025).

3. Task architecture and construction principles

The benchmark’s high-level design principle is one-to-one replacement: for every BBH task, the authors created a new task in the same reasoning domain that probes similar or stronger abilities at substantially higher difficulty. Construction was semi-adversarial. The reference general-purpose model was Gemini 1.5 Flash, and the reference reasoning-specialized model was Gemini Thinking Experimental, later Gemini-2.0-Flash-Thinking-Exp-01-21. For each task, the authors iteratively increased difficulty and re-evaluated; if the reference models performed too well, the task was replaced or further hardened until both reference models scored below 70% accuracy on each task. The paper explicitly notes a caveat: because the benchmark was partly adversarially tuned against specific reference models, it may be biased toward their failure modes (Kazemi et al., 26 Feb 2025).

BBEH task	BBH task replaced	Salient design change
Boardgame QA	Logical Deduction	Depth 6–8 reasoning, defeasible rules, learned conflict resolution
Boolean Expressions	Boolean Expressions	Five candidates, only one true, operands can be propositions
Buggy Tables	Penguins in a Table	Reconstruct corrupted table, then answer statistical queries
Causal Understanding	Causal Judgement	Three-way labels plus necessary/sufficient cause reasoning
Disambiguation QA	Disambiguation QA	Longer, more complex pronoun disambiguation with more options
Dyck Language	Dyck Languages	Find first mistake in a reasoning trace
Geometric Shapes	Geometric Shapes	Multiple shapes, broken segments, intersections, distractor commands
Hyperbaton	Hyperbaton	Infer a new adjective order from 50–250 examples
Linguini	Salient Translation Errors	IOL-style linguistic puzzles across four categories
Movie Recommendation	Movie Recommendation	Choose the set whose movies are all likely to be liked
Multi-step Arithmetic	Multi-step Arithmetic	New recursive operators and long compositional expressions
New Yorker Cartoon Caption (NYCC)	Ruin Names	Pick funniest caption from 10 options
Object Counting	Object Counting	Long lists, distractors, sum/difference queries
Object Properties	Colored Objects	Temporal updates over object collections and properties
SARC Triples	Snarks	Three sarcasm judgments per item from Reddit post-reply pairs
Shuffled Objects	Shuffled Objects	No-op switches and long-range references to named actions
Spatial Reasoning	Navigate	More hops, intersecting paths, forward and backward reasoning
SportQA	Sports Understanding	Hardest Level 3 multi-hop compositional questions
Temporal Sequences	Temporal Sequences	Weekly meeting scheduling with zones, buffers, and constraints
Time Arithmetic	Date Understanding	Compositional date/time questions with derived variables
Web of Lies	Web of Lies	LiveBench-style cases plus harder cyclic cases
Word Sorting	Word Sorting	Error detection in traces or sorting under modified alphabets
Zebra Puzzles	Formal Fallacies	5x5 to 8x8 puzzles with distracting clues

Several task-specific construction details illustrate the benchmark’s philosophy. In Boolean Expressions, longer expressions alone were insufficient because the model could simply write Python; the authors therefore inserted textual/mathematical propositions in place of raw True/False literals. In Hyperbaton, each sample uses a unique induced adjective ordering, forcing induction and prior override rather than memorized English adjective order. In Temporal Sequences, the model must output both the longest possible meeting length and the number of such possibilities, and both must be correct for credit. In Web of Lies, the harder subset includes cyclic cases where some truth values remain unknown but useful conclusions still follow (Kazemi et al., 26 Feb 2025).

BBEH mixes newly authored tasks with adapted prior resources. The benchmark description identifies Disambiguation QA, Hyperbaton, Boolean Expressions, Buggy Tables, Object Properties, Temporal Sequences, many Web of Lies examples, some Spatial Reasoning constructs, modified Movie Recommendation, and modified Multi-step Arithmetic as new author-created or heavily modified components. It also lists adapted sources such as BoardgameQA, MoCa causal stories, Kiciman et al., BIG-Bench Mistake traces, GeomVerse, Linguini, New Yorker caption datasets, SARC, SpatialLLMEval, SportQA, Test of Time, LiveBench Web of Lies, and generated zebra puzzles following prior work. Curation was sometimes substantial: Disambiguation QA involved 10 annotators, a separate reviewer, and 25 examples revised after review; the necessary/sufficient-cause component of Causal Understanding was corrected by three causal reasoning experts, with outputs changed for six examples (Kazemi et al., 26 Feb 2025).

4. Evaluation protocol and scoring

The benchmark evaluates both general-purpose and reasoning-specialized systems. The general-purpose models listed in the benchmark paper are Llama 3.1 8B Instruct, Gemma2 27B IT, Gemini 2.0 Flash-Lite, Gemini 2.0 Flash, and GPT-4o (2024-11-20). The reasoning-specialized models are Distill R1 Qwen 32B, DeepSeek R1, and o3-mini (high). The paper also reports a score for Gemini Thinking Experimental / Gemini-2.0-Flash-Thinking-Exp-01-21, though it is not in the main task table (Kazemi et al., 26 Feb 2025).

All tasks append the same standardized suffix instructing the model to think step by step and to end with a machine-extractable answer prefixed by “The answer is:”. Extraction searches for one of four answer prefixes—"The answer is: ", "The answer is ", "The final answer is: ", or "The final answer is "—and then applies minimal cleaning, including stripping wrappers, lowercasing, and allowing small formatting equivalences such as a bare letter instead of (A) and comma-spacing normalization for multi-element labels. The reported evaluation stack is also benchmark-specific: AI Studio for Gemini 2.0 and Gemma2, OpenAI API for GPT-4o and o3-mini (high), Together AI API for DeepSeek R1, and local GPU loading for publicly available Llama and Distill R1 Qwen (Kazemi et al., 26 Feb 2025).

The headline metric is the adjusted harmonic mean of task accuracies. The rationale is that micro and macro averages can obscure severe weakness on individual tasks, whereas robust general reasoning should require competence across the entire suite. To avoid zero values in harmonic aggregation, the benchmark paper states that it adds 1 to all accuracy numbers. It also reports micro average accuracy for completeness. This dual reporting is important because the two views can diverge sharply: a model can obtain a relatively high micro average while being badly penalized by harmonic mean for failing on a subset of tasks (Kazemi et al., 26 Feb 2025).

5. Empirical profile, saturation evidence, and failure modes

The main empirical result is that BBEH remains difficult even for strong models. On harmonic mean accuracy, the benchmark paper reports: Random 2.4, Llama 3.1 8B Instruct 3.6, Gemma2 27B IT 4.0, Gemini 2.0 Flash-Lite 8.0, Gemini 2.0 Flash 9.8, GPT-4o 6.0, Distill R1 Qwen 32B 5.2, DeepSeek R1 6.8, and o3-mini (high) 44.8. On micro average accuracy, it reports: Random 8.4, Llama 3.1 8B Instruct 10.6, Gemma2 27B IT 14.8, Gemini 2.0 Flash-Lite 19.7, Gemini 2.0 Flash 23.9, GPT-4o 22.3, Distill R1 Qwen 32B 19.2, DeepSeek R1 34.9, and o3-mini (high) 54.2. The paper also notes that the reference Gemini thinking model reaches 20.2% harmonic mean (Kazemi et al., 26 Feb 2025).

The benchmark’s anti-saturation claim is supported by direct BBH-to-BBEH comparison. For Gemini 2.0 Flash, the overall BBH counterpart score is 85.2, while the corresponding BBEH score is 23.90. Selected task drops include Boolean Expressions 97.6 → 27.0, Hyperbaton 94.8 → 4.5, Multistep Arithmetic 99.6 → 9.5, Object Properties / Colored Objects 96.8 → 1.5, Object Counting 97.6 → 11.0, Shuffled Objects 100.0 → 9.0, Spatial Reasoning / Navigate 97.6 → 18.5, Temporal Sequences 98.8 → 0.5, Web of Lies 94.8 → 18.5, and Buggy Tables / Penguins in a table 98.6 → 3.5. The only noted exception is Disambiguation QA, which improves from 42.0 → 48.3, partly attributed to ambiguity in BBH zero-shot prompting (Kazemi et al., 26 Feb 2025).

Performance differences across model classes are highly nonuniform. The paper states that reasoning-specialized models gain the most on tasks involving counting, planning, arithmetic, and data structures and algorithms. Examples in the comparison between o3-mini (high) and GPT-4o include Object Counting 90.0 vs 6.5, Buggy Tables 59.5 vs 0.5, Temporal Sequences 68.5 vs 0.0, Word Sorting 77.5 vs 22.0, and Multi-step Arithmetic 73.0 vs 5.5. By contrast, gains are small or absent on “softer” reasoning tasks such as commonsense, humour, sarcasm, and causal understanding: NYCC 23.0 vs 16.0, SARC Triples 38.5 vs 24.0, Causal Understanding 54.0 vs 54.0, and SportQA 25.0 vs 26.5. Benchmark-wide ceilings remain low: o3-mini (high) exceeds 70% on only 4 of 23 tasks, DeepSeek R1 on only 3 of 23, and all other models on none (Kazemi et al., 26 Feb 2025).

The error analyses are unusually rich. In BoardgameQA, models overpredict unknown; the paper reports unknown prediction rates of 77.6% for Gemma2 27B IT, 67.4% for Gemini 2.0 Flash-Lite, 73.3% for Gemini 2.0 Flash, 82.4% for GPT-4o, 39.7% for DeepSeek R1, and 65.5% for o3-mini (high), despite only one-third of labels actually being unknown. In Dyck Language, when models flag an erroneous step, most errors come from missing the first error and flagging a later one; the share of such errors is 98.7% for o3-mini, 100% for Gemini 2.0 Flash, 94.9% for Gemini 2.0 Flash-Lite, and 96.8% for GPT-4o. In Geometric Shapes, intersections are especially damaging: for o3-mini, no intersections 72% versus intersections 33%. In Spatial Reasoning, backward reasoning is much harder than forward-only reasoning: for o3-mini, 58.8% on forward-only versus 19.2% on backward; for DeepSeek R1, 48.6% versus 3.8%. In Temporal Sequences, requiring both the longest meeting length and the number of ways raises difficulty substantially: for o3-mini, 68.5% on the full task versus 78% if only the longest length is asked; for Gemini Flash, 0.5% versus 5%. In Shuffled Objects, the paper notes that Gemini 2.0 Flash runs out of effective output tokens on 25% of problems (Kazemi et al., 26 Feb 2025).

The benchmark paper also ties system-level trends to structural properties of tasks. It reports that o3-mini vs GPT-4o gains generally increase as both context length and the output-length proxy for required thinking increase, whereas Gemini 2.0 Flash vs Flash-Lite gains increase with context length but not much with the output-length proxy. This suggests that reasoning-specialized models benefited from both greater context and greater reasoning depth, while scaling a general model primarily helped with longer input handling rather than the hardest multi-step inference (Kazemi et al., 26 Feb 2025).

6. Later uses and research significance

Subsequent work quickly adopted BBEH as a hard reasoning benchmark. In "Automatic Prompt Generation via Adaptive Selection of Prompting Techniques", BBEH is the sole benchmark used to validate an automatic prompt-generation framework. That paper uses all 23 tasks, explicitly treating BBEH as appropriate for “clear measurement of prompt effects at high difficulty” and “evaluation of versatility.” Its reported aggregate scores on BBEH are Original 23.9 / 9.7, Anthropic 24.7 / 10.5, Ours 28.0 / 12.5, and Ours (temperature-optimized) 28.5 / 13.3 for arithmetic mean / harmonic mean across tasks (Ikenoue et al., 20 Oct 2025).

In "Once Upon an Input: Reasoning via Per-Instance Program Synthesis", BBEH again appears in full: the paper evaluates all 23 BBEH tasks and argues that BBEH is appropriate because it mixes more and less algorithmic instances, making it suitable for testing both routing between CoT and synthesis and iterative program refinement. For Gemini-2.0-Flash, the appendix reports BBEH-only harmonic mean comparisons of PoT 0.095, PoT-retries 0.098, CodeAct 0.040, BoT 0.027, and PIPS 0.171; the main BBEH ablation table reports PIPS 20.8, PIPS (no switch) 18.3, PIPS0 12.9, and PIPS0 4.3, showing that switching, iterative refinement, and explicit symbols all matter on the benchmark (Stein et al., 26 Oct 2025).

In "Prompting Policies for Multi-step Reasoning and Tool-Use in Black-box LLMs with Iterative Distillation of Experience", BBEH is used more narrowly: the reported BBEH experiments focus on three tasks—Disambiguation QA, Dyck Languages, and Web of Lies—rather than the full suite. Even on that subset, the benchmark is used to test whether a learned prompting policy can induce more reliable structured reasoning in a frozen worker model. The reported results are 57.08% → 65.41% on Disambiguation QA, 63.33% → 91.25% on Dyck Languages, and 52.50% → 90.12% on Web of Lies for baseline versus prompter policy with buffer (Sayana et al., 14 May 2026).

In "Evolutionary Generation of Multi-Agent Systems", BBEH serves as the paper’s main hard reasoning benchmark for testing automatically generated multi-agent systems. That paper describes BBEH as a 23-task benchmark with 4,520 samples and evaluates performance using the benchmark’s standard accuracy metric. Its headline BBEH result is 58.7% for EvoMAS (LLM-Selection), compared with 52.8% for the best fixed-backbone EvoMAS, 48.2% for the best EvoAgent result, and 46.2% for the best predefined MAS result; it also reports 98.9% execution rate on BBEH. The appendix notes a benchmark-specific systems fact: BBEH does not provide external tools, so on BBEH mutations are limited to prompt editing, model selection, and communication topology (Hu et al., 6 Feb 2026).

These later uses collectively establish BBEH as a benchmark for adaptive prompting, per-instance program synthesis, prompt-policy learning, and multi-agent inference architecture. A plausible implication is that BBEH’s primary value lies in difficulty and heterogeneity rather than in serving as a compact predictive proxy for broader BIG-Bench behavior, because related work on BIG-Bench found that BIG-Bench Hard was not especially good for recovering full-benchmark performance (Ye et al., 2023). A second plausible implication is that BBEH embodies a different philosophy from difficulty-calibrated suites such as Easy2Hard-Bench: BBEH hardens tasks through one-to-one replacement and semi-adversarial construction, whereas Easy2Hard-Bench emphasizes continuous item-level difficulty labels estimated with IRT or Glicko-2 (Ding et al., 2024).

BBEH therefore occupies a distinctive position in the evaluation landscape. It is not a math-only or code-only stress test, nor a purely item-difficulty-calibrated continuum. It is a task-diverse, automatically scorable, anti-saturation benchmark that preserves BBH’s breadth while substantially increasing the demands of long-context reasoning, distractor handling, inductive rule learning, prior override, reasoning-trace critique, and cross-task robustness (Kazemi et al., 26 Feb 2025).