---
title: Big Bench Extra Hard (BBEH)
url: https://www.emergentmind.com/topics/big-bench-extra-hard-bbeh-d2db077b-9839-4d02-a9b4-e3a52c1c082c
type: topic
---

# Big Bench Extra Hard (BBEH)

BIG-Bench Extra Hard (BBEH) is a 2025 benchmark for evaluating general reasoning in large language models after saturation on BIG-Bench and BIG-Bench Hard (BBH). It contains 23 tasks, each replacing one BBH task with a novel task in a similar reasoning domain but with substantially increased difficulty. The benchmark was introduced to preserve BBH’s breadth while increasing context length, reasoning depth, distractor density, and resistance to shortcuts. On BBEH, the best general-purpose model reported in the benchmark paper reaches **9.8% harmonic mean accuracy** and **23.9% micro average accuracy**, while the best reasoning-specialized model reaches **44.8% harmonic mean accuracy** and **54.2% micro average accuracy**, indicating substantial remaining headroom for robust cross-domain reasoning [2502.19187].

## 1. Historical position and motivation

BBEH is best understood as a successor to BBH. BBH was introduced as a suite of 23 challenging BIG-Bench tasks selected from tasks on which prior language model evaluations did not outperform the average human-rater; that work also showed that chain-of-thought prompting could substantially change measured difficulty, with PaLM surpassing the average human-rater on 10 of 23 tasks and Codex on 17 of 23 tasks under CoT prompting [2210.09261]. BBEH begins from the claim that this earlier hard subset had itself become too easy for frontier systems.

The BBEH paper identifies several reasons why BBH was no longer sufficient. It states that **state-of-the-art models now score in the 90%+ range on BBH**, that **8/23 BBH tasks are binary**, that **another 5/23 have at most 5 options**, that some tasks admit shortcuts, that BBH task inputs average about **700 characters** macro-averaged across tasks, that many BBH problems require only a small number of reasoning steps, and that BBH does not sufficiently test capabilities such as **long-context stitching, many-hop reasoning, strong-prior override, finding errors in reasoning traces,** and **induction from examples**. BBEH was therefore designed to restore a benchmark that is difficult for frontier models, diverse across reasoning types, automatically scorable, lower in chance performance, more robust against shortcuts, and more reflective of real-world reasoning demands [2502.19187].

This design goal is explicitly benchmark-level rather than task-local. BBEH aims to preserve the broad capability coverage that made BIG-Bench and BBH useful while replacing each task with a harder analogue in the same reasoning domain. The result is not a subset of BIG-Bench, but a new benchmark inspired by BBH’s structure and intended to measure broad, robust, cross-domain reasoning under more difficult conditions [2502.19187].

## 2. Scope, composition, and targeted capabilities

BBEH contains **23 tasks** and, for most tasks, **200 questions**. The only exception is **Disambiguation QA**, which has **120 questions**, yielding a total of **4520 examples** via \(22 \times 200 + 120 = 4520\). Each BBEH task replaces one BBH task, which preserves BBH’s diversity profile while changing the actual task instances and often the task format itself [2502.19187].

The benchmark targets both the broad reasoning skills already associated with BBH-like evaluation and an expanded set of harder capabilities. The paper first identifies the following skills as necessary for BBH-like reasoning: **temporal understanding; spatial and geometric understanding; commonsense understanding; humour understanding; causal understanding; reasoning about world entities and events; deductive logical reasoning; reasoning through linguistic knowledge; counting and filtering; data structures and algorithms;** and **performing arithmetic operations**. BBEH then adds explicit emphasis on **many-hop reasoning; very long-range dependency; going against strong prior; learning on the fly; dealing with distractors; long-context reasoning; needle in a haystack; finding errors in reasoning traces; inductive reasoning; constraint satisfaction; compositional understanding;** and **knowledge-intense reasoning** [2502.19187].

The benchmark paper quantifies the increase in difficulty in two benchmark-wide ways. It states that the **macro average context length** in BBEH is about **6×** that of BBH, and that the **macro average output length** of Gemini 2.0 Flash responses, used as a proxy for required thinking, is about **7×** larger than on BBH. This scaling of context and response length is central to the benchmark’s design: BBEH is intended not merely to be “harder” in the sense of lower accuracy, but harder because it forces longer-range state maintenance, denser compositional reasoning, and more brittle interaction between instructions, priors, and distractors [2502.19187].

## 3. Task architecture and construction principles

The benchmark’s high-level design principle is one-to-one replacement: for every BBH task, the authors created a new task in the same reasoning domain that probes similar or stronger abilities at substantially higher difficulty. Construction was **semi-adversarial**. The reference general-purpose model was **Gemini 1.5 Flash**, and the reference reasoning-specialized model was **Gemini Thinking Experimental**, later **Gemini-2.0-Flash-Thinking-Exp-01-21**. For each task, the authors iteratively increased difficulty and re-evaluated; if the reference models performed too well, the task was replaced or further hardened until **both reference models scored below 70% accuracy** on each task. The paper explicitly notes a caveat: because the benchmark was partly adversarially tuned against specific reference models, it may be biased toward their failure modes [2502.19187].

| BBEH task | BBH task replaced | Salient design change |
|---|---|---|
| Boardgame QA | Logical Deduction | Depth 6–8 reasoning, defeasible rules, learned conflict resolution |
| Boolean Expressions | Boolean Expressions | Five candidates, only one true, operands can be propositions |
| Buggy Tables | Penguins in a Table | Reconstruct corrupted table, then answer statistical queries |
| Causal Understanding | Causal Judgement | Three-way labels plus necessary/sufficient cause reasoning |
| Disambiguation QA | Disambiguation QA | Longer, more complex pronoun disambiguation with more options |
| Dyck Language | Dyck Languages | Find first mistake in a reasoning trace |
| Geometric Shapes | Geometric Shapes | Multiple shapes, broken segments, intersections, distractor commands |
| Hyperbaton | Hyperbaton | Infer a new adjective order from 50–250 examples |
| Linguini | Salient Translation Errors | IOL-style linguistic puzzles across four categories |
| Movie Recommendation | Movie Recommendation | Choose the set whose movies are all likely to be liked |
| Multi-step Arithmetic | Multi-step Arithmetic | New recursive operators and long compositional expressions |
| New Yorker Cartoon Caption (NYCC) | Ruin Names | Pick funniest caption from 10 options |
| Object Counting | Object Counting | Long lists, distractors, sum/difference queries |
| Object Properties | Colored Objects | Temporal updates over object collections and properties |
| SARC Triples | Snarks | Three sarcasm judgments per item from Reddit post-reply pairs |
| Shuffled Objects | Shuffled Objects | No-op switches and long-range references to named actions |
| Spatial Reasoning | Navigate | More hops, intersecting paths, forward and backward reasoning |
| SportQA | Sports Understanding | Hardest Level 3 multi-hop compositional questions |
| Temporal Sequences | Temporal Sequences | Weekly meeting scheduling with zones, buffers, and constraints |
| Time Arithmetic | Date Understanding | Compositional date/time questions with derived variables |
| Web of Lies | Web of Lies | LiveBench-style cases plus harder cyclic cases |
| Word Sorting | Word Sorting | Error detection in traces or sorting under modified alphabets |
| Zebra Puzzles | Formal Fallacies | 5x5 to 8x8 puzzles with distracting clues |

Several task-specific construction details illustrate the benchmark’s philosophy. In **Boolean Expressions**, longer expressions alone were insufficient because the model could simply write Python; the authors therefore inserted **textual/mathematical propositions** in place of raw `True/False` literals. In **Hyperbaton**, each sample uses a unique induced adjective ordering, forcing **induction** and **prior override** rather than memorized English adjective order. In **Temporal Sequences**, the model must output both the **longest possible meeting length** and the **number of such possibilities**, and both must be correct for credit. In **Web of Lies**, the harder subset includes cyclic cases where some truth values remain unknown but useful conclusions still follow [2502.19187].

BBEH mixes newly authored tasks with adapted prior resources. The benchmark description identifies **Disambiguation QA, Hyperbaton, Boolean Expressions, Buggy Tables, Object Properties, Temporal Sequences, many Web of Lies examples, some Spatial Reasoning constructs, modified Movie Recommendation,** and **modified Multi-step Arithmetic** as new author-created or heavily modified components. It also lists adapted sources such as **BoardgameQA, MoCa causal stories, Kiciman et al., BIG-Bench Mistake traces, GeomVerse, Linguini, New Yorker caption datasets, SARC, SpatialLLMEval, SportQA, Test of Time, LiveBench Web of Lies,** and generated zebra puzzles following prior work. Curation was sometimes substantial: **Disambiguation QA** involved **10 annotators**, a separate reviewer, and **25 examples** revised after review; the necessary/sufficient-cause component of **Causal Understanding** was corrected by **three causal reasoning experts**, with outputs changed for **six examples** [2502.19187].

## 4. Evaluation protocol and scoring

The benchmark evaluates both general-purpose and reasoning-specialized systems. The general-purpose models listed in the benchmark paper are **Llama 3.1 8B Instruct, Gemma2 27B IT, Gemini 2.0 Flash-Lite, Gemini 2.0 Flash,** and **GPT-4o (2024-11-20)**. The reasoning-specialized models are **Distill R1 Qwen 32B, DeepSeek R1,** and **o3-mini (high)**. The paper also reports a score for **Gemini Thinking Experimental / Gemini-2.0-Flash-Thinking-Exp-01-21**, though it is not in the main task table [2502.19187].

All tasks append the same standardized suffix instructing the model to think step by step and to end with a machine-extractable answer prefixed by **“The answer is:”**. Extraction searches for one of four answer prefixes—`"The answer is: "`, `"The answer is "`, `"The final answer is: "`, or `"The final answer is "`—and then applies minimal cleaning, including stripping wrappers, lowercasing, and allowing small formatting equivalences such as a bare letter instead of `(A)` and comma-spacing normalization for multi-element labels. The reported evaluation stack is also benchmark-specific: **AI Studio** for Gemini 2.0 and Gemma2, **OpenAI API** for GPT-4o and o3-mini (high), **Together AI API** for DeepSeek R1, and local GPU loading for publicly available Llama and Distill R1 Qwen [2502.19187].

The headline metric is the **adjusted harmonic mean** of task accuracies. The rationale is that micro and macro averages can obscure severe weakness on individual tasks, whereas robust general reasoning should require competence across the entire suite. To avoid zero values in harmonic aggregation, the benchmark paper states that it **adds 1 to all accuracy numbers**. It also reports **micro average accuracy** for completeness. This dual reporting is important because the two views can diverge sharply: a model can obtain a relatively high micro average while being badly penalized by harmonic mean for failing on a subset of tasks [2502.19187].

## 5. Empirical profile, saturation evidence, and failure modes

The main empirical result is that BBEH remains difficult even for strong models. On harmonic mean accuracy, the benchmark paper reports: **Random 2.4**, **Llama 3.1 8B Instruct 3.6**, **Gemma2 27B IT 4.0**, **Gemini 2.0 Flash-Lite 8.0**, **Gemini 2.0 Flash 9.8**, **GPT-4o 6.0**, **Distill R1 Qwen 32B 5.2**, **DeepSeek R1 6.8**, and **o3-mini (high) 44.8**. On micro average accuracy, it reports: **Random 8.4**, **Llama 3.1 8B Instruct 10.6**, **Gemma2 27B IT 14.8**, **Gemini 2.0 Flash-Lite 19.7**, **Gemini 2.0 Flash 23.9**, **GPT-4o 22.3**, **Distill R1 Qwen 32B 19.2**, **DeepSeek R1 34.9**, and **o3-mini (high) 54.2**. The paper also notes that the reference Gemini thinking model reaches **20.2% harmonic mean** [2502.19187].

The benchmark’s anti-saturation claim is supported by direct BBH-to-BBEH comparison. For **Gemini 2.0 Flash**, the overall **BBH counterpart score** is **85.2**, while the corresponding **BBEH score** is **23.90**. Selected task drops include **Boolean Expressions 97.6 → 27.0**, **Hyperbaton 94.8 → 4.5**, **Multistep Arithmetic 99.6 → 9.5**, **Object Properties / Colored Objects 96.8 → 1.5**, **Object Counting 97.6 → 11.0**, **Shuffled Objects 100.0 → 9.0**, **Spatial Reasoning / Navigate 97.6 → 18.5**, **Temporal Sequences 98.8 → 0.5**, **Web of Lies 94.8 → 18.5**, and **Buggy Tables / Penguins in a table 98.6 → 3.5**. The only noted exception is **Disambiguation QA**, which improves from **42.0 → 48.3**, partly attributed to ambiguity in BBH zero-shot prompting [2502.19187].

Performance differences across model classes are highly nonuniform. The paper states that reasoning-specialized models gain the most on tasks involving **counting, planning, arithmetic,** and **data structures and algorithms**. Examples in the comparison between **o3-mini (high)** and **GPT-4o** include **Object Counting 90.0 vs 6.5**, **Buggy Tables 59.5 vs 0.5**, **Temporal Sequences 68.5 vs 0.0**, **Word Sorting 77.5 vs 22.0**, and **Multi-step Arithmetic 73.0 vs 5.5**. By contrast, gains are small or absent on “softer” reasoning tasks such as **commonsense, humour, sarcasm,** and **causal understanding**: **NYCC 23.0 vs 16.0**, **SARC Triples 38.5 vs 24.0**, **Causal Understanding 54.0 vs 54.0**, and **SportQA 25.0 vs 26.5**. Benchmark-wide ceilings remain low: **o3-mini (high)** exceeds **70%** on only **4 of 23 tasks**, **DeepSeek R1** on only **3 of 23**, and all other models on none [2502.19187].

The error analyses are unusually rich. In **BoardgameQA**, models overpredict `unknown`; the paper reports `unknown` prediction rates of **77.6%** for Gemma2 27B IT, **67.4%** for Gemini 2.0 Flash-Lite, **73.3%** for Gemini 2.0 Flash, **82.4%** for GPT-4o, **39.7%** for DeepSeek R1, and **65.5%** for o3-mini (high), despite only one-third of labels actually being unknown. In **Dyck Language**, when models flag an erroneous step, most errors come from missing the first error and flagging a later one; the share of such errors is **98.7%** for o3-mini, **100%** for Gemini 2.0 Flash, **94.9%** for Gemini 2.0 Flash-Lite, and **96.8%** for GPT-4o. In **Geometric Shapes**, intersections are especially damaging: for o3-mini, **no intersections 72%** versus **intersections 33%**. In **Spatial Reasoning**, backward reasoning is much harder than forward-only reasoning: for o3-mini, **58.8%** on forward-only versus **19.2%** on backward; for DeepSeek R1, **48.6%** versus **3.8%**. In **Temporal Sequences**, requiring both the longest meeting length and the number of ways raises difficulty substantially: for o3-mini, **68.5%** on the full task versus **78%** if only the longest length is asked; for Gemini Flash, **0.5%** versus **5%**. In **Shuffled Objects**, the paper notes that Gemini 2.0 Flash runs out of effective output tokens on **25%** of problems [2502.19187].

The benchmark paper also ties system-level trends to structural properties of tasks. It reports that **o3-mini vs GPT-4o** gains generally increase as both **context length** and the **output-length proxy for required thinking** increase, whereas **Gemini 2.0 Flash vs Flash-Lite** gains increase with context length but not much with the output-length proxy. This suggests that reasoning-specialized models benefited from both greater context and greater reasoning depth, while scaling a general model primarily helped with longer input handling rather than the hardest multi-step inference [2502.19187].

## 6. Later uses and research significance

Subsequent work quickly adopted BBEH as a hard reasoning benchmark. In **"Automatic Prompt Generation via Adaptive Selection of Prompting Techniques"**, BBEH is the **sole benchmark** used to validate an automatic prompt-generation framework. That paper uses all **23 tasks**, explicitly treating BBEH as appropriate for **“clear measurement of prompt effects at high difficulty”** and **“evaluation of versatility.”** Its reported aggregate scores on BBEH are **Original 23.9 / 9.7**, **Anthropic 24.7 / 10.5**, **Ours 28.0 / 12.5**, and **Ours (temperature-optimized) 28.5 / 13.3** for arithmetic mean / harmonic mean across tasks [2510.18162].

In **"Once Upon an Input: Reasoning via Per-Instance Program Synthesis"**, BBEH again appears in full: the paper evaluates **all 23 BBEH tasks** and argues that BBEH is appropriate because it mixes more and less algorithmic instances, making it suitable for testing both routing between CoT and synthesis and iterative program refinement. For **Gemini-2.0-Flash**, the appendix reports BBEH-only harmonic mean comparisons of **PoT 0.095**, **PoT-retries 0.098**, **CodeAct 0.040**, **BoT 0.027**, and **PIPS 0.171**; the main BBEH ablation table reports **PIPS 20.8**, **PIPS (no switch) 18.3**, **PIPS[0] (no switch) 12.9**, and **PIPS[0] (no switch, no symbols) 4.3**, showing that switching, iterative refinement, and explicit symbols all matter on the benchmark [2510.22849].

In **"Prompting Policies for Multi-step Reasoning and Tool-Use in Black-box LLMs with Iterative Distillation of Experience"**, BBEH is used more narrowly: the reported BBEH experiments focus on **three tasks**—**Disambiguation QA, Dyck Languages,** and **Web of Lies**—rather than the full suite. Even on that subset, the benchmark is used to test whether a learned prompting policy can induce more reliable structured reasoning in a frozen worker model. The reported results are **57.08% → 65.41%** on Disambiguation QA, **63.33% → 91.25%** on Dyck Languages, and **52.50% → 90.12%** on Web of Lies for baseline versus prompter policy with buffer [2605.14443].

In **"Evolutionary Generation of Multi-Agent Systems"**, BBEH serves as the paper’s main hard reasoning benchmark for testing automatically generated multi-agent systems. That paper describes BBEH as a **23-task** benchmark with **4,520 samples** and evaluates performance using the benchmark’s **standard accuracy metric**. Its headline BBEH result is **58.7%** for **EvoMAS (LLM-Selection)**, compared with **52.8%** for the best fixed-backbone EvoMAS, **48.2%** for the best **EvoAgent** result, and **46.2%** for the best predefined MAS result; it also reports **98.9%** execution rate on BBEH. The appendix notes a benchmark-specific systems fact: **BBEH does not provide external tools**, so on BBEH mutations are limited to **prompt editing, model selection,** and **communication topology** [2602.06511].

These later uses collectively establish BBEH as a benchmark for adaptive prompting, per-instance program synthesis, prompt-policy learning, and multi-agent inference architecture. A plausible implication is that BBEH’s primary value lies in difficulty and heterogeneity rather than in serving as a compact predictive proxy for broader BIG-Bench behavior, because related work on BIG-Bench found that BIG-Bench Hard was not especially good for recovering full-benchmark performance [2305.14947]. A second plausible implication is that BBEH embodies a different philosophy from difficulty-calibrated suites such as Easy2Hard-Bench: BBEH hardens tasks through one-to-one replacement and semi-adversarial construction, whereas Easy2Hard-Bench emphasizes continuous item-level difficulty labels estimated with IRT or Glicko-2 [2409.18433].

BBEH therefore occupies a distinctive position in the evaluation landscape. It is not a math-only or code-only stress test, nor a purely item-difficulty-calibrated continuum. It is a task-diverse, automatically scorable, anti-saturation benchmark that preserves BBH’s breadth while substantially increasing the demands of long-context reasoning, distractor handling, inductive rule learning, prior override, reasoning-trace critique, and cross-task robustness [2502.19187].

Source: https://www.emergentmind.com/topics/big-bench-extra-hard-bbeh-d2db077b-9839-4d02-a9b4-e3a52c1c082c