---
title: 'HumorBench: Benchmark for Humor Reasoning'
url: https://www.emergentmind.com/topics/humorbench
type: topic
---

# HumorBench: Benchmark for Humor Reasoning

HumorBench designates a rigorously-constructed benchmark for evaluating large language models’ (LLMs) reasoning and explanatory capacity regarding sophisticated humor in cartoon captions. Distinct from standard STEM or recall-based benchmarks, HumorBench targets the core reasoning processes involved in humor comprehension—hypothesis generation, backtracking, and multi-step association across concepts, cultural references, wordplay, and figurative language. It provides a systematically-annotated evaluation suite, leveraging objective joke-element rubrics and automated autograder protocols, to assess both model explanations and their coverage of essential humor mechanisms [2507.21476].

## 1. Dataset Construction and Content

HumorBench consists of approximately 300 unique cartoon–caption instances sourced from two domains:
- **New Yorker Caption Contest (NYCC):** Only cartoons whose winning captions placed in the top three were selected, ensuring each example embodies a cohesive, editor-vetted joke structure.
- **Cartoonstock.com:** Added to diversify humor styles beyond the NYCC’s specific editorial profile.

To decouple humor understanding from visual recognition, every cartoon is paired with a standardized, jargon-free textual scene description detailing objects, characters, and salient visual cues deemed essential for joke interpretation (e.g., “Two sharks face each other; a human stands on one shark’s back”).

Annnotation involved an initial set of 650+ candidate “joke elements”—atomic conceptual leaps crucial to “getting” the joke. Following iterative human and automated review cycles, this was refined to 499 high-quality elements covering the full corpus. An explicit “Hard” subset of 100 challenging examples, where state-of-the-art (SOTA) models showed pass rates from 60% down to 0%, further separates easy and hard humor reasoning cases [2507.21476].

## 2. Objective Annotation Protocols and Joke Element Taxonomy

Central to HumorBench is a taxonomy of *objective* humor elements, inspired by humor theory and professional editing practices. Each cartoon-caption pair is annotated with 1–3 minimal, checkable elements, each representing exactly one logical or associative leap required for full joke comprehension. Categories include:
- **Wordplay and ambiguity:** Puns, polysemy, idiomatic twists.
- **Cultural or historical references:** Celebrities, products, events.
- **Perspective shifts:** Role reversal, anthropomorphism.
- **Literal vs. figurative interpretation:** Tension between direct and metaphorical meanings.

Annotation guidelines strictly mandate single-concept focus, explicit textual checkability, and freedom from raters’ subjective humor judgments. Quality control includes audits by an experienced New Yorker editor and iterative autograder-based pruning, reducing inter-annotator disagreement to less than 5% [2507.21476].

## 3. Task Definitions and Evaluation Framework

HumorBench establishes two principal evaluation tasks:
- **Explanation Generation:** Given a cartoon’s description and caption, the LLM generates a concise (≤200 words), strictly objective explanation encapsulated within a prescribed XML format, `<explanation>…</explanation>`, to support autograder parsing and comparability.
- **Joke-Element Identification:** Using the above inputs and generated explanation, an LLM-based autograder evaluates whether each gold-standard joke element is covered explicitly—formulating a per-element binary PASS/FAIL label.

All model outputs are truncated at 1,000 tokens. The autograder (GPT-4o) is calibrated to approximate human judgment, achieving 92% agreement (14.8% false-positive, 6.5% false-negative) with domain experts on joke-element coverage [2507.21476].

## 4. Formal Evaluation Metrics

HumorBench employs standard information retrieval metrics for element-level scoring:
- **Element Identification Accuracy:** Let TP = # elements correctly identified, FP = # false-positive elements, FN = # missed ground-truth elements:
  - Precision = TP / (TP + FP)
  - Recall = TP / (TP + FN)
  - F₁ = 2 × (Precision × Recall) / (Precision + Recall)
  
Aggregate model performance is reported as the mean F₁ or accuracy across all examples, with separate breakdowns for the challenging Hard subset [2507.21476].

While HumorBench does not explicitly measure funniness, the autograder-based binary coverage proxy is corroborated by high agreement with human element judgments, thereby serving as a surrogate for human-explanation alignment.

## 5. Benchmark Outcomes and Principal Insights

Systematic benchmarking of a wide array of contemporary LLMs (OpenAI o3, GPT-4o, Gemini 2.5 Pro, Claude 3.7 Sonnet, DeepSeek R1, Llama 4 Maverick, Qwen 2.5) yields the following key results:
1. **Transferability of Reasoning:** LLM progress on STEM reasoning benchmarks has a strong positive correlation with humor reasoning (Spearman’s ρ = 0.736 vs. GPQA, ρ = 0.943 vs. ARC-AGI), revealing that hypothesis-driven, backtracking reasoning underlying STEM tasks generalizes effectively to sophisticated humor explanation tasks.
2. **Sufficiency of STEM-Only Training:** Models trained solely on STEM/self-play corpora (e.g., DeepSeek R1-Zero, Phi-4-Reasoning-Plus) perform comparably to those with general-domain training, indicating that high-level logical chaining is domain-general and not tethered to STEM content.
3. **Test-Time Compute Scaling Effects:** Increasing “thinking” token budgets at inference time generally increases performance for small/medium models but may degrade outcomes for already well-calibrated large models (e.g., Claude 3.7 Sonnet’s drop beyond a 1,024-token budget), suggesting an upper bound to useful chain-of-thought expansion for humor tasks.

**Performance Ceilings:** OpenAI o3 achieves ~87.5% element accuracy overall (Gemini 2.5 Pro and Claude 3.7 Sonnet ~80%); all models fall to ~60% on the Hard split, with certain fine-grained idioms, celebrity references, or culturally specific puns eluding all current models [2507.21476].

## 6. Implications, Limitations, and Future Work

The robust transfer from STEM reasoning to humor explanation undermines any strict separation between “logical” and “creative” model capabilities, positing ‘reasoning’ as a broadly abstract, cross-domain cognitive mechanism in modern LLMs. Nonetheless, HumorBench’s Hard subset exposes persistent limitations rooted in:
- Knowledge of obscure cultural references.
- Subtle wordplay and idiomatic nuances.
- The lack of integrated multimodal (visual–linguistic) representation—HumorBench currently supplies only textual scene summaries, not images.

Direct future extensions envisioned:
- **Multimodal integration:** Incorporating original cartoon images to evaluate joint visual–linguistic humor understanding.
- **Rubric-supervised learning:** Leveraging element annotations as intermediate supervision in LLM training or as reward signals in RL.
- **Improved autograders:** Enhancing robustness and reducing leniency bias for more reliable element verification, especially in cases of linguistic creativity or explanation ambiguity.

HumorBench will remain critical for tracking reasoning progress in next-generation models and may enable the principled study of humor generation and cultural resonance by directly benchmarking the coverage of stepwise, theory-grounded joke mechanisms [2507.21476].

Source: https://www.emergentmind.com/topics/humorbench