HumorBench: Benchmark for Humor Reasoning

• HumorBench is a rigorously-constructed benchmark that evaluates LLMs’ ability to reason through sophisticated humor in cartoon captions using multi-step association tasks.
• It employs standardized textual scene descriptions and objective joke-element rubrics to assess both explanation generation and precise joke-element identification.
• Systematic evaluation shows strong correlations with STEM reasoning tasks while revealing challenges with cultural nuances and complex wordplay in humor comprehension.

HumorBench designates a rigorously-constructed benchmark for evaluating LLMs’ (LLMs) reasoning and explanatory capacity regarding sophisticated humor in cartoon captions. Distinct from standard STEM or recall-based benchmarks, HumorBench targets the core reasoning processes involved in humor comprehension—hypothesis generation, backtracking, and multi-step association across concepts, cultural references, wordplay, and figurative language. It provides a systematically-annotated evaluation suite, leveraging objective joke-element rubrics and automated autograder protocols, to assess both model explanations and their coverage of essential humor mechanisms (Narad et al., 29 Jul 2025).

1. Dataset Construction and Content

HumorBench consists of approximately 300 unique cartoon–caption instances sourced from two domains:

• New Yorker Caption Contest (NYCC): Only cartoons whose winning captions placed in the top three were selected, ensuring each example embodies a cohesive, editor-vetted joke structure.
• Cartoonstock.com: Added to diversify humor styles beyond the NYCC’s specific editorial profile.

To decouple humor understanding from visual recognition, every cartoon is paired with a standardized, jargon-free textual scene description detailing objects, characters, and salient visual cues deemed essential for joke interpretation (e.g., “Two sharks face each other; a human stands on one shark’s back”).

Annnotation involved an initial set of 650+ candidate “joke elements”—atomic conceptual leaps crucial to “getting” the joke. Following iterative human and automated review cycles, this was refined to 499 high-quality elements covering the full corpus. An explicit “Hard” subset of 100 challenging examples, where state-of-the-art (SOTA) models showed pass rates from 60% down to 0%, further separates easy and hard humor reasoning cases (Narad et al., 29 Jul 2025).

2. Objective Annotation Protocols and Joke Element Taxonomy

Central to HumorBench is a taxonomy of objective humor elements, inspired by humor theory and professional editing practices. Each cartoon-caption pair is annotated with 1–3 minimal, checkable elements, each representing exactly one logical or associative leap required for full joke comprehension. Categories include:

• Wordplay and ambiguity: Puns, polysemy, idiomatic twists.
• Cultural or historical references: Celebrities, products, events.
• Perspective shifts: Role reversal, anthropomorphism.
• Literal vs. figurative interpretation: Tension between direct and metaphorical meanings.

Annotation guidelines strictly mandate single-concept focus, explicit textual checkability, and freedom from raters’ subjective humor judgments. Quality control includes audits by an experienced New Yorker editor and iterative autograder-based pruning, reducing inter-annotator disagreement to less than 5% (Narad et al., 29 Jul 2025).

3. Task Definitions and Evaluation Framework

HumorBench establishes two principal evaluation tasks:

• Explanation Generation: Given a cartoon’s description and caption, the LLM generates a concise (≤200 words), strictly objective explanation encapsulated within a prescribed XML format, <explanation>…</explanation>, to support autograder parsing and comparability.
• Joke-Element Identification: Using the above inputs and generated explanation, an LLM-based autograder evaluates whether each gold-standard joke element is covered explicitly—formulating a per-element binary PASS/FAIL label.

All model outputs are truncated at 1,000 tokens. The autograder (GPT-4o) is calibrated to approximate human judgment, achieving 92% agreement (14.8% false-positive, 6.5% false-negative) with domain experts on joke-element coverage (Narad et al., 29 Jul 2025).

4. Formal Evaluation Metrics

HumorBench employs standard information retrieval metrics for element-level scoring:

• Element Identification Accuracy: Let TP = # elements correctly identified, FP = # false-positive elements, FN = # missed ground-truth elements:
  • Precision = TP / (TP + FP)
  • Recall = TP / (TP + FN)
  • F₁ = 2 × (Precision × Recall) / (Precision + Recall)

Aggregate model performance is reported as the mean F₁ or accuracy across all examples, with separate breakdowns for the challenging Hard subset (Narad et al., 29 Jul 2025).

While HumorBench does not explicitly measure funniness, the autograder-based binary coverage proxy is corroborated by high agreement with human element judgments, thereby serving as a surrogate for human-explanation alignment.

5. Benchmark Outcomes and Principal Insights

Systematic benchmarking of a wide array of contemporary LLMs (OpenAI o3, GPT-4o, Gemini 2.5 Pro, Claude 3.7 Sonnet, DeepSeek R1, Llama 4 Maverick, Qwen 2.5) yields the following key results:

1. Transferability of Reasoning: LLM progress on STEM reasoning benchmarks has a strong positive correlation with humor reasoning (Spearman’s ρ = 0.736 vs. GPQA, ρ = 0.943 vs. ARC-AGI), revealing that hypothesis-driven, backtracking reasoning underlying STEM tasks generalizes effectively to sophisticated humor explanation tasks.
2. Sufficiency of STEM-Only Training: Models trained solely on STEM/self-play corpora (e.g., DeepSeek R1-Zero, Phi-4-Reasoning-Plus) perform comparably to those with general-domain training, indicating that high-level logical chaining is domain-general and not tethered to STEM content.
3. Test-Time Compute Scaling Effects: Increasing “thinking” token budgets at inference time generally increases performance for small/medium models but may degrade outcomes for already well-calibrated large models (e.g., Claude 3.7 Sonnet’s drop beyond a 1,024-token budget), suggesting an upper bound to useful chain-of-thought expansion for humor tasks.

Performance Ceilings: OpenAI o3 achieves ~87.5% element accuracy overall (Gemini 2.5 Pro and Claude 3.7 Sonnet ~80%); all models fall to ~60% on the Hard split, with certain fine-grained idioms, celebrity references, or culturally specific puns eluding all current models (Narad et al., 29 Jul 2025).

6. Implications, Limitations, and Future Work

The robust transfer from STEM reasoning to humor explanation undermines any strict separation between “logical” and “creative” model capabilities, positing ‘reasoning’ as a broadly abstract, cross-domain cognitive mechanism in modern LLMs. Nonetheless, HumorBench’s Hard subset exposes persistent limitations rooted in:

• Knowledge of obscure cultural references.
• Subtle wordplay and idiomatic nuances.
• The lack of integrated multimodal (visual–linguistic) representation—HumorBench currently supplies only textual scene summaries, not images.

Direct future extensions envisioned:

• Multimodal integration: Incorporating original cartoon images to evaluate joint visual–linguistic humor understanding.
• Rubric-supervised learning: Leveraging element annotations as intermediate supervision in LLM training or as reward signals in RL.
• Improved autograders: Enhancing robustness and reducing leniency bias for more reliable element verification, especially in cases of linguistic creativity or explanation ambiguity.

HumorBench will remain critical for tracking reasoning progress in next-generation models and may enable the principled study of humor generation and cultural resonance by directly benchmarking the coverage of stepwise, theory-grounded joke mechanisms (Narad et al., 29 Jul 2025).