Oogiri-Master: Benchmarking Humor Understanding via Oogiri

Abstract: Humor is a salient testbed for human-like creative thinking in LLMs. We study humor using the Japanese creative response game Oogiri, in which participants produce witty responses to a given prompt, and ask the following research question: What makes such responses funny to humans? Previous work has offered only limited reliable means to answer this question. Existing datasets contain few candidate responses per prompt, expose popularity signals during ratings, and lack objective and comparable metrics for funniness. Thus, we introduce Oogiri-Master and Oogiri-Corpus, which are a benchmark and dataset designed to enable rigorous evaluation of humor understanding in LLMs. Each prompt is paired with approximately 100 diverse candidate responses, and funniness is rated independently by approximately 100 human judges without access to others' ratings, reducing popularity bias and enabling robust aggregation. Using Oogiri-Corpus, we conduct a quantitative analysis of the linguistic factors associated with funniness, such as text length, ambiguity, and incongruity resolution, and derive objective metrics for predicting human judgments. Subsequently, we benchmark a range of LLMs and human baselines in Oogiri-Master, demonstrating that state-of-the-art models approach human performance and that insight-augmented prompting improves the model performance. Our results provide a principled basis for evaluating and advancing humor understanding in LLMs.

• The paper introduces a novel benchmark and dataset, Oogiri-Master and Oogiri-Corpus, to evaluate humor understanding in LLMs using Japanese creative response data.
• The paper leverages comprehensive linguistic analyses, including measures of response length, ambiguity, and perspective shift, to quantify humor.
• The paper demonstrates that feature-augmented prompting and native corpus pretraining can help LLMs match or exceed human performance in humor recognition tasks.

Benchmarking Humor Understanding in LLMs with Oogiri-Master

Introduction

The paper "Oogiri-Master: Benchmarking Humor Understanding via Oogiri" (2512.21494) addresses a critical gap in the rigorous evaluation of humor comprehension within LLMs by introducing Oogiri-Master, a novel benchmark, and Oogiri-Corpus, a large-scale dataset based on the Japanese Oogiri creative response game. Oogiri, where participants generate witty responses to prompts, functions as a demanding testbed for context-dependent creativity, linguistic nuance, and humor—a set of capabilities inadequately evaluated by previous benchmarks, which often suffer from methodological and data bias.

Dataset Construction and Properties

Oogiri-Corpus is constructed from the public Oogiri Sogo platform, comprising 908 prompts corresponding to 82,536 prompt–response pairs, each annotated with ~100 diverse candidate responses and independent funniness ratings from ~100 human judges. This scale and design decisively mitigate the popularity and structural bias inherent in earlier datasets such as Oogiri-GO. The curation process emphasizes the exclusion of prompts with insufficient human ratings, ensuring more robust aggregation of funniness and fairer representation of what constitutes humor for a specific community.

Key dataset properties:

• Average responses per prompt: ~96
• Average votes per response: 1.8
• The dataset is approximately seven times larger than Oogiri-GO for the text-to-text Japanese Oogiri domain.

Quantitative Linguistic Analysis of Funniness

The study advances the objective quantification of humor by systematically associating linguistic features with human funniness ratings. Using the constructed corpus, the authors conduct a two-group comparative analysis: "high-rated" (top 3 per prompt, average 8.5 votes) and "low-rated" (bottom 3, zero votes) responses. The feature set spans basic linguistic statistics, prompt–response interrelations, semantic metrics, information-theoretic quantities (e.g., surprisal, normalized PMI), and a range of higher-order features scored by competitive LLMs (e.g., ambiguitiy, associative distance, incongruity resolution, perspective shift, metaphor).

Key findings include:

• High-rated humorous responses are significantly shorter (Cohen's d $=-0.28$ for response length) with less lexical novelty, reinforcing classic wit metrics, and are more effective when they exploit ambiguity, shift perspective, or resolve incongruity (Cohen's d $=0.42$ for ambiguity exploitation, $0.36$ for incongruity resolution, $0.50$ for perspective shift).
• Simple surface-level features (e.g., response length) and complex interpretive cues (e.g., benign violation, as in McGraw and Warren's theory [McGraw2010-kg]) independently contribute to predicting funniness. This reinforces theoretical models of humor such as incongruity-resolution and benign violation accounts [Morreall2024PhilosophyHumor].
• Other features, such as semantic distance and surprisal, though statistically significant, exhibit minimal effect sizes and limited role in humor prediction in this task formulation.

Oogiri-Master: Benchmark Design and LLM Evaluation

Oogiri-Master formalizes humor understanding into five tasks: four relative-judgment multiple-choice question answering (MCQA) tasks and one absolute binary classification task. The negative class for MCQA incorporates both low-rated responses from the same prompt and unrelated high-rated responses from other prompts, testing both within-topic and context-sensitive humor discrimination.

The evaluation suite encompasses a spectrum of open and closed LLMs, as well as human baselines (crowdsourced Japanese speakers with majority vote aggregation). The analysis investigates both standard and insight-augmented prompting, where the latter exposes LLMs to computed linguistic features identified as important for funniness.

Numerical Results and Model Comparison

• Proprietary LLMs (Claude-Opus-4, GPT-5, Gemini-2.5-Pro) outperform open-source and smaller Japanese-adapted models (e.g., LLM-jp-3.1-13b).
• Claude-Opus-4 and GPT-5 achieve average task accuracy of 68.7% and 67.6% respectively with baseline prompts, matching human annotator consensus (68.7%).
• Insight-augmented prompting yields absolute improvements (+3.1 points for GPT-5, achieving 70.7%), exceeding both the strongest human baseline and competitive LLMs using naive prompting.
• Models pretrained or continually pretrained on a Japanese corpus (e.g., DeepSeek-R1-ja) exhibit consistent gains (+3.3 points baseline vs. DeepSeek-R1), underlining the importance of native-corpus adaptation for culture-sensitive humor tasks.

Ablation studies indicate that basic features such as length and surface ratios, when included alone, can improve top-tier LLM accuracy as much as higher-order feature inclusion. However, the effect is maximized when both classes are provided, especially under instruction paradigms that limit the use of these features to uncertainty scenarios rather than hard constraints, which can lead to overfitting and heuristic exploitation in less capable models.

Theoretical and Practical Implications

The research presents a reproducible, fine-grained framework for evaluating creative linguistic competence in LLMs, focusing specifically on humor—a domain requiring high-level understanding of cultural and contextual cues. The identification of humor-predictive linguistic features, validated both with robust statistics and in model performance, enables the systematic analysis of failures and successes in model reasoning.

Practically, Oogiri-Master justifies incorporating feature-guided instruction in prompt engineering and highlights pretraining strategies for improved local and cultural humor comprehension. The results suggest:

• Feature-based prompting, when judiciously applied (e.g., under uncertainty), provides measurable performance benefits and minimizes over-reliance on single heuristics.
• Continued pretraining with native corpus material significantly boosts higher-level pragmatic and cultural knowledge in LLMs.

Theoretically, this work supports an overview of classic humor theories and data-driven metrics, as features such as incongruity resolution and perspective shift emerge as generically predictive across annotator pools and model architectures.

Limitations and Future Directions

The methodology is, at present, limited to Japanese Oogiri; humor transfer and feature salience may not be directly preserved when ported to other languages or modalities. The benchmark focuses on "understanding" (recognition, ranking) rather than generation or explanation. Future work should extend the approach to multimodal and cross-cultural humor, incorporate attribute-aware annotator modeling, and evaluate generative as well as interpretive model behaviors.

Conclusion

Oogiri-Master and Oogiri-Corpus provide an extensible, high-fidelity platform for quantifying humor understanding in LLMs, with results indicating that state-of-the-art LLMs can—under feature-augmented and instruction-sensitive prompting—match or exceed average human performance in contextually anchored humor selection. The identification of robustly predictive linguistic features provides actionable insights for both LLM evaluation and future model development, with clear evidence that cultural adaptation and prompt engineering are critical levers for real-world creative NLP applications.