Oogiri-Corpus: A Humor Dataset
- Oogiri-Corpus is a family of datasets derived from Japanese improvisational comedy, integrating human responses and LLM-generated content with six-dimensional humor ratings.
- It consolidates resources from sources like Bokete, Oogiri-Chaya, and Oogiri Sogo, employing rigorous filtering and annotation protocols across multimodal data.
- The corpus supports benchmark tasks for humor generation and evaluation, revealing that LLMs excel in novelty yet lag behind in empathy compared to human judgments.
Searching arXiv for the cited Oogiri papers to ground the article in current preprints. I’m going to look up the relevant arXiv entries for Oogiri-Corpus and adjacent Oogiri humor benchmarks. Oogiri-Corpus is a dataset resource for studying humor through Oogiri, a Japanese creative-response game in which a participant is given a topic/prompt and must produce a humorous response or punchline. In the 2025 literature, the term is used in two closely related senses. In "Assessing the Capabilities of LLMs in Humor: A Multi-dimensional Analysis of Oogiri Generation and Evaluation" it denotes an annotated corpus built from existing Oogiri data, newly collected Oogiri data, and LLM-generated responses, with six-dimensional ratings for humor generation and evaluation (Sakabe et al., 12 Nov 2025). In "Oogiri-Master: Benchmarking Humor Understanding via Oogiri" it denotes the underlying dataset used for linguistic analysis and for constructing a humor-understanding benchmark from Oogiri Sogo data (Murakami et al., 25 Dec 2025). This suggests a terminological ambiguity: Oogiri-Corpus is best understood as a family of Oogiri-centered research resources rather than a single immutable artifact.
1. Oogiri as a computational humor substrate
Oogiri is described as a Japanese improvisational comedy format, and in its modern form as Tonchi (頓智), where participants are given an image, text, or both and must produce an unexpected, witty, and humorous response. This makes it a prompt-conditioned humor task rather than a generic joke-generation problem. The format is especially valuable for computational humor because a good response must typically be on-topic, clear, novel, show intelligence, feel relatable / empathetic, and be funny overall (Sakabe et al., 12 Nov 2025, Zhong et al., 2023).
The Oogiri setting is used to argue for two complementary views of humor. One is a multidimensional view, in which humor is not reducible to a binary “funny/not funny” judgment. The other is a creative-reasoning view, in which Oogiri requires remote association, conceptual leaps, and thinking outside the box, rather than only step-by-step logical chaining. The Oogiri-GO work frames this as Leap-of-Thought (LoT), a non-sequential associative reasoning mode distinct from Chain-of-Thought (CoT) (Zhong et al., 2023). The six-dimensional Oogiri-Corpus work frames it in terms of incongruity-resolution, arguing that LLMs can often produce the “incongruity” part via novelty but struggle with the human-like grounding needed for humorous resolution, especially Empathy (Sakabe et al., 12 Nov 2025).
A common misconception is that Oogiri datasets are merely humor corpora in the narrow sense of isolated joke texts. The literature instead treats them as structured resources for examining how humor depends on the relation between prompt and response, on shared social context, and on human judgments collected under specific voting or annotation protocols (Sakabe et al., 12 Nov 2025, Murakami et al., 25 Dec 2025).
2. Corpus lineages and dataset construction
The main Oogiri-Corpus lineages are closely tied to earlier Oogiri dataset construction efforts. The precursor dataset is Oogiri-GO, introduced as a multimodal and multilingual Oogiri-GO dataset containing more than 130,000 high-quality samples after filtering, with inputs in English, Chinese, and Japanese and task types I2T, T2T, and IT2T (Zhong et al., 2023). That resource was built from Bokete, Twitter, and Weibo, beginning with over 200,000 raw samples, then applying Qwen-VL screening, NudeNet-based NSFW filtering, and two rounds of iterative manual screening (Zhong et al., 2023).
The six-dimensional Oogiri-Corpus extends this lineage by combining existing Oogiri data, newly collected Oogiri data, and LLM-generated responses. Its two named human sources are Oogiri-GO, taken from Bokete, and Oogiri-Chaya, collected from another online Oogiri platform. The filtered source sizes are 1,329 topics for Oogiri-GO, consisting of 313 text-based and 1,016 image-based topics, and 551 topics for Oogiri-Chaya, consisting of 425 text-based and 126 image-based topics. For Oogiri-Chaya, the authors collected all samples between 21 Nov 2021 and 17 Jul 2024, used GPT-4o to detect potentially discriminatory, violent, sexual, or otherwise socially inappropriate content, manually reviewed flagged items, and removed inappropriate ones (Sakabe et al., 12 Nov 2025).
The Oogiri-Master usage of Oogiri-Corpus follows a different construction path. It was built by web crawling from the public Japanese Oogiri competition platform Oogiri Sogo. The authors collected 2,165 prompts, then excluded prompts with fewer than 100 total votes, leaving 908 prompts and 82,536 total prompt–response pairs, with about 95.9 candidate responses per prompt on average and about 171.6 votes per prompt on average (Murakami et al., 25 Dec 2025).
| Resource | Data basis | Key properties |
|---|---|---|
| Oogiri-GO (Zhong et al., 2023) | Bokete, Twitter, Weibo | More than 130,000 high-quality samples; I2T, T2T, IT2T; English, Chinese, Japanese |
| Oogiri-Corpus (Sakabe et al., 12 Nov 2025) | Oogiri-GO, Oogiri-Chaya, LLM-generated responses | Oogiri-GO: 1,329 topics; Oogiri-Chaya: 551 topics; benchmark selects 200 topics and 8 response types |
| Oogiri-Corpus (Murakami et al., 25 Dec 2025) | Oogiri Sogo | 908 prompts; 82,536 prompt–response pairs; about 95.9 responses and about 171.6 votes per prompt |
A central issue across these resources is popularity bias. The six-dimensional paper emphasizes that Bokete / Oogiri-GO allows users to see other responses and vote counts, which can create first-mover advantage and conformity / bandwagon bias, whereas Oogiri-Chaya hides responses during submission and hides vote counts during evaluation, making voting closer to “pure funniness” (Sakabe et al., 12 Nov 2025). The Oogiri-Master paper makes a parallel argument for Oogiri Sogo, where vote counts are not displayed during voting and voters cannot see others’ ratings / vote counts while voting (Murakami et al., 25 Dec 2025).
3. Annotation regimes and humor dimensions
The most explicit annotation protocol appears in the six-dimensional Oogiri-Corpus. It uses six 5-point absolute rating dimensions, described as 0–4 integer ratings where 0 = not at all / worst and 4 = excellent / best. The dimensions are Novelty, Clarity, Relevance, Intelligence, Empathy, and Overall Funniness (Sakabe et al., 12 Nov 2025). Formally, the rubric is
for each of
The human annotation setup recruits annotators via Lancers. Each response is rated by four native Japanese speakers, and the authors create eight distinct subsets so each annotator evaluates only one response type per topic. The stated purpose is to keep the rating fair and avoid direct comparisons within the same topic set. The same models used for generation—GPT-4.1, Gemini 2.5 Pro, and Claude Sonnet 4—are also used as judges, with the same rubric and instructions as human annotators (Sakabe et al., 12 Nov 2025).
The Oogiri-Master usage of Oogiri-Corpus does not rely on a separate annotation campaign for the main corpus. Instead, it uses the platform’s naturally collected vote counts as the funniness signal. Each response has a vote count indicating perceived funniness, and those votes are collected independently in the platform’s voting phase. For analysis, responses are stratified by vote rank within each prompt: High-rated responses are the top 3 responses by vote count, and Low-rated responses are the bottom 3 responses. This yields 5,448 responses total, i.e. 908 prompts × 6 responses (Murakami et al., 25 Dec 2025).
The Oogiri-Master paper explicitly notes that it does not report a standard inter-rater reliability statistic such as Cohen’s kappa or Krippendorff’s alpha for the corpus votes, because the votes are collected through the platform rather than a controlled multi-rater annotation study. Its main quality-control measures are filtering out prompts with fewer than 100 votes, hiding vote visibility, and using top-vs-bottom response selection for analysis. For the separate human baseline in Oogiri-Master, each item is answered by 21 crowdworkers, with attention checks with unambiguous answers, and final labels are determined by majority vote among workers who passed the checks (Murakami et al., 25 Dec 2025).
4. Benchmark tasks and experimental formulations
The six-dimensional Oogiri-Corpus supports two benchmark tasks. Task 1: Oogiri generation asks whether an LLM can generate humorous Oogiri responses that humans judge as funny. The model is given a topic and asked to generate a humorous response in Japanese, and the generated outputs are then rated by humans on the six dimensions. Task 2: Six-dimensional Oogiri evaluation asks whether an LLM can judge humorous responses in a way that aligns with human evaluation. Here, each model is given a topic-response pair and asked to score it on the same six dimensions; the output is compared against human ratings using Spearman correlation (Sakabe et al., 12 Nov 2025).
For the benchmark evaluation in that work, the authors select 200 topics total, split into 100 text-based and 100 image-based topics. For each topic they prepare 8 response types: High-voted human response, Mid-voted human response, Low-voted human response, Unrelated response, GPT-4.1 serious, GPT-4.1 humorous, Gemini 2.5 Pro humorous, and Claude Sonnet 4 humorous. The models evaluated are GPT-4.1 (gpt-4.1-2025-04-14), Gemini 2.5 Pro (gemini-2.5-pro-preview-05-06), and Claude Sonnet 4 (claude-sonnet-4-20250514-v1:0). Generation and evaluation are both prompted in Japanese using a base prompt template, use default settings, and report no additional training (Sakabe et al., 12 Nov 2025).
Oogiri-Master transforms Oogiri-Corpus into a benchmark for humor understanding rather than generation. It contains 600 items total, with 400 items for relative judgment tasks and 200 items for binary classification. The five tasks are Binary_same, Binary_diff, Triple, Quad, and an absolute binary classification task that decides whether a response to a prompt is funny or not funny. The paper emphasizes Binary_diff, which compares a high-rated response for the prompt with a high-rated response from a different prompt, because it tests whether the model actually understands prompt-conditioned humor rather than generic funny style (Murakami et al., 25 Dec 2025).
The Oogiri-GO literature introduces a different task family oriented to creative reasoning. It includes choice questions such as 2T1, 3T1, 4T1, and 5T2, evaluated by Accuracy, as well as ranking questions evaluated by Top-1 accuracy and NDCG. In that setting, Oogiri is used to test associative generation, associative discrimination, and creative humor generation, and to support the CLoT framework for improving LoT ability (Zhong et al., 2023).
5. Empirical findings on generation and judgment
The most prominent finding from the six-dimensional Oogiri-Corpus is that LLMs generate Oogiri responses at a level between low-tier and mid-tier humans. The best model is Gemini 2.5 Pro. In Table 1, mean Overall Funniness is 2.563 for Human High, 1.913 for Human Mid, 1.232 for Human Low, 1.621 for GPT-4.1, 1.803 for Gemini 2.5 Pro, and 1.504 for Claude Sonnet 4 (Sakabe et al., 12 Nov 2025).
| Response source | Mean Overall Funniness |
|---|---|
| Human High | 2.563 |
| Human Mid | 1.913 |
| Human Low | 1.232 |
| GPT-4.1 | 1.621 |
| Gemini 2.5 Pro | 1.803 |
| Claude Sonnet 4 | 1.504 |
Dimension-level analysis shows that LLMs are relatively decent at Clarity, Relevance, and partly Novelty, but notably weaker on Empathy. The paper identifies Empathy as the biggest gap between humans and LLMs, and further argues that this deficit helps explain why LLMs fail to replicate human humor assessment (Sakabe et al., 12 Nov 2025).
Agreement between model judgments and human judgments is modest to low on Overall Funniness. The reported Spearman coefficients are 0.224 for GPT-4.1, 0.169 for Gemini 2.5 Pro, and 0.266 for Claude Sonnet 4. A key qualitative result is a difference in evaluation logic: Humans prioritize Empathy, whereas LLMs prioritize Novelty. The paper also notes that LLMs show stronger agreement on more “objective” dimensions like Clarity and Relevance, but that this does not carry over to the more subjective Overall Funniness dimension (Sakabe et al., 12 Nov 2025).
The paper’s qualitative error analysis names several biases. Positivity bias refers to the tendency to rate many responses too highly, including unrelated responses, weaker human responses, and sometimes even serious responses. Self-preference bias refers to LLMs often rating their own generated responses highly. Context negligence refers to the tendency to score unrelated responses surprisingly positively when humans sharply penalize them. The broader explanation is a Novelty-over-Empathy mismatch, in which models overvalue unexpectedness, originality, and surface-level cleverness, while humans care more about empathy, situatedness, and emotional resonance (Sakabe et al., 12 Nov 2025).
A concrete example illustrates this contrast. For the topic “Tear a coupon to pieces and say one line.”, the human high-rated response “A funeral director shouldn't say things like, ‘See you next time!’” received a human Empathy score of 3.00 and a human Funniness score of 2.00, whereas the GPT-4.1 response “So, this makes it zero calories now, right?” received 1.25 for Empathy and 0.75 for Funniness (Sakabe et al., 12 Nov 2025).
6. Linguistic analysis, benchmark performance, and research significance
The Oogiri-Master paper uses Oogiri-Corpus as a basis for quantitative analysis of what linguistic features correlate with funniness. It compares high-rated and low-rated responses using an independent two-sample Student’s t-test and Cohen’s d, with the interpretation that correspond to small, medium, and large effects. The feature families are basic linguistic features, semantic distance and textual entailment, surprisal and nPMI, and LLM-scored higher-order features (Murakami et al., 25 Dec 2025).
The higher-order features are rated on a 1–5 scale for ambiguity exploitation, associative distance, benign violation, coherence, expectedness, incongruity resolution, metaphor use, and perspective shift. The main findings are that funny responses are significantly shorter, have fewer unique characters and lower lexical novelty, and are especially associated with higher-order traits such as ambiguity exploitation, associative distance, benign violation, incongruity resolution, metaphor use, and perspective shift. Among these, perspective shift and ambiguity show the largest effects, while semantic distance and surprisal are weaker signals (Murakami et al., 25 Dec 2025).
The reported table-level results include: length: high 14.12 vs low 16.40, ; lexical novelty: high 0.80 vs low 0.93, ; ambiguity exploitation: high 2.10 vs low 1.61, ; associative distance: high 4.38 vs low 3.90, ; incongruity resolution: high 3.71 vs low 3.35, ; and perspective shift: high 2.40 vs low 1.87, (Murakami et al., 25 Dec 2025). These results provide an empirical complement to the six-dimensional finding that humor judgments depend on more than novelty alone.
In benchmark performance, Claude-Opus-4 and the human baseline both achieve the best average accuracy, 68.7%, while GPT-5 reaches 67.6%, Gemini-2.5-Pro reaches 53.4%, and LLM-jp-3.1-13b reaches 49.8%. The paper’s insight-augmented prompting incorporates features such as length, unique character count, prompt–response length ratio, symbol ratio, katakana ratio, ambiguity exploitation, associative distance, benign violation, incongruity resolution, metaphor use, and perspective shift. For GPT-5, performance rises from 67.6% to 70.7%, an improvement of +3.1 points (Murakami et al., 25 Dec 2025).
Several broader implications follow. First, Oogiri-Corpus supports the study of humor as a multidimensional cognitive and social phenomenon rather than a single scalar label (Sakabe et al., 12 Nov 2025). Second, it connects humor evaluation to creative-reasoning research, including LoT and CLoT, where Oogiri serves as a benchmark for associative generation and discrimination (Zhong et al., 2023). Third, it highlights persistent limitations: some humor is culture-specific, the Oogiri-Master benchmark is limited to understanding/judging humor rather than generation or explanation, the current work is text-to-text only in that benchmark, and human evaluation could be improved by accounting for annotator attributes (Murakami et al., 25 Dec 2025). A further misconception is that higher novelty alone should predict humor; the combined evidence instead points to a more structured interaction among prompt relevance, incongruity, perspective shift, and especially humanly grounded Empathy (Sakabe et al., 12 Nov 2025, Murakami et al., 25 Dec 2025).