Grounded Satirical Generation with RAG

Abstract: Humor generation remains challenging task for LLMs, due to their subjective nature. We focus on satire, a form of humor strongly shaped by context. In this work, we present a novel pipeline for grounded satire generation that uses Retrieval-Augmented Generation (RAG) over current news to produce satirical dictionary definitions in the Finnish context. We also introduce a new task-specific evaluation framework and annotate 100 generated definitions with six human annotators, enabling analysis across multiple experimental conditions, including cultural background, source-word type, and the presence or absence of RAG. Our results show that the generated definitions are perceived as more political than humorous. Both topic-based word selection and RAG improve the political relevance of the outputs, but neither yields clear gains in humor generation. In addition, our LLM-as-a-judge evaluation of five state-of-the-art models indicates that LLMs correlate well with human judgments on political relevance, but perform poorly on humor. We release our code and annotated dataset to support further research on grounded satire generation and evaluation.

• The paper demonstrates that RAG-augmented pipelines enhance political relevance in satirical definition generation while humor remains low.
• It employs a multi-step process using web scraping, topic modeling, semantic retrieval, and Llama-3-8B-Instruct for context-aware satire generation.
• Quantitative evaluations reveal strong alignment between LLM and human ratings for political relevance, highlighting key challenges in humor assessment.

Grounded Satirical Generation with RAG: Technical Summary and Implications

Introduction and Motivation

The paper "Grounded Satirical Generation with RAG" (2605.10853) addresses the persistent challenge of automatic humor generation, with a focus on satire—a cultural and context-dependent form of humor. Unlike generic joke generation, satire demands both political relevance and a nuanced grounding in local sociopolitical discourse. Existing approaches in humor detection and generation with LLMs have consistently struggled with generation quality, particularly for subjective and culturally loaded phenomena like satire.

This work proposes a pipeline leveraging Retrieval-Augmented Generation (RAG) and news data from the Finnish public broadcaster Yle to generate satirical dictionary definitions that are grounded in recent, local news. The evaluation encompasses both human and LLM-based judgment and a new annotated dataset, supporting rigorous, multi-faceted analysis.

Methodology

The core pipeline operates in several sequential stages:

1. Web Scraping and Filtering: An in-house scraper collects English-language Finnish news articles, filtered for recency (≤ 30 days) and acceptable sentiment (using NLP-Town’s BERT-based sentiment model).
2. Topic Modeling and Candidate Selection: Candidate words are selected using BERTopic topic modeling, with embeddings produced via paraphrase-MiniLM. This procedure yields both news-driven and random word lists for experimentation.
3. Contextual Retrieval: For each candidate, semantic search retrieves up to three relevant news snippets. Relevance is enforced by cosine similarity (MiniLM-based) above a defined threshold.
4. RAG-based Satirical Generation: The retrieved contextual information is provided to Llama-3-8B-Instruct via a strictly formulated prompt that focuses on cynical, context-specific satire and prohibits reliance on generic stereotypes or knowledge not in the retrieved context.

   Figure 1: Overview of the satirical dictionary generation pipeline, integrating web scraping, topic word selection, semantic retrieval, and RAG-based generation.

Evaluation Framework

To address the subjectivity of satire and humor, the authors devise a two-pronged evaluation strategy:

• Human Annotation: Six annotators (split evenly between Finnish and international backgrounds) assess 100 definitions according to two dimensions: funniness and political relevance, both rated on 5-point Likert scales. Experimental variables include cultural background, word type (topic-modeled vs. random), and use of RAG.
• LLM-as-a-Judge: Five open-weight, mid-size instruction-tuned LLMs (including Qwen2.5, EuroLLM-9B, Aya-Expanse-8B, Mistral-7B, Llama-3.1) are tasked with scoring the same definitions using identical annotation instructions.

Quantitative Results and Claims

The generated satirical definitions demonstrate stronger political relevance than humor according to human judges. The mean "funny" score is 1.98 (SD=1.06), indicating low perceived humor, whereas the "political" mean is 2.53 (SD=1.55), reflecting a broader score distribution and a modest tendency toward political interpretation.

Figure 2: Distribution of funniness scores among annotators demonstrates pronounced skew toward low humor ratings.

Figure 3: Comparative ratings from Finnish and international annotators show minor, non-significant variations across cultural backgrounds for both dimensions.

Notably, neither RAG nor topic-based candidate selection yield statistically significant improvement in funniness, although both enhance political relevance (p<0.001 for topic words and RAG on the political score). The Wilcoxon signed-rank and Mann-Whitney U tests confirm these results. The inter-annotator agreement on humor (Krippendorff's α ≈ 0.07) is low, consistent with the known subjectivity of the task; agreement on the political dimension is higher (α ≈ 0.51 overall, higher for Finnish annotators).

Figure 4: Comparison of average political relevance and funniness scores for topic-based versus random word definitions.

Figure 5: Effect of RAG augmentation on average ratings—RAG improves political relevance but not funniness.

Regarding LLM judgments, all evaluated models achieve high correlation with human ratings on the political dimension (Spearman's ρ ≈ 0.66–0.83), but very low correlation on funniness (ρ < 0.20). LLMs systematically overestimate funniness relative to human annotators and show compressed variance, failing to capture subjectivity and nuance in humor perception.

Figure 6: Correlation scatter between human and Aya-Expanse-8B funniness scores reveals weak concordance.

Figure 7: Correlation scatter for political relevance between humans and Aya-Expanse-8B demonstrates tight alignment.

Broader Implications

The primary conclusion—grounded generation robustly increases political relevance but does not solve satirical humor generation—has several implications:

• RAG in Humor Generation: While RAG robustly anchors outputs in verifiable external information, it does not, in the current prompt and LLM regime, provide the subtlety or creativity required for humor. The pipeline's design may improve explainability and relevance, but funniness remains elusive.
• Evaluation of Humor by LLMs: The dissociation between LLM and human agreement on humor indicates fundamental limitations in current LLM training or alignment for subjective, creative evaluative tasks. Political relevance, more objectively signaled in text, is better detected.
• Cultural and Linguistic Transfer: No significant cultural background effect on annotation was detected, contrary to standard expectations around satire's locality. This could reflect either the globalized nature of evaluated news topics, or limited annotator diversity and sample size.
• Creation and Release of Datasets: The annotated Finnish-English satirical definitions corpus, with human and LLM scores, constitutes a significant resource for future benchmarking and system development.

Figure 8: Distribution of political relevance ratings across all experimental conditions.

Limitations and Future Directions

Current limitations include restriction to English-language content, the limited size and cultural diversity of the annotator pool, and the inherent challenge in automating the humor annotation task. Extending the methodology to other languages, refining prompts to exploit target cultural humor, and developing or fine-tuning models for context-sensitive irony or dark humor detection are promising directions.

Furthermore, integrating alternative retrieval sources (e.g., social media, satirical publications) and applying generative feedback or adversarial training could incrementally enhance both the variety and funniness of generated outputs.

Conclusion

This study rigorously demonstrates that RAG pipelines grounded in local news can enhance the political relevance of generated satirical definitions, but effective humor generation remains an unsolved, highly subjective challenge. The misalignment between LLM and human judgments of humor underscores the complexity of computational creativity. The presented annotated corpus and evaluation framework will facilitate further research at the intersection of NLG, humor, and sociopolitical language modeling, with future work needed to bridge the gap between mechanical relevance and cultural comedic effect.