Humor Generation Score (HGS)
- HGS is a multi-signal metric designed to assess AI-generated humor by combining subjective ratings, multi-persona feedback, pairwise win rates, and contextual relevance.
- The metric employs a weighted-sum of four signals—direct ratings, multi-persona assessments, pairwise comparisons, and topic relevance—offering a structured evaluation of joke quality.
- Empirical studies show that integrating a Knowledge Graph with iterative revision boosts mean HGS by 15.4%, highlighting its potential to enhance contextual humor generation.
Humor Generation Score (HGS) is a multi-signal evaluation metric introduced in “HumorPlanSearch: Structured Planning and HuCoT for Contextual AI Humor” to assess both the comedic quality and the contextual relevance of generated jokes. It is motivated as a more robust alternative to single-signal humor evaluation because humor is subjective and context-dependent. Within the HumorPlanSearch framework, HGS is used to evaluate context-sensitive humor by combining direct ratings, multi-persona feedback, pairwise win-rates, and topic relevance into a single weighted aggregation (Dubey, 15 Aug 2025).
1. Conceptual role and scope
HGS is defined as a metric for evaluating jokes in settings where humor is treated as situated rather than generic. The underlying motivation is that automated humor generation with LLMs often produces jokes that feel generic, repetitive, or tone-deaf because humor depends on the listener’s cultural background, mindset, and immediate context. HGS is therefore framed not merely as a funniness score, but as an evaluation device for contextual, cultural, and audience-sensitive humor quality (Dubey, 15 Aug 2025).
In this formulation, the metric is explicitly tied to the broader HumorPlanSearch pipeline, which models context through Plan-Search, Humor Chain-of-Thought (HuCoT) templates, a Knowledge Graph, novelty filtering via semantic embeddings, and an iterative judge-driven revision loop. The paper also emphasizes cultural nuance: HuCoT templates include Indian and Gen Z-Indian styles, and HGS is meant to reflect humor that is culturally and contextually grounded rather than only generically amusing (Dubey, 15 Aug 2025).
A common misconception is to equate HGS with a single direct humor rating. The paper rejects that reduction. HGS is constructed to measure both whether a joke is funny and whether it remains aligned with the initial topic and audience context (Dubey, 15 Aug 2025).
2. Formal definition and mathematical structure
The paper gives an explicit weighted-sum definition:
Here, denotes the weight for the -th evaluation signal, denotes the score for the -th signal, and the weights sum to $1$. The formal structure is therefore a weighted linear combination of four component scores (Dubey, 15 Aug 2025).
The paper does not provide the numerical values of the weights . It also does not specify the exact normalization method for each signal, any subcomponent formula beyond the weighted-sum statement, or any rescaling procedure. As a result, HGS is formally defined but operationally underspecified in several implementation details (Dubey, 15 Aug 2025).
That partial specification is important for interpretation. The metric has a clear aggregation form, but the exact calibration of the component signals remains implicit. This suggests that HGS should be read as a principled evaluation schema rather than a fully standardized benchmark with fixed universal coefficients.
3. Constituent signals
HGS aggregates four signals: Direct Vote Scoring, Multi-Persona Scoring, Pairwise Win-Rate, and Topic Relevance. Each captures a different aspect of humor evaluation.
| Component | Definition in the paper | Evaluative role |
|---|---|---|
| Direct ratings | direct 1–5 rating of the joke | straightforward subjective humor score |
| Multi-persona feedback | three personas evaluate both the joke and its HuCoT reasoning | multi-perspective judgment |
| Pairwise win-rates | binary comparison against other jokes to determine a win rate | relative quality assessment |
| Topic relevance | cosine similarity between the joke and the initial topic prompt | prompt alignment |
The direct rating is the simplest component: a direct 1–5 rating of the joke. The multi-persona component is more structured. Three distinct personas evaluate both the joke and its HuCoT reasoning: Enthusiastic Fan, Critical Critic, and Academic Analyst. This extends evaluation beyond the surface text by including the reasoning trace that produced the humor (Dubey, 15 Aug 2025).
The pairwise win-rate component is based on binary head-to-head comparisons between jokes, aggregated into a win-rate style score. This introduces relative judgment rather than relying only on absolute ratings. Topic relevance is defined as the cosine similarity between the generated joke and the initial topic prompt, so HGS also rewards topical fidelity (Dubey, 15 Aug 2025).
Taken together, these four signals mix direct funniness, persona-conditioned review, comparative preference, and contextual alignment. This suggests that HGS is intended to combine absolute and relative evaluation modes rather than privileging any single one.
4. Evaluation protocol and operational use
In the reported experiments, HGS is used within a Judge LLM-centered evaluation protocol. The default Judge LLM is llama3-70b-8192. The paper also reports a preliminary human evaluation involving 13 human judges from mixed cultural backgrounds, aged 20–45. These human judges informed the system design and contributed to the observation that a generic HuCoT could feel “dry” (Dubey, 15 Aug 2025).
The experimental study spans nine topics. The paper refers to aggregate analysis across all humor styles, reports mean performance, and indicates that figure error bars represent 95% confidence intervals. It therefore aggregates HGS at least across generated jokes or samples, across styles, and across system configurations, although the exact aggregation formula across topics or samples is not spelled out (Dubey, 15 Aug 2025).
HGS-like judging signals also participate in an iterative refinement process. In the judge-guided plan revision loop, strategies are revised if their average score is below 6.0, revision must be projected to improve the score by at least 0.2, and the maximum number of iterations is 2. The paper explicitly distinguishes this loop from the formal definition of HGS itself, but it shows that the same evaluative machinery is used not only for post hoc measurement but also for generation-time refinement (Dubey, 15 Aug 2025).
5. Reported empirical findings
The reported headline result is that the full HumorPlanSearch pipeline, denoted KG + Revision, achieves the highest mean performance among the compared configurations. The paper compares four configurations: Baseline (No KG, No Revision), KG Only, Revision Only, and KG + Revision (Full Pipeline) (Dubey, 15 Aug 2025).
Relative to a strong baseline, the full pipeline boosts mean HGS by 15.4%, with . The paper states that KG + Revision consistently performs best, and that aggregate analysis across all humor styles supports this conclusion (Dubey, 15 Aug 2025).
These results position HGS as the principal outcome variable for evaluating whether structured planning, knowledge reuse, and iterative revision improve contextual humor generation. The paper also notes that smaller LLMs, such as the “Llama3-8B class,” were less effective for nuanced joke generation. That claim concerns generation quality rather than the metric definition, but it bears directly on the observed HGS outcomes (Dubey, 15 Aug 2025).
6. Relation to adjacent humor metrics and major limitations
HGS belongs to a broader research trend toward richer humor evaluation, but it is unusual in being explicitly defined as a weighted multi-signal metric. Several nearby humor-generation papers do not define an HGS. “On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation” evaluates primarily with pass@k, human 1–5 funniness ratings, and evaluator rankings rather than a dedicated HGS formula (Shang et al., 6 Feb 2026). “Small But Funny: A Feedback-Driven Approach to Humor Distillation” uses Win Tie Rate, including WTR-H and WTR-C, as the operative humor-quality score (Ravi et al., 2024). “OxfordTVG-HIC: Can Machine Make Humorous Captions from Images?” uses a learned humour score from a binary image–text classifier together with a benign score (Li et al., 2023). “Assessing the Capabilities of LLMs in Humor: A Multi-dimensional Analysis of Oogiri Generation and Evaluation” uses 0–4 human ratings on Novelty, Clarity, Relevance, Intelligence, Empathy, and Overall Funniness rather than a single weighted scalar (Sakabe et al., 12 Nov 2025).
This broader landscape suggests that HGS occupies a specific methodological position: more structured than a single funniness score, but more compact than a fully multi-axis manual annotation framework. Its design also resonates with earlier work treating humorousness as a continuum rather than a binary label, including Gaussian Process Preference Learning for latent humorousness ranking (Miller et al., 2020).
The paper is explicit about HGS’s limitations. The human-evaluation sample is small, with only 13 judges, which limits generalizability. HGS “has not yet been grounded against real-world performance data.” The evaluation was limited in scale by computational constraints. The authors also caution that individual LLM-generated scores should be interpreted with caution due to contextual biases, meaning that HGS may inherit biases from the judge model and the persona-based evaluation process (Dubey, 15 Aug 2025).
The paper therefore treats HGS as an early-stage multi-factor metric rather than a validated gold standard. Proposed next steps include direct comparison to other humor generation frameworks, live audience validation, and field studies in stand-up or open-mic settings (Dubey, 15 Aug 2025).
