"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?

Abstract: Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. In multimodal puns, visual and textual elements synergize to ground the literal sense and evoke the figurative meaning simultaneously. Although Vision-LLMs (VLMs) are widely used in multimodal understanding and generation, their ability to understand puns has not been systematically studied due to a scarcity of rigorous benchmarks. To address this, we first propose a multimodal pun generation pipeline. We then introduce MultiPun, a dataset comprising diverse types of puns alongside adversarial non-pun distractors. Our evaluation reveals that most models struggle to distinguish genuine puns from these distractors. Moreover, we propose both prompt-level and model-level strategies to enhance pun comprehension, with an average improvement of 16.5% in F1 scores. Our findings provide valuable insights for developing future VLMs that master the subtleties of human-like humor via cross-modal reasoning.

• The paper introduces the MultiPun dataset and evaluation suite to detect, localize, and explain multimodal puns.
• It reveals that VLMs show high sensitivity but poor specificity, with prompt-induced biases and hallucination issues affecting performance.
• Interventions like Pun-CoT and Pun-Tuning significantly improve pun detection metrics, enhancing TNR and F1 scores.

Multimodal Pun Understanding in Large Vision-LLMs: An Expert Analysis

Introduction

The paper "I See What You Did There": Can Large Vision-LLMs Understand Multimodal Puns? (2604.05930) systematically investigates the ability of Vision-LLMs (VLMs) to interpret and explain multimodal puns—instances of visual-textual wordplay that require integrating phonetic, orthographic, and semantic cues across modalities. Recognizing the absence of robust benchmarks for this cognitive task, the authors introduce MultiPun, a dataset comprising rigorously constructed and filtered multimodal pun and non-pun instances. The study features a comprehensive evaluation suite and presents both prompt- and model-level interventions to improve VLM performance, followed by substantive empirical and error analyses.

The MultiPun Benchmark: Scope, Construction, and Task Suite

The MultiPun dataset embodies a linguistically principled approach to benchmarking multimodal pun understanding. It consists of 445 pun cases (homophonic and homographic) and 890 matched negative distractors, constructed through adversarial positive- and negative-sample generation pipelines Figure 1. The authors employ advanced LLMs for scenario synthesis, detailed filtering protocols (including semantic diversity and human verification), and two adversarial negative construction strategies: Explicative Substitution and Random Substitution, each specifically designed to disrupt the pun mechanism while preserving surface coherence.

Figure 1: Construction pipeline for MultiPun, illustrating controlled generation of both pun and adversarial non-pun items for systematic evaluation.

The evaluation suite targets three graduated tasks:

• Detection: Binary pun recognition without stimulus guidance.
• Localization: Extraction of explicit pun and alternative word pairs.
• Explanation: Structured rationale generation, including tuple extraction of the pun word, alternative, literal object, and figurative meaning.

Empirical Evaluation of VLMs on Multimodal Puns

The authors conduct a systematic evaluation on 11 state-of-the-art VLMs spanning both proprietary (e.g., GPT-5.1, GPT-4o, Gemini-3-Pro, Claude-Sonnet-4.5) and open-source (e.g., Qwen3-VL, LLaVA, Llama-4) architectures. Metrics capture true positive/negative rates, F1, Cohen’s $\kappa$, and task-specific criteria (component mention ratios, explanation win rates).

Key findings include:

• VLMs exhibit high sensitivity (TPR) but poor specificity (TNR), often classifying non-puns as puns. For instance, multiple closed and open models display TNRs below 0.4 on detection, indicative of superficial pattern matching.
• Prompt-induced biases are pronounced in open-source models. Models such as LLaVA-V1.6-Vicuna-13B show catastrophic swings ($\Delta_{TPR} > 0.9$) between positive and negative prompt phrasings, revealing alignment and sycophancy artifacts.
• Explanation tasks reduce false positives by enforcing rationale consistency, but at the expense of missed true puns (TPR reduction). The demand for structured explanations forces models to reject ambiguous or weakly justified positives, exposing the limits of shallow, non-grounded reasoning.

Figure 2: Example set highlighting phonetic (pear/pair), polysemous (fan), and negative cases, illustrating the required cross-modal reasoning for robust pun detection.

Pun Explanation: Component Analysis and Failure Modes

To probe model explanation ability, the authors distinguish between mention ratios for pun words ($w_p$) and alternatives ($w_a$) across tasks and compare human-grounded versus model-generated rationales Figure 3.

• Detection of explicit pun words ($w_p$) is trivial for most VLMs due to direct caption presence; inference of alternatives ($w_a$), especially for homophonic puns, remains a bottleneck. Reasoning-enhanced models such as Qwen3-VL-30B-A3B-Thinking show improved alternative recovery, but strong open-source models still trail proprietary LLMs in recall and rationale plausibility.
• Pairwise explanation scoring reveals that, even for high-performing models, over 90% of model explanations are inferior to gold rationales. Model-produced justifications frequently lack cross-modal integration and revert to surface associations.

Figure 3: Results of pairwise comparison for pun explanations, demonstrating the gap between VLM-generated and gold standard rationales.

Error typology in explanations includes:

• Pun keyword hallucination
• Phonetic hallucination (claiming non-existent sound similarity)
• Semantic hallucination (inventing meanings for anchor or alternative words)
• Visual object hallucination (projecting textual cues into image perception)

Pun Robustification: Prompt and Model-Level Interventions

To mitigate recurring hallucination and bias patterns, the authors propose:

• Pun-CoT (Chain-of-Thought Prompting): A three-stage protocol that enforces visual grounding, lexical anchoring, and cross-modal bridge verification to conservatively admit only genuine puns, reducing spurious associations. Pun-CoT consistently raises TNR and F1, especially in hallucination-prone open-source models.
• Pun-Tuning (Fine-Tuning with Adversarial Data): Supervised instruction tuning on balanced pun/non-pun samples, including both biased and adversarial prompts, which diminishes prompt-induced bias and hallucinations while raising TNR and F1 by substantial margins (e.g., Qwen3-VL-8B post-tuning F1: 0.505→0.679 on homophonic puns).

Implications and Future Prospects

The findings directly challenge the validity of using VLMs for nuanced multimodal linguistic reasoning tasks without carefully curated, adversarial benchmarks and context-sensitive reasoning mechanisms. The inability to distinguish between genuine multimodal ambiguity and superficial pattern matching has broad implications—for interpretability, adversarial robustness, and trust in vision-language systems in sensitive applications such as creative writing, social interaction, and cross-cultural communication.

Figure 4: Qualitative illustration of adversarial negative (distractor) samples used to rigorously probe hallucination and overfitting in VLMs.

The proposed MultiPun, coupled with robust prompting and training techniques, establishes a methodological paradigm for probing higher-order linguistic and visual reasoning in foundation models. Further, the results emphasize the need for cross-modal representation architectures that can explicitly model phonetic and semantic bridges, robustly reject non-puns, and generate interpretable evidence chains.

Future directions include:

• Extension to multilingual/cross-cultural wordplay (addressing language-specific phonology)
• Cross-generation and explanation tasks (pun creation and justification)
• Scaling fine-tuning with larger, more diverse adversarial datasets
• Incorporation into general multimodal reasoning benchmarks and agent alignment studies

Figure 5: Comparative results for explanation before and after Pun-Tuning, highlighting the enhancement in explanation quality through adversarial fine-tuning.

Conclusion

This work demonstrates neither current large VLMs nor sophisticated prompting is sufficient for truly human-like interpretation of multimodal puns without specialized datasets and carefully constructed reasoning protocols. Reliable multimodal understanding remains contingent on dataset diversity, adversarial robustness, and access to explicit phonetic and semantic bridging mechanisms—well beyond current supervised or alignment-based protocols. MultiPun and the associated methodologies therefore constitute essential assets and baselines for the next generation of cognitively and linguistically capable VLMs.