Multimodal Sarcasm Analysis

• Multimodal Sarcasm Analysis is a field that detects sarcasm by integrating text, image, audio, and video to reveal mismatches between literal and intended meanings.
• It employs methods such as feature fusion, contrastive attention, and cross-modal reasoning, achieving high accuracy on benchmarks like MMSD2.0 and MUStARD.
• Research in MSA addresses annotation biases, subjective judgment, and multilingual challenges, paving the way for uncertainty-aware and robust multimodal models.

Multimodal Sarcasm Analysis (MSA) studies how sarcastic intent is recognized, explained, or localized when meaning is distributed across text, image, audio, and video rather than conveyed by language alone. In this literature, the same general area also appears under the names Multimodal Sarcasm Detection (MSD) and Multimodal Sarcasm Understanding (MSU). Across these formulations, sarcasm is typically characterized as a disparity between literal and intended meaning, with multimodal instances introducing cross-modal incongruity such as positive language paired with negative visual, acoustic, or situational evidence (Farabi et al., 2024). Early multimodal work already showed that images are often indispensable for human judgment: in a crowdsourcing study over Instagram and Tumblr, text alone was insufficient for 62.20% and 76.40% of posts respectively, and when images were added, 60.13% and 58.25% of those cases became sarcastic for annotators (Schifanella et al., 2016).

1. Scope, formalization, and task structure

MSA is most often instantiated as binary classification over a multimodal input. In the text–image setting, one common formulation is MSD: given a sample $s^i = (\mathcal{T}^i, I^i, Y^i)$, a model $\mathcal{F}$ predicts $\hat{Y}^i = \mathcal{F}(\mathcal{T}^i, I^i)$ (Wang et al., 5 Aug 2025). In benchmark descriptions centered on social-media posts, the label space is usually $y \in \{0,1\}$, where 1 denotes sarcasm and 0 denotes non-sarcasm; the positive class is typically associated with text–image incongruity, lexical polarity reversal, or pragmatic mismatch (Qin et al., 2023). In audiovisual dialogue, the input may additionally include aligned speech, frames, speaker identity, and dialogue history, and models may predict sarcasm per utterance rather than per post (Zhang et al., 2021).

The field has expanded beyond detection into explanation and localization. MuSE, or Multimodal Sarcasm Explanation, takes a sarcastic post $P=(I,T)$ and generates a natural-language explanation $E=[e_1,\dots,e_p]$ of the underlying irony (Desai et al., 2021). Sarcasm Explanation in Dialogue (SED) extends this idea to multi-party, code-mixed conversational settings with audio-video context (Kumar et al., 2022). Document-level MSU further broadens the problem by treating sarcasm understanding as both document-level detection and localization of textual spans and visual regions in long-form news (Du et al., 2023).

Evaluation remains dominated by standard classification metrics. For binary detection, the literature commonly reports

$$Acc = \frac{TP + TN}{TP + TN + FP + FN}, \quad P = \frac{TP}{TP + FP}, \quad R = \frac{TP}{TP + FN}, \quad F1 = \frac{2PR}{P + R},$$

with macro-F1 also used when class balance or per-class averaging is emphasized (Qin et al., 2023). Generative tasks instead use BLEU, ROUGE, METEOR, BERTScore, or human adequacy and fluency judgments (Desai et al., 2021).

2. Benchmarks, modalities, and annotation regimes

The empirical landscape of MSA is organized around several benchmark families that differ sharply in modality, domain, and supervision.

Benchmark family	Modalities	Distinctive property
MMSD / MMSD2.0	Text + image	Twitter posts; MMSD2.0 removes spurious cues and corrects labels
MORE	Image + caption + explanation	3,510 sarcastic posts with English explanations
MUStARD / MUStARD++ / Balanced	Text + audio + video	Sitcom dialogue clips with conversational context
MaSaC / WITS	Text + audio + video	Hindi-English code-mixed dialogue and explanation
Spanish audiovisual dataset	Text + audio + video	First multimodal sarcasm dataset for Spanish
DocMSU	Document text + image	102,588 news items with detection and localization

MMSD and MMSD2.0 are the dominant text–image benchmarks. MMSD2.0 preserves the original split sizes of 19,816 training, 2,410 validation, and 2,409 test samples, but it removes hashtag and emoji shortcuts and re-annotates unreasonable negatives, making it more balanced and explicitly reliability-oriented (Qin et al., 2023). MORE contributes 3,510 image–caption–explanation triples for MuSE, with splits of 2,983/175/352 and separate OCR and non-OCR subsets (Desai et al., 2021). DocMSU shifts the field from sentence-level posts to document-level news, with 102,588 text–image pairs across nine topics and additional supervision in the form of textual spans and visual bounding boxes marking sarcasm clues (Du et al., 2023).

Audiovisual resources center on conversation. MUStARD contains 690 balanced tri-modal samples, while MUStARD++ expands this space with 1,202 balanced clips and richer annotations such as sarcasm type, emotion, valence, and arousal; MUStARD++ Balanced further expands the corpus to 1,365 clips by adding House MD examples to reduce sarcasm-type skew (Bhosale et al., 2023). MaSaC contributes more than 15,000 utterances across approximately 1,200 dialogs from a Hindi-English sitcom setting, with sarcasm and humor labels and aligned text and audio; WITS then layers explanation generation on top of this conversational regime, yielding 2,240 multi-party dialogues with code-mixed explanations (Bedi et al., 2021). In Spanish, the first aligned text-audio-video dataset was built from dubbed episodes of South Park and Archer, with utterance-level alignment and two dialectal varieties, Latin American and Peninsular Spanish (Alnajjar et al., 2021).

Annotation schemes differ substantially. MMSD2.0 uses trained human annotators, an onboarding set of 100 curated samples with at least 85% agreement against expert labels, and a 1,000-sample double check yielding Cohen’s $\kappa = 0.811$, reported as “almost perfect agreement” (Qin et al., 2023). MUStARD++ Balanced reports Fleiss’ $\kappa = 0.743$ and Krippendorff’s $\alpha = 0.798$ for its added clips (Bhosale et al., 2023). DocMSU instead uses IoU-based confidence scoring over span and box annotations and escalates the most difficult 5% of samples for re-labeling (Du et al., 2023). These differences are consequential, because MSA error profiles are often entangled with annotation ambiguity rather than architecture alone.

3. Core modeling paradigms

The earliest multimodal sarcasm systems treated the problem as feature fusion. One influential line concatenated textual features with visual semantics features derived from externally trained concept detectors or with CNN image embeddings, showing consistent gains over text-only models on Instagram and Tumblr (Schifanella et al., 2016). In Spanish audiovisual sarcasm detection, early concatenation of sentence embeddings, acoustic descriptors from librosa, and frame-level ResNet-152 features, followed by an RBF-kernel SVM, produced 93.1% accuracy when all three modalities were combined, surpassing text-only and text+audio settings (Alnajjar et al., 2021).

Subsequent work shifted from raw fusion to explicit incongruity modeling. ConAttSD, proposed for video conversations, combines a sequential context encoder with inter-modality contrastive attention, producing contrastive features $\mathcal{F}$0 and $\mathcal{F}$1 that encode disagreement between text and audio or vision; on MUSTARD, it achieved $\mathcal{F}$2, improving over a GRU-based encoder and an earlier two-attention baseline (Zhang et al., 2021). Hierarchical congruity modeling for Twitter sarcasm formalized sarcasm as low cross-modal congruity, using multi-head cross-attention for atomic token–patch alignment and GATs for composition-level congruity; with image captions as knowledge, the model reached 88.26 accuracy and 84.84 F1 on MMSD (Liu et al., 2022). FiLM-based models further conditioned image features on text through FiLMed ResNet blocks while separately using bilinear co-attention over text and image attributes; one such system reported 0.9219 F1 and 0.9366 accuracy on the public Twitter dataset (Gupta et al., 2021).

Conversational and code-mixed settings required context-sensitive multimodal encoders. MSH-COMICS used hierarchical attention over words, dialog-level contextual attention over preceding utterances, and a cross-modal filter for text and audio, obtaining sarcasm $\mathcal{F}$3 and humor $\mathcal{F}$4 on MaSaC (Bedi et al., 2021). For explanation generation in dialogues, MAF introduced Modality Aware Fusion with Multimodal Context-Aware Attention and Global Information Fusion adapters inside BART or mBART, with the best MAF-TAV configuration reaching ROUGE-1/2/L of 39.69/17.10/37.37 and BLEU-4 of 8.58 on WITS (Kumar et al., 2022). In text–audio sarcasm detection on MUStARD, AMuSeD combined back-translation, synthetic speech augmentation, and self-attention fusion, reaching an F1-score of 81.0% using only text and audio (Gao et al., 2024).

Recent text–image models have become increasingly CLIP-centric. Multi-view CLIP constructs text, image, and text–image interaction views on top of CLIP ViT-B/32, with late fusion over three prediction heads and joint BCE loss $\mathcal{F}$5; on MMSD2.0 it achieved 85.64 accuracy and 84.10 F1 (Qin et al., 2023). InterCLIP-MEP instead injects cross-modal information directly into top CLIP self-attention layers via conditional attention and LoRA, then adds a dynamic dual-channel memory at inference time; its best T2V configuration reported 86.72 accuracy and 85.61 F1 on MMSD2.0 (Chen et al., 2024). RCLMuFN combined four feature extractors—CLIP image, ResNet-50, CLIP text, and BERT—with relational context learning and multiplex fusion, reporting 91.57 accuracy and 90.25 F1 on MMSD2.0 (Wang et al., 2024).

4. Reliability, bias, and the problem of subjective judgment

A central development in MSA has been the recognition that many strong reported results were artifacts of benchmark bias rather than robust multimodal understanding. The clearest case is MMSD. On the original benchmark, a text-only RoBERTa surpassed the best multimodal baseline HKE by 6.6% F1, with RoBERTa at 92.45 F1 and HKE at 84.09, indicating that models were exploiting textual artifacts rather than genuine text–image reasoning (Qin et al., 2023). The diagnosed artifacts were concrete: average hashtag counts in MMSD were 1.90/1.63/1.93 for positive train/validation/test samples versus 0.55/0.64/0.70 for negatives, 80.7% of emoji tokens appeared only in one class, and the absence of “#sarcasm” was effectively treated as evidence for the negative class (Qin et al., 2023).

MMSD2.0 addresses these issues by removing all hashtag and emoji tokens from the text and re-annotating all negative samples under guidelines that emphasize sarcastic intention from multimodal incongruity rather than explicit tags (Qin et al., 2023). The resulting drop in performance is itself diagnostic: RoBERTa’s F1 falls from 92.45 to 76.21, while Multi-view CLIP remains comparatively strong at 84.10 F1, suggesting that debiasing changes what the benchmark measures (Qin et al., 2023). This reliability agenda has become one of the field’s central methodological themes.

A different but related problem is subjectivity. Large vision-LLM analyses show that sarcasm perception varies not only across models but within the same model under different prompt framings. On the 2,409-sample MMSD2.0 test set, classification-oriented prompts were more internally consistent than interpretive prompts, neutral predictions were sparse and poorly overlapping across models, and literal readings tended to be the more stable default (Chen et al., 15 Mar 2025). Aggregate voting across 12 LVLMs produced 73.01% accuracy for binary sarcasm classification and 75.70% accuracy for ternary classification over non-neutral predictions, but the same study argued that rigid binary labels can obscure ambiguous cases for which sarcastic and literal readings are both defensible (Chen et al., 15 Mar 2025). This suggests that benchmark reliability and interpretive uncertainty are distinct but interacting axes of evaluation.

5. From detection to explanation, localization, and long-context understanding

One of the most important conceptual extensions of MSA has been the shift from deciding whether a post is sarcastic to explaining why. MORE formalized this as MuSE and paired 3,510 sarcastic multimodal posts with natural-language English explanations (Desai et al., 2021). Its ExMore baseline used a multimodal Transformer encoder with text-to-vision cross-attention and a BART-based autoregressive decoder. On the overall test set, ExMore exceeded the strongest baseline M-Transf on every reported automatic metric, including BLEU-4 4.26 versus 1.67 implied by the reported +2.59 gain, ROUGE-L 25.23 versus 18.77, METEOR 19.16 versus 15.06, and BERTScore F1 87.9 versus 86.2, while human adequacy rose from 0.37 to 0.69 (Desai et al., 2021).

Dialogue explanation adds additional discourse structure. WITS defines SED over multi-party Hindi-English code-mixed conversations where the last utterance is sarcastic and the output explanation must specify source, target, action word, and optional description (Kumar et al., 2022). The dataset contains 2,240 dialogues and 9,080 utterances, with 7,526 code-mixed utterances and an average of 4.05 utterances per dialogue (Kumar et al., 2022). MAF-TAV improved source identification markedly, reaching 91.07 accuracy versus 77.23 for text-only BART, although target accuracy remained lower at 46.42 (Kumar et al., 2022). This imbalance indicates that identifying who is sarcastic and identifying whom or what the sarcasm targets are separable problems.

Document-level MSU extends the scope further. DocMSU contains 102,588 news items with text-image pairs across nine topics, averages 63 tokens and 5 sentences per sample, and annotates both detection labels and sarcasm clues via text spans and image boxes (Du et al., 2023). Its fine-grained model aligns a square tensor derived from BERT token representations with sliding image windows before Swin-Transformer fusion and YOLOX-based localization. The strongest reported detection configuration reached 97.83 accuracy and 87.25 F1, while image localization achieved AP50 35.29 and text localization reached EM$\mathcal{F}$6 52.19 (Du et al., 2023). A plausible implication is that once MSA moves to news-scale documents, localization becomes as central as classification, because sarcasm cues are sparse and rarely coincide with the entire document.

6. Foundation models, external knowledge, and reasoning-oriented MSA

The rise of CLIP, LVLMs, and multimodal LLMs has shifted MSA toward explicit reasoning and knowledge integration. A 2025 study evaluated LLaVA, MiniGPT, InstructBLIP, and GPT-4o-mini in zero-shot settings on both MSD and explanation, finding weak baseline performance: on MSD, LLaVA, MiniGPT, and InstructBLIP scored 41.1/57.4, 44.2/54.5, and 42.5/55.2 in accuracy/F1, while GPT-4o reached 65.9/68.6 (Wang et al., 5 Aug 2025). The same work identified limited visual grounding, conceptual knowledge gaps, and weak object–text alignment as key failure modes, then improved all four models with a training-free pipeline that adds Fast R-CNN object extraction and ConceptNet-derived conceptual links to the prompt; GPT-4o rose to 75.3 accuracy and 72.1 F1 on detection, and explanation metrics improved significantly across all models (Wang et al., 5 Aug 2025).

Reasoning-oriented prompting has also been formalized more explicitly. IRONIC proposes a two-step zero-shot framework in which an MLLM first predicts a Multi-modal Coherence Relation—Insertion, Concretization, Projection, Restatement, or Extension—and then uses that coherence analysis for sarcasm classification (Ramakrishnan et al., 22 May 2025). With GPT-4o on MMSD2.0, IRONIC reported 0.7659 accuracy and 0.7670 weighted F1, outperforming both vanilla zero-shot and zero-shot CoT prompting; on the out-of-domain RedEval benchmark it reached 0.8406 accuracy and 0.8422 F1 (Ramakrishnan et al., 22 May 2025). MiDRE pushes this line into supervised modeling by mixing an internal reasoning expert over the original image–text pair with an external reasoning expert conditioned on image-only, text-only, and joint rationales generated by Qwen2-VL-7B-Instruct. On MMSD2.0, MiDRE reached 86.18 accuracy and 87.79 F1, compared with 84.00 F1 for its reported best baseline (Jana et al., 6 Jul 2025).

These developments indicate a methodological shift. Earlier MSA systems mostly learned fused representations and classifiers; newer systems increasingly attempt to model explicit rationales, structured external context, and modular reasoning paths. At the same time, papers in this area repeatedly note that noisy rationales, hallucinated objects, or ungrounded cultural references remain unresolved sources of error (Jana et al., 6 Jul 2025).

7. Open problems and future directions

Several open problems recur across the literature. The first is commonsense and cultural dependence. Explanation models report failures when sarcasm hinges on handicapped parking norms, speed-test conventions, hyperbole, or other world knowledge absent from the input pair (Desai et al., 2021). CLIP-based and reasoning-based detectors likewise note that broader and more diverse corpora, external commonsense knowledge, and evaluation protocols that probe “right-for-the-right-reasons” behavior remain necessary (Qin et al., 2023).

The second is modality coverage and cross-lingual breadth. The field now spans English social-media posts, Spanish audiovisual sarcasm, Hindi-English code-mixed dialogue, and Chinese or multilingual resources noted in survey work, but cross-cultural and multilingual coverage remains limited, especially for speech-based sarcasm where prosodic cues vary across languages (Alnajjar et al., 2021). A later systematic review emphasized that sarcasm in speech is underrepresented, that most corpora are small and TV-based, and that multilingual, spontaneous, multimodal datasets with standardized splits and richer labels are still needed (Gao et al., 4 Sep 2025).

The third is uncertainty. LVLM studies argue that MSA should move beyond rigid binaries toward multi-perspective and uncertainty-aware modeling, because confidence, prompt framing, and neutral interpretations vary sharply across models and samples (Chen et al., 15 Mar 2025). This does not invalidate binary detection tasks, but it suggests that benchmark design may need soft labels, rater distributions, or explicit neutral options in addition to standard sarcastic/non-sarcastic supervision.

Finally, there is a continuing architectural tension between task-specific supervised models and general multimodal foundation models. Task-specific systems such as Multi-view CLIP, InterCLIP-MEP, MiDRE, and RCLMuFN remain very strong on curated benchmarks (Qin et al., 2023), while zero-shot LVLMs still exhibit grounding and reasoning deficits (Wang et al., 5 Aug 2025). A plausible implication is that future MSA systems will increasingly combine both traditions: reliable debiased benchmarks, explicit grounding and rationales, parameter-efficient adaptation of foundation models, and evaluation protocols that measure not only accuracy but explanation quality, localization fidelity, robustness to shortcut cues, and sensitivity to ambiguity.