MelCap: Benchmark for Meme Caption Generation

• MelCap is a multimodal benchmark that defines meme captioning as generating humorous, sentiment-controlled captions tailored for both single-image and multi-image memes.
• It employs a structured dataset with explicit sentiment labels and sub-image adaptability, using methods like chain-of-humor representations and contrastive learning.
• The evaluation combines human assessments with traditional caption metrics to validate improved performance in informativeness, relevance, and creativity.

Searching arXiv for papers related to “MelCap” and adjacent meme captioning / deployment-time profiling terms. MelCap is best understood, in the available arXiv descriptions, as a meme caption generation benchmark or task in which a system must generate a caption that is congruent with a meme’s visual structure and sentiment. The most directly aligned work, "XMeCap: Meme Caption Generation with Sub-Image Adaptability" (Chen et al., 2024), frames the problem as multimodal humor generation rather than generic image captioning, and places particular emphasis on the distinction between single-image memes and multi-image memes with multiple sub-images and a shared caption. Under that framing, MelCap involves not only visual-text matching but also humor semantics, sentiment control, and structure-aware generation.

1. Task definition and formal objective

The operative definition given for meme caption generation is: “The meme caption generation task $G$ aims to produce a caption $c_i$ for a given image $m_i$ in a meme.” The objective is written as

$$\hat{O} = G(m_i), \quad O = \max \sum_{i=1}^{N} S(m_i, C_i, \hat{C}_i).$$

The intended optimization target is a scoring function comparing generated captions with ground truth. In this formulation, the caption is not evaluated solely for literal descriptiveness; it must fit both the meme image or image set and the humor intent.

Two annotation axes define the task space. The first is structure, with the two principal cases being single-image memes and multi-image memes with multiple sub-images and a shared caption. The second is sentiment, with four categories: self-praise, praise others, self-mockery, and mock others. A common simplification is to treat meme caption generation as ordinary vision-language captioning. The XMeCap formulation rejects that simplification by making structure and sentiment explicit conditioning variables and by identifying multi-image memes as a distinct benchmark challenge (Chen et al., 2024).

2. Dataset design and benchmark dimensions

A directly relevant benchmark-style resource is the new Chinese meme dataset introduced alongside XMeCap. Its initial crawl contains 18,110 memes, and the final balanced dataset after downsampling contains 12,320 memes. The structure split is 54% single-image and 46% multi-image. The sentiment distribution is 21% self-praise, 23% praise others, 29% self-mockery, and 27% mock others.

The dataset is significant because the work explicitly emphasizes that multi-image memes had no prior benchmark for caption generation. That claim is central to the motivation for sub-image-aware modeling. Caption statistics are also reported, including average and maximum token lengths, with the overall average caption length around 16.6 tokens. This combination of structure labels, sentiment labels, and caption text makes the dataset suitable for evaluating whether a system captures both multimodal alignment and humor style.

The benchmark design also encodes a substantive difficulty difference between the two structure types. For single-image memes, the caption depends on one visual scene. For multi-image memes, the caption must reflect a set of sub-images, each potentially carrying different themes or emotions, while still producing one shared caption. The paper identifies two main challenges in that setting: effective integration of composite information and handling shared captions across sub-images (Chen et al., 2024).

3. XMeCap architecture and sub-image adaptability

XMeCap is organized into three stages: feature extraction, adaptive transformation, and attention-guided text generation. Its overall training regime combines Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), and a reward model that factors in both global and local image-text similarity.

In the feature extraction stage, the system explicitly separates single-image and multi-image memes. For multi-image memes, it manually identifies each sub-image and its boundaries using OpenCV-Python and ROI selection. It then applies image enhancement independently to each sub-image using AutoAugment, including operations such as cropping and rotation. LLaVA-1.5-7B serves as the visual encoder for original and augmented sub-images, with the stated goal of capturing detailed visual cues such as morphology and color gradients.

On the text side, multi-image meme text is treated as a shared caption across all sub-images. The text pipeline applies back-translation and uses Baichuan2-7B to enrich textual representation. The system also generates descriptions for each sub-image and converts them into a structured narrative using a “chain-of-humor” template consisting of core concept, emotion, event, consequence, and humor element. The paper explicitly distinguishes this from simple chain-of-thought prompting; the chain-of-humor template is intended to organize humor-relevant semantic cues rather than merely expand intermediate reasoning.

The adaptive transformation stage maps visual and textual features into a shared space through trainable linear projections:

$$M_i = W_{M_i} \cdot MI_i + b_{M_i}, \quad T_i = W_{T_i} \cdot TI + b_T$$

Token-level similarity is then computed via the LLM attention mechanism:

$E_{l i,j} = (M_i) W^Q \cdot (T_j, W^K)$

$$SI_{ij} = \frac{\exp(E_{l i,j}/\sqrt{d_k})}{\sum_u \exp(E_{l i,u}/\sqrt{d_k})}$$

where $W^Q$ and $W^K$ are query and key matrices, $d_k$ is the key dimension, and $c_i$0 is token-wise similarity between image region $c_i$1 and caption word $c_i$2. A global attention score is also computed:

$c_i$3

A contrastive objective is applied:

$c_i$4

where $c_i$5 is the temperature. This objective is intended to pull matched image-caption pairs together and push mismatched ones apart at both token and global levels. The architectural novelty most relevant to MelCap is the explicit handling of sub-image adaptability, namely the computation of sub-image-to-text similarity at token and global levels for a shared caption (Chen et al., 2024).

4. Supervised fine-tuning, reward modeling, and reinforcement learning

The attention-guided text generation stage couples supervised learning with preference optimization. During SFT, the LLM is fine-tuned to generate captions close to ground truth while also aligning the generated caption similarity with prior similarity signals from the visual-text alignment stage. The paper introduces KL-divergence-based losses comparing prior similarity $c_i$6 and model-predicted similarity $c_i$7:

$c_i$8

and a token-wise version:

$c_i$9

The corresponding losses are

$m_i$0

and the total SFT objective is

$m_i$1

The default weights are reported as $m_i$2, $m_i$3, and $m_i$4. The paper explicitly states that SFT is more important because it is supervised, and that token-level similarity is especially important for sub-image alignment.

The reward model is built by sampling 1% of captions generated during SFT, manually ranking them using human evaluation criteria, and requiring agreement above 0.7 among three volunteers. Human rankings are combined with attention-based rankings to create final ordered caption sequences. The reward model uses Baichuan2-7B as its backbone, with the softmax layer replaced by a linear layer, and assigns a scalar score $m_i$5, where $m_i$6 is the generated caption and $m_i$7 is the ground truth.

Its pairwise ranking loss is

$m_i$8

with $m_i$9 the original input, $$\hat{O} = G(m_i), \quad O = \max \sum_{i=1}^{N} S(m_i, C_i, \hat{C}_i).$$0 the better caption, $$\hat{O} = G(m_i), \quad O = \max \sum_{i=1}^{N} S(m_i, C_i, \hat{C}_i).$$1 the worse caption, and $$\hat{O} = G(m_i), \quad O = \max \sum_{i=1}^{N} S(m_i, C_i, \hat{C}_i).$$2 the set of ranking pairs. The human criteria used for ranking are Informativeness, Relevance, Creativity, and Humorousness, each scored from 1 to 5 and then scaled to 1–100.

After SFT, RL further optimizes toward human preference. The RL stage feeds SFT-generated captions into the reward model, maximizes expected reward, and constrains the RL policy to stay close to the SFT model through KL-style regularization. The text notes some formatting inconsistency in the displayed equations, but the intended balance is explicit: reward maximization plus policy preservation. The default RL weights are reported as 0.4 and 0.6 (Chen et al., 2024).

5. Experimental protocol, metrics, and empirical results

The experiments are run on 4 Nvidia A100 GPUs, 80GB each, using PyTorch + DeepSpeed, with a maximum sequence length of 1024 and training for 20 epochs. Because most LMMs are English-pretrained, Chinese captions are translated into English for training and evaluation via Tencent Cloud API, and generated outputs are translated back into Chinese for presentation.

The baseline set includes S2S, Dank Learning, Transformer, MEMEIFY, BLIP-2-7B, MiniGPT-4-7B, InstructBLIP-7B, LLaVA-7B, Unified-IOXL-2B, Shikra-7B, Qwen-VL-Chat-7B, LLaVA-1.5-7B, and GPT-4v. Evaluation uses two metric groups: meme-specific metrics (Informativeness, Relevance, Creativity, Humorousness) and general captioning metrics (BLEU, ROUGE, CIDEr, METEOR). Human evaluation uses three volunteers and Krippendorff’s Alpha; low-agreement cases below 0.7 are discarded.

The reported XMeCap scores are as follows:

Metric	Single-image memes	Multi-image memes
Average human evaluation score	75.85	66.32
Informativeness	83.58	76.42
Relevance	76.77	65.77
Creativity	63.82	55.92
Humorousness	61.17	52.4
BLEU	62.98	62.65
ROUGE	94.87	78.11
CIDEr	97.26	93.24
METEOR	66.31	52.02

For single-image memes, the detailed synthesis reports a best baseline of 72.14 and a relative gain of 3.71%. For multi-image memes, it reports a best baseline of around 63.29 and a relative gain of 4.82%. The abstract, however, states that XMeCap outperforms the best baseline by 6.75% on single-image memes and 8.56% on multi-image memes. The coexistence of these figures indicates a reporting discrepancy between summary levels, but not a disagreement about the main result: XMeCap outperforms the strongest baselines in both settings.

Ablation experiments remove IA (image augmentation), TA (text augmentation), COH (chain-of-humor), A (attention mechanism), CL (contrastive learning), and RL (reinforcement learning). The main conclusions are that removing attention hurts the most, removing chain-of-humor also causes a large drop, removing contrastive learning degrades performance notably, and removing RL lowers human-preferred caption quality. Image and text augmentation help, but less dramatically. The paper also reports that GPT-4o scores correlate with human scores at 0.935, suggesting that GPT-4o can partially replace manual evaluation (Chen et al., 2024).

6. Significance for MelCap-style benchmarks

The principal methodological significance for MelCap lies in the treatment of meme caption generation as a joint problem of multimodal alignment and humor control. The strongest contribution is the explicit handling of single-image and multi-image memes within one framework, with a shared-caption mechanism for multi-image inputs and token-level as well as global similarity signals for sub-image alignment.

The model also performs better on the negative or contrastive meme types self-mockery and mock others. This suggests that the learned representation is especially effective for humor forms that rely on contrastive framing rather than simple positive appraisal. The same work further fine-tunes XMeCap on UR-FUNNY, MUSTARD, and MHD, and reports strong performance there as well. A plausible implication is that the representations learned for meme caption generation encode transferable multimodal humor cues rather than only dataset-specific lexical patterns.

For benchmark design, the XMeCap evidence supports at least four principles. First, a MelCap-style benchmark should distinguish single-image from multi-image cases rather than collapsing them into one captioning pool. Second, sentiment labels such as self-praise, praise others, self-mockery, and mock others are useful because the caption must fit humorous tone as well as content. Third, structured intermediate representations such as chain-of-humor can be operationally useful because they organize humor-relevant semantic cues. Fourth, evaluation benefits from combining conventional caption metrics with meme-specific criteria such as Informativeness, Relevance, Creativity, and Humorousness (Chen et al., 2024).

7. Terminological ambiguity: MelCap and MCAP

A plausible source of confusion is the proximity between MelCap and MCAP in recent arXiv discourse. MCAP, in "MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference" (Das, 22 Apr 2026), is unrelated to meme caption generation. It denotes Monte Carlo Activation Profiling, a load-time, gradient-free, weight-free per-layer importance estimator for memory-constrained LLM inference. MCAP profiles activation magnitudes over 12 calibration prompts, produces a lightweight per-layer signal, and uses that signal to drive both precision dispatch—high-importance layers to W4A16, low-importance layers to W4A8—and residency tier assignment across GPU, CPU RAM, and SSD.

The distinction is substantive. MelCap, as inferred from the meme-captioning literature, concerns multimodal humor generation, dataset construction, sentiment-aware captioning, and sub-image adaptability. MCAP concerns hardware-adaptive deployment, layer importance estimation, dynamic precision routing, and virtual weight paging. The overlap is only orthographic. Treating them as interchangeable would collapse two different research programs: one in multimodal meme understanding and generation, the other in systems optimization for LLM inference.