---
title: 'MelCap: Benchmark for Meme Caption Generation'
url: https://www.emergentmind.com/topics/melcap
type: topic
---

# MelCap: Benchmark for Meme Caption Generation

Searching arXiv for papers related to “MelCap” and adjacent meme captioning / deployment-time profiling terms.
MelCap is best understood, in the available arXiv descriptions, as a meme caption generation benchmark or task in which a system must generate a caption that is congruent with a meme’s visual structure and sentiment. The most directly aligned work, "XMeCap: Meme Caption Generation with Sub-Image Adaptability" [2407.17152], frames the problem as multimodal humor generation rather than generic image captioning, and places particular emphasis on the distinction between single-image memes and multi-image memes with multiple sub-images and a shared caption. Under that framing, MelCap involves not only visual-text matching but also humor semantics, sentiment control, and structure-aware generation.

## 1. Task definition and formal objective

The operative definition given for meme caption generation is: “The meme caption generation task \(G\) aims to produce a caption \(c_i\) for a given image \(m_i\) in a meme.” The objective is written as

$$
\hat{O} = G(m_i), \quad O = \max \sum_{i=1}^{N} S(m_i, C_i, \hat{C}_i).
$$

The intended optimization target is a scoring function comparing generated captions with ground truth. In this formulation, the caption is not evaluated solely for literal descriptiveness; it must fit both the meme image or image set and the humor intent.

Two annotation axes define the task space. The first is **structure**, with the two principal cases being single-image memes and multi-image memes with multiple sub-images and a shared caption. The second is **sentiment**, with four categories: self-praise, praise others, self-mockery, and mock others. A common simplification is to treat meme caption generation as ordinary vision-language captioning. The XMeCap formulation rejects that simplification by making structure and sentiment explicit conditioning variables and by identifying multi-image memes as a distinct benchmark challenge [2407.17152].

## 2. Dataset design and benchmark dimensions

A directly relevant benchmark-style resource is the new Chinese meme dataset introduced alongside XMeCap. Its initial crawl contains **18,110 memes**, and the final balanced dataset after downsampling contains **12,320 memes**. The structure split is **54% single-image** and **46% multi-image**. The sentiment distribution is **21% self-praise**, **23% praise others**, **29% self-mockery**, and **27% mock others**.

The dataset is significant because the work explicitly emphasizes that **multi-image memes had no prior benchmark for caption generation**. That claim is central to the motivation for sub-image-aware modeling. Caption statistics are also reported, including average and maximum token lengths, with the overall average caption length around **16.6 tokens**. This combination of structure labels, sentiment labels, and caption text makes the dataset suitable for evaluating whether a system captures both multimodal alignment and humor style.

The benchmark design also encodes a substantive difficulty difference between the two structure types. For single-image memes, the caption depends on one visual scene. For multi-image memes, the caption must reflect a set of sub-images, each potentially carrying different themes or emotions, while still producing one shared caption. The paper identifies two main challenges in that setting: **effective integration of composite information** and **handling shared captions across sub-images** [2407.17152].

## 3. XMeCap architecture and sub-image adaptability

XMeCap is organized into three stages: **feature extraction**, **adaptive transformation**, and **attention-guided text generation**. Its overall training regime combines **Supervised Fine-Tuning (SFT)**, **Reinforcement Learning (RL)**, and a reward model that factors in both global and local image-text similarity.

In the feature extraction stage, the system explicitly separates single-image and multi-image memes. For multi-image memes, it manually identifies each sub-image and its boundaries using **OpenCV-Python** and ROI selection. It then applies image enhancement independently to each sub-image using **AutoAugment**, including operations such as cropping and rotation. **LLaVA-1.5-7B** serves as the visual encoder for original and augmented sub-images, with the stated goal of capturing detailed visual cues such as morphology and color gradients.

On the text side, multi-image meme text is treated as a **shared caption across all sub-images**. The text pipeline applies **back-translation** and uses **Baichuan2-7B** to enrich textual representation. The system also generates descriptions for each sub-image and converts them into a structured narrative using a **“chain-of-humor”** template consisting of **core concept**, **emotion**, **event**, **consequence**, and **humor element**. The paper explicitly distinguishes this from simple chain-of-thought prompting; the chain-of-humor template is intended to organize humor-relevant semantic cues rather than merely expand intermediate reasoning.

The adaptive transformation stage maps visual and textual features into a shared space through trainable linear projections:

$$
M_i = W_{M_i} \cdot MI_i + b_{M_i}, \quad T_i = W_{T_i} \cdot TI + b_T
$$

Token-level similarity is then computed via the LLM attention mechanism:

$$
E_{l i,j} = (M_i) W^Q \cdot (T_j, W^K)
$$

$$
SI_{ij} = \frac{\exp(E_{l i,j}/\sqrt{d_k})}{\sum_u \exp(E_{l i,u}/\sqrt{d_k})}
$$

where \(W^Q\) and \(W^K\) are query and key matrices, \(d_k\) is the key dimension, and \(SI_{ij}\) is token-wise similarity between image region \(i\) and caption word \(j\). A global attention score is also computed:

$$
SI_{i\cdot j} = \sum_i SI_{ij}
$$

A contrastive objective is applied:

$$
L_I = - \log \frac{\exp(SI_{ij}/t)}{\sum_{k=1}^{N'} \exp(SI_{i,k}/t)}
$$

where \(t\) is the temperature. This objective is intended to pull matched image-caption pairs together and push mismatched ones apart at both token and global levels. The architectural novelty most relevant to MelCap is the explicit handling of **sub-image adaptability**, namely the computation of sub-image-to-text similarity at token and global levels for a shared caption [2407.17152].

## 4. Supervised fine-tuning, reward modeling, and reinforcement learning

The attention-guided text generation stage couples supervised learning with preference optimization. During **SFT**, the LLM is fine-tuned to generate captions close to ground truth while also aligning the generated caption similarity with prior similarity signals from the visual-text alignment stage. The paper introduces KL-divergence-based losses comparing prior similarity \(S_I\) and model-predicted similarity \(S_{SFT}\):

$$
p_{SI} = \frac{\exp(S_I)}{\exp(S_I)+\exp(S_{SFT})}, \quad q_{SI} = \frac{\exp(S_{SFT})}{\exp(S_I)+\exp(S_{SFT})}
$$

and a token-wise version:

$$
p_{SI_{ij}} = \frac{\exp(SI_{ij})}{\exp(SI_{ij})+\exp(S_{T})}, \quad q_{SI_{ij}} = \frac{\exp(S_{T})}{\exp(SI_{ij})+\exp(S_{T})}
$$

The corresponding losses are

$$
L_g = \lambda_g KL(p_{SI} \parallel q_{SI}), \quad L_t = \lambda_t KL(p_{SI_{ij}} \parallel q_{SI_{ij}})
$$

and the total SFT objective is

$$
L_{SFT} = \lambda_{SFT-ori} L_{SFT-ori} + \lambda_g L_g + \lambda_t L_t.
$$

The default weights are reported as **\(\lambda_{SFT-ori} = 0.4\)**, **\(\lambda_g = 0.2\)**, and **\(\lambda_t = 0.4\)**. The paper explicitly states that SFT is more important because it is supervised, and that token-level similarity is especially important for sub-image alignment.

The reward model is built by sampling **1%** of captions generated during SFT, manually ranking them using human evaluation criteria, and requiring agreement above **0.7** among **three volunteers**. Human rankings are combined with attention-based rankings to create final ordered caption sequences. The reward model uses **Baichuan2-7B** as its backbone, with the softmax layer replaced by a **linear layer**, and assigns a scalar score \(r_i = R_i(y, t)\), where \(y\) is the generated caption and \(t\) is the ground truth.

Its pairwise ranking loss is

$$
I = \mathbb{E}_{(x,y_w,y_l)\sim D}\left[\log \sigma(r_e(x,y_w)-r_e(x,y_l))\right],
$$

with \(x\) the original input, \(y_w\) the better caption, \(y_l\) the worse caption, and \(D\) the set of ranking pairs. The human criteria used for ranking are **Informativeness**, **Relevance**, **Creativity**, and **Humorousness**, each scored from **1 to 5** and then scaled to **1–100**.

After SFT, RL further optimizes toward human preference. The RL stage feeds SFT-generated captions into the reward model, maximizes expected reward, and constrains the RL policy to stay close to the SFT model through KL-style regularization. The text notes some formatting inconsistency in the displayed equations, but the intended balance is explicit: reward maximization plus policy preservation. The default RL weights are reported as **0.4** and **0.6** [2407.17152].

## 5. Experimental protocol, metrics, and empirical results

The experiments are run on **4 Nvidia A100 GPUs**, **80GB each**, using **PyTorch + DeepSpeed**, with a maximum sequence length of **1024** and training for **20 epochs**. Because most LMMs are English-pretrained, Chinese captions are translated into English for training and evaluation via **Tencent Cloud API**, and generated outputs are translated back into Chinese for presentation.

The baseline set includes **S2S**, **Dank Learning**, **Transformer**, **MEMEIFY**, **BLIP-2-7B**, **MiniGPT-4-7B**, **InstructBLIP-7B**, **LLaVA-7B**, **Unified-IOXL-2B**, **Shikra-7B**, **Qwen-VL-Chat-7B**, **LLaVA-1.5-7B**, and **GPT-4v**. Evaluation uses two metric groups: meme-specific metrics (**Informativeness**, **Relevance**, **Creativity**, **Humorousness**) and general captioning metrics (**BLEU**, **ROUGE**, **CIDEr**, **METEOR**). Human evaluation uses three volunteers and **Krippendorff’s Alpha**; low-agreement cases below **0.7** are discarded.

The reported XMeCap scores are as follows:

| Metric | Single-image memes | Multi-image memes |
|---|---:|---:|
| Average human evaluation score | 75.85 | 66.32 |
| Informativeness | 83.58 | 76.42 |
| Relevance | 76.77 | 65.77 |
| Creativity | 63.82 | 55.92 |
| Humorousness | 61.17 | 52.4 |
| BLEU | 62.98 | 62.65 |
| ROUGE | 94.87 | 78.11 |
| CIDEr | 97.26 | 93.24 |
| METEOR | 66.31 | 52.02 |

For single-image memes, the detailed synthesis reports a best baseline of **72.14** and a relative gain of **3.71%**. For multi-image memes, it reports a best baseline of around **63.29** and a relative gain of **4.82%**. The abstract, however, states that XMeCap outperforms the best baseline by **6.75%** on single-image memes and **8.56%** on multi-image memes. The coexistence of these figures indicates a reporting discrepancy between summary levels, but not a disagreement about the main result: XMeCap outperforms the strongest baselines in both settings.

Ablation experiments remove **IA** (image augmentation), **TA** (text augmentation), **COH** (chain-of-humor), **A** (attention mechanism), **CL** (contrastive learning), and **RL** (reinforcement learning). The main conclusions are that removing **attention** hurts the most, removing **chain-of-humor** also causes a large drop, removing **contrastive learning** degrades performance notably, and removing **RL** lowers human-preferred caption quality. Image and text augmentation help, but less dramatically. The paper also reports that **GPT-4o** scores correlate with human scores at **0.935**, suggesting that GPT-4o can partially replace manual evaluation [2407.17152].

## 6. Significance for MelCap-style benchmarks

The principal methodological significance for MelCap lies in the treatment of meme caption generation as a joint problem of multimodal alignment and humor control. The strongest contribution is the explicit handling of **single-image and multi-image memes within one framework**, with a shared-caption mechanism for multi-image inputs and token-level as well as global similarity signals for sub-image alignment.

The model also performs better on the negative or contrastive meme types **self-mockery** and **mock others**. This suggests that the learned representation is especially effective for humor forms that rely on contrastive framing rather than simple positive appraisal. The same work further fine-tunes XMeCap on **UR-FUNNY**, **MUSTARD**, and **MHD**, and reports strong performance there as well. A plausible implication is that the representations learned for meme caption generation encode transferable multimodal humor cues rather than only dataset-specific lexical patterns.

For benchmark design, the XMeCap evidence supports at least four principles. First, a MelCap-style benchmark should distinguish **single-image** from **multi-image** cases rather than collapsing them into one captioning pool. Second, sentiment labels such as **self-praise**, **praise others**, **self-mockery**, and **mock others** are useful because the caption must fit humorous tone as well as content. Third, structured intermediate representations such as **chain-of-humor** can be operationally useful because they organize humor-relevant semantic cues. Fourth, evaluation benefits from combining conventional caption metrics with meme-specific criteria such as **Informativeness**, **Relevance**, **Creativity**, and **Humorousness** [2407.17152].

## 7. Terminological ambiguity: MelCap and MCAP

A plausible source of confusion is the proximity between **MelCap** and **MCAP** in recent arXiv discourse. MCAP, in "MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference" [2604.21026], is unrelated to meme caption generation. It denotes **Monte Carlo Activation Profiling**, a **load-time, gradient-free, weight-free per-layer importance estimator** for memory-constrained LLM inference. MCAP profiles activation magnitudes over **12 calibration prompts**, produces a lightweight per-layer signal, and uses that signal to drive both **precision dispatch**—high-importance layers to **W4A16**, low-importance layers to **W4A8**—and **residency tier assignment** across **GPU**, **CPU RAM**, and **SSD**.

The distinction is substantive. MelCap, as inferred from the meme-captioning literature, concerns multimodal humor generation, dataset construction, sentiment-aware captioning, and sub-image adaptability. MCAP concerns hardware-adaptive deployment, layer importance estimation, dynamic precision routing, and virtual weight paging. The overlap is only orthographic. Treating them as interchangeable would collapse two different research programs: one in multimodal meme understanding and generation, the other in systems optimization for large language model inference.

Source: https://www.emergentmind.com/topics/melcap