---
title: 'AVMeme Exam: Multimodal Meme Benchmark'
url: https://www.emergentmind.com/topics/avmeme-exam
type: topic
---

# AVMeme Exam: Multimodal Meme Benchmark

AVMeme Exam is a human-curated benchmark developed for evaluating the contextual and cultural understanding capabilities of multimodal large language models (MLLMs) using iconic internet audio-visual memes. It is designed to assess surface, contextual, emotional, and pragmatic comprehension across a broad spectrum of linguistic, auditory, and cultural domains [2601.17645].

## 1. Dataset Structure and Composition

The AVMeme Exam dataset consists of 1,032 ("meme-full") rigorously curated audio-visual clips, selected to probe a diverse range of multimodal understanding tasks. After removal of "text-cheat" items—memes whose content is fully recoverable by text-only models—the primary evaluation split ("meme-main") contains 846 items.

Each meme is categorized into one of four audio-centric modalities, with approximate proportions:

| Modality                    | Proportion of N (%) | Description                                   |
|-----------------------------|--------------------|-----------------------------------------------|
| Speech (Sp)                 | ~35                | Spoken language clips                         |
| Songs (So)                  | ~26                | Clips with lyrics or singing                  |
| Music (Mu, textless)        | ~23                | Nonverbal instrumental music                  |
| Sound Effects (Sfx)         | ~16                | Isolated non-musical audio cues               |

Memes span over ten languages, including English, Mandarin Chinese, Japanese, Korean, Persian (Farsi), and several less-common tongues, reflecting contributions from researchers with backgrounds rooted in the U.S., China, Japan, India, the Middle East, and elsewhere. Five major global regions are covered: North America, Europe, Middle East, East Asia, and South Asia.

## 2. Question and Answer Design Framework

Every clip in AVMeme Exam is paired with a single multiple-choice question (MCQ), curated to assess a specific dimension of multimodal understanding. Each question is independently labeled by human verifiers along one of seven understanding levels:

| Level    | Description (with example)                                            |
|----------|-----------------------------------------------------------------------|
| Audio Analysis (A)       | Surface-level audio features ("What processing was applied to the voice?")         |
| Language Analysis (L)    | Literal linguistic content ("What does the speaker claim about himself?")           |
| Contextual Inference (C) | Pragmatic context ("Which situation matches the tone and intent?")                  |
| Emotion Analysis (E)     | Affect and feeling ("What will listeners feel on hearing this?")                    |
| Humor/Popularity (H)     | Social/cultural signal ("Which is not a reason the clip is humorous?")              |
| Usage/Application (U)    | Norms/pragmatic use ("When is this meme typically used?")                           |
| World Knowledge (W)      | External facts ("Who performs the original version of this track?")                 |

Each MCQ comprises 4–6 randomly shuffled options. Model and human performance is measured per category as the fraction of correct responses:
\[
\text{acc}_i = \frac{1}{N_i}\sum_{j=1}^{N_i} \mathbb{1}[\hat{y}_j = y_j], \quad
A_m = \sum_i p_i \,\text{acc}_i
\]
where $N_i$ is the count of items in category $i$ and $p_i$ is its proportion.

## 3. Metadata Annotation and Signal Control

Each meme is extensively annotated with standardized metadata:

- Source URL, onset/offset timestamps, and year of origin
- Language and sound category (Sp, So, Mu, Sfx)
- Transcript (when applicable)
- Concise human-written summary
- Typical usage description
- Emotion tags (e.g., happy, ironic, nostalgic)
- Sensitivity tags (sex, violence, race/gender, etc.)
- Visual-hint tags (no text, transcription subtitle, title/name, visual_cheat)

Metadata guides question construction and enables analysis stratified by year, sensitivity, and visual hinting. Strict controls are implemented to prevent leakage of explicit textual cues; ablation studies indicate that presence of meme names or subtitles (“visual_cheat”) can inflate model accuracy by 10–40%.

## 4. Evaluation Protocol and Metrics

Evaluation involves both SOTA MLLMs and human baselines. Models are partitioned into:

- Audio-only LLMs (SALMONN, Qwen2-Audio, Audio Reasoner, Kimi-Audio, Audio Flamingo 3, Step-Audio 2-mini, MiDashengLM, MiMo-Audio, Music Flamingo, GPT-4o Audio)
- Audio-visual LLMs (Baichuan-Omni, MiniCPM, Phi-4-Multimodal, Qwen2.5-Omni, Gemma 3n, Qwen3-Omni, Gemini 2.5 Flash, Gemini 3 Flash, Gemini 3 Pro)

Twenty native speakers (10 English, 10 Chinese) serve as the human baseline, each answering ~37–38 items, filtered by language and format.

Audio is standardized to mono 16 kHz WAV (≤30s); video is converted to 360p @1fps MP4. MCQ options are formatted for randomized presentation. Model input strictly uses raw byte files to prevent meta-data leakage.

Primary metric is accuracy; standard metrics (precision, recall, F1) are acknowledged though not emphasized. Human performance is further stratified by prior exposure ("familiar" vs. "novice" items).

A paired t-test is prescribed for significance testing between accuracy vectors across N clips:
\[
t = \frac{\bar{d}}{s_d/\sqrt{N}}, \quad d_i = \text{acc}_i^h - \text{acc}_i^m
\]
where $s_d$ is the sample standard deviation of differences.

## 5. Empirical Results and Observations

Performance is assessed across question type, neural modality, language, and human/model comparison.

- **By Question Category:** Surface-level language tasks (L) achieve >90% with top models. Audio analysis (A) drops to ~50–60%. Deep context (C), emotion (E), usage (U), humor (H), and world knowledge (W) are substantially more difficult, with accuracies spanning 20–70%. Gemini 3 Pro achieves the highest overall: 80.0% on "meme-main" with visual input.
- **Modality and Language Effects:** Speech and song yield the highest accuracies (70–90%), while instrumental music and sound effects remain problematic (30–60%) even for leading models. Models perform best on English and Chinese; Japanese, Korean, and especially Persian impose greater difficulty.
- **Human vs. Model:** Human familiarity with a meme correlates with top performance (avg ≈ 85%). Leading models match or slightly exceed human novice scores on familiar items, but broad human competence remains superior, especially on unfamiliar clips.
- **Temporal and Cultural Variation:** Performance peaks for memes originating from 1980–2000; accuracy dips for older (<1980) or very recent (>2020) items, indicating possible distributional biases in pretraining data. LLMs underperform for memes in less-represented languages or with implicit cultural references.

## 6. Diagnostic Analyses and Gaps

Several systematic trends are evident:

- **Textless Audio Bottleneck:** Loss of linguistic cues in instrumental music and sound effects degrades model accuracy by 20–30% relative to speech or song. This reflects current MLLMs' reliance on language for inference and highlights limitations in prosodic, timbral, and genre-level comprehension.
- **Surface vs. Deep Understanding:** All models exhibit a performance hierarchy: L > A > {C, E} > {H, U} > W. Tasks involving context, emotion, usage, and world knowledge remain substantially below surface-level comprehension even in best-in-class systems (with drops of 15–30% for Gemini 3 Pro from L to pragmatic categories).
- **Cheat Pathways:** Allowing visual or on-screen textual hints artificially boosts scores; elimination of such cues is necessary for assessing genuine multimodal grounding.

## 7. Outlook and Prospects

The AVMeme Exam demonstrates a fundamental gap in current MLLM capability: effective surface content and language parsing, but persistent deficits in textless audio interpretation, contextual reasoning, emotion recognition, pragmatic usage, and world knowledge.

Recommended directions for bridging these gaps include:

- Expansion of audio training corpora to encompass both non-linguistic sounds and broader cultural artifacts.
- Supervision that targets emotion, typical use, and pragmatic context, emphasizing human-centered annotation.
- Objective function and architectural refinements prioritizing inference and contextualization over shallow recognition.
- Extension of the benchmark to further linguistic, cultural, and generational categories, and the development of generative, open-ended evaluation protocols complementing multiple-choice assessment.

AVMeme Exam provides not only a comprehensive diagnostic tool for multimodal intelligence but also a structured research agenda for advancing culturally and contextually aligned LLMs [2601.17645].

Source: https://www.emergentmind.com/topics/avmeme-exam