XGC-AVQuiz: Audio-Visual Benchmark

• XGC-AVQuiz is a comprehensive benchmark that integrates PGC, UGC, and AIGC content to assess multimodal audio-visual understanding.
• It organizes 2,685 multiple-choice question–answer pairs into 20 distinct tasks covering recognition, localization, reasoning, and quality perception.
• The design emphasizes both low-level sensory evaluation and high-level semantic reasoning to diagnose model performance and reveal precise failure modes.

XGC-AVQuiz is a benchmark introduced alongside XGC-AVis for comprehensively assessing multimodal large models’ understanding capabilities in both real-world and AI-generated audio-visual scenarios (Cao et al., 27 Sep 2025). It contains 2,685 multiple-choice question–answer pairs over 2,232 video clips and structures evaluation across 20 tasks spanning recognition, localization, reasoning, and quality perception. Its two stated innovations are AIGC Scenario Expansion, which combines professionally generated content (PGC), user-generated content (UGC), and AI-generated content (AIGC), and Quality Perception Dimension, which requires models to integrate low-level sensory capabilities with high-level semantic understanding in order to assess audio-visual quality, synchronization, and coherence.

1. Corpus composition and scenario coverage

XGC-AVQuiz contains 2,232 videos and 2,685 QA pairs. The source breakdown is 1,102 PGC videos, 753 UGC videos, and 377 AIGC videos (Cao et al., 27 Sep 2025).

Source	Videos
PGC	1,102
UGC	753
AIGC	377

The benchmark covers 10 major domains and 53 fine-grained categories. The major domains include Film & TV, Music Performance, Live Streaming, Gaming, Vlogging, Educational, Advertising, Animation, Virtual-Avatar Interactions, and Experimental AIGC. The fine-grained categories include, for example, action, drama, and comedy within Film; pop, rock, and classical within Music; game commentary and dance tutorials within UGC; and avatar speech and generative animation within AIGC.

The dataset design explicitly ties content diversity to evaluation difficulty. The combination of PGC, UGC, and AIGC is stated to ensure diversity in production values, styles, and the realism-versus-surrealism of audio-visual events. Within the AIGC subset, clips are sourced from AI-avatar videos on YouTube and the Kling platform and are further enriched by the AGAVQA-3k dataset of AI video clips exhibiting known distortions, including misalignment, semantic inconsistencies, and audio artifacts. Manual curation ensured that each selected clip has both dynamic visual frames and an AIGC audio track, together with a balance of high- versus low-quality examples.

2. Task hierarchy and question design

XGC-AVQuiz organizes its 2,685 QA pairs into four levels of cognitive demand—recognition, localization, reasoning, and quality perception—across 20 distinct task types (Cao et al., 27 Sep 2025). Every task is cast as a multiple-choice QA problem.

The recognition tier contains three tasks. Audio-Source Recognition asks which object or event is producing a non-speech sound, using 10–15 sampled frames together with the full audio segment. Music Recognition asks for the genre or type of music. Counting asks for the number of repeated audio events such as footsteps or claps.

The localization tier also contains three tasks, each phrased as “When does $X$ occur in the video?” with timed candidate intervals. These are Audio-Source Localization, Speaker Localization, and Music Onset/Change Localization. The output is an interval selection of the form $[\hat s,\hat e]$.

The reasoning tier contains seven tasks: Music Understanding, Event Causal Reasoning, Human Interaction Reasoning, Identity Reasoning, Audio Causal Reasoning, A/V Prediction, and Emotion Reasoning. These are multiple-choice inference problems over concatenated vision, audio, and subtitle tokens.

The quality-perception tier contains seven tasks: A/V Content Matching, Music Temporal Matching, Audio Temporal Matching, Speech Temporal Matching, Distortion Type Classification, Distortion Localization, and A/V Overall Quality. This family is the benchmark’s most distinctive addition because it couples perceptual fidelity judgments with cross-modal consistency judgments.

A common simplification is to treat XGC-AVQuiz as an audio-visual recognition benchmark with added AIGC examples. That characterization is incomplete. The stated task taxonomy makes quality perception and temporal alignment coequal with recognition and reasoning rather than peripheral extensions.

3. Evaluation protocol and formal metrics

The benchmark uses zero-shot evaluation and reports classification accuracy as the primary metric (Cao et al., 27 Sep 2025). For each video, 15 uniformly sampled frames are tokenized, the full audio stream is provided, optionally as raw spectrogram or via subtitles/descriptions, and the model outputs the letter of the correct answer. Accuracy is aggregated by task category and across all 2,685 QA pairs.

The benchmark’s operative metric is

$$\mathrm{Acc} \;=\; \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\hat{y}_i = y_i\}.$$

Here $N$ is the total number of examples, $y_i$ is the ground-truth option label, and $\hat y_i$ is the model choice.

Although the localization tasks are evaluated by interval-selection accuracy, the technical summary notes that one can define a temporal-error metric,

$$E_{\mathrm{loc}} \;=\; \frac{1}{N} \sum_{i=1}^N \bigl|\hat s_i - s_i\bigr| + \bigl|\hat e_i - e_i\bigr|,$$

with $(s_i,e_i)$ denoting the ground-truth window for example $i$. Likewise, for synchronization one can in principle quantify performance through

$$\mathrm{SyncAcc}_\tau \;=\; \frac{1}{N}\sum_{i=1}^N \mathbf{1}\!\bigl\{|\Delta t_i| < \tau\bigr\},$$

where $[\hat s,\hat e]$0 is the audio-video offset error and $[\hat s,\hat e]$1 is a tolerance.

These latter expressions are not presented as the benchmark’s main reporting protocol. The benchmark operationalizes localization, synchronization, and quality judgments as multiple-choice classification tasks rather than as continuous regression or offset-estimation problems. This distinction matters when comparing XGC-AVQuiz to evaluation suites centered on temporal IoU, offset estimation, or full-reference quality regression.

4. Quality perception as a benchmark dimension

In XGC-AVQuiz, audio-visual quality perception is defined as a model’s ability to detect low-level distortions such as noise, blur, and stuttering, together with high-level mismatches such as semantic or temporal misalignment, in a way that mirrors how humans assess both sensory fidelity and narrative coherence (Cao et al., 27 Sep 2025).

The benchmark specifies two interacting strata. The low-level signal side includes audio noise floor, video sharpness, frame drops, and bitrate artifacts. The high-level semantic side includes consistency between audio events and object motion, lip-sync correctness, and musical emphasis on visuals. This framing places quality perception between classical signal-quality assessment and multimodal semantic reasoning rather than equating it with either one.

The summary further notes that one can imagine a unified score

$[\hat s,\hat e]$2

where $[\hat s,\hat e]$3 measures signal degradations and $[\hat s,\hat e]$4 measures cross-modal alignment. However, in XGC-AVQuiz these dimensions are not evaluated through real-valued regression; they are operationalized as classification QAs. This design choice helps explain why tasks such as distortion localization and overall A/V quality can be benchmarked within the same multiple-choice framework as recognition and reasoning.

A common misconception is that “quality perception” in this benchmark is only a synonym for lip-sync detection. The task list shows a broader scope: it includes content matching, music temporal matching, non-speech audio temporal matching, speech temporal matching, distortion type classification, distortion localization, and overall A/V quality.

5. Empirical findings and diagnostic role

The reported results indicate that current multimodal large models struggle on quality perception and temporal alignment tasks (Cao et al., 27 Sep 2025). Many open-source vision-LLMs, including LLaVA-OneVision and InternVL3, and omni-LLMs struggle when audio is added naïvely, often showing a drop in performance compared with video-only input. Temporal localization tasks, including counting and event-span detection, remain difficult, and even top models hover around 50% accuracy.

The quality-perception category is especially weak. The lowest scores are reported for distortion localization, at approximately 40%, and overall A/V quality, at approximately 38% in closed-source MLLMs. Performance on AIGC clips is generally lower than on PGC clips, indicating reduced robustness under generative-content shifts.

A quantitative snapshot, under video+audio+subtitle input, illustrates the benchmark’s category-level spread:

Model	Overall	Quality perception
Qwen2.5-Omni (7B)	46.7%	41.8%
Gemini 2.0 Flash (closed)	51.3%	38.1%
XGC-AVis (multi-agent)	58.2%	51.0%

The same snapshot reports category averages of 53.7% recognition, 33.0% localization, 41.8% perception, and 55.1% reasoning for Qwen2.5-Omni (7B); 55.7%, 49.3%, 38.1%, and 68.1% for Gemini 2.0 Flash; and 59.5%, 51.7%, 51.0%, and 69.3% for XGC-AVis. Relative to the best single-agent MLLM, Gemini 2.0 Flash, XGC-AVis gains $[\hat s,\hat e]$5 in recognition, $[\hat s,\hat e]$6 in localization, $[\hat s,\hat e]$7 in quality perception, and $[\hat s,\hat e]$8 in reasoning, for a $[\hat s,\hat e]$9 absolute gain in overall accuracy.

These results make XGC-AVQuiz diagnostic rather than merely comparative. The benchmark is described as surfacing precise failure modes, especially where models must jointly assess sensory fidelity and semantic coherence.

6. Relation to XGC-AVis and broader significance

XGC-AVQuiz was introduced together with XGC-AVis, a multi-agent framework intended to enhance audio-video temporal alignment and improve the efficiency of retrieving key video segments through four stages: perception, planning, execution, and reflection (Cao et al., 27 Sep 2025). XGC-AVis is reported to improve capabilities without requiring additional training, and its gains are validated on two benchmarks.

Within that pairing, XGC-AVQuiz functions as the evaluation substrate. It is the component that brings AIGC into the benchmark loop, defines the four-tier task hierarchy, and provides a unified multiple-choice QA evaluation that exposes where off-the-shelf MLLMs underperform. The benchmark’s significance therefore lies not only in aggregate accuracy reporting, but in its ability to separate failures in recognition, temporal localization, causal and social reasoning, and quality perception.

The benchmark’s inclusion of AIGC content is especially consequential. The stated rationale is that PGC tends to reflect high-budget filmmaking, UGC everyday candid scenes, and AIGC novel and sometimes physically impossible events such as synthetic creatures or exaggerated physics. This suggests that XGC-AVQuiz is intended to probe whether a model’s cross-modal alignment mechanisms generalize beyond ordinary real-world regularities.

A further implication is methodological. Because the benchmark keeps all 20 tasks within a unified multiple-choice, zero-shot setting, it allows direct comparison across heterogeneous capabilities while avoiding a proliferation of task-specific scoring rules. At the same time, the summary explicitly notes definable continuous metrics for localization and synchronization, which suggests a possible future expansion toward hybrid evaluation protocols.