UGC-VideoCap: Omnimodal Captioning Benchmark

• UGC-VideoCap is a benchmark for detailed omnimodal captioning of short-form user-generated videos, integrating audio, visual, OCR, and temporal cues.
• It employs a three-stage human-in-the-loop annotation process to separately capture audio-only, visual-only, and fused audio-visual descriptions.
• The benchmark includes a QA component to test unimodal and cross-modal understanding, guiding effective training strategies for balanced model performance.

UGC-VideoCap is a benchmark for detailed omnimodal captioning and question answering on short-form user-generated videos, introduced alongside the UGC-VideoCaptioner-3B model in 2025. In the terminology of the originating paper, UGC-VideoCap denotes the evaluation resource, whereas UGC-VideoCaptioner-3B denotes the corresponding 3B-parameter captioning model built on top of Qwen2.5-Omni-3B. The benchmark was motivated by the claim that existing video-captioning datasets and models are predominantly visual-centric and underrepresent audio-grounded semantics that are often indispensable in short-form UGC such as TikTok videos, where speech, music, sound effects, OCR, scene dynamics, and overall video purpose are tightly coupled (Wu et al., 15 Jul 2025).

1. Benchmark identity and problem formulation

UGC-VideoCap is explicitly defined as a benchmark for detailed omnimodal captioning of short-form user-generated videos. The benchmark is built from 1,000 TikTok videos, each under 60 seconds, selected to contain meaningful audio segments of at least 5 seconds. Its domain is therefore short-form UGC rather than curated movies or studio footage. The paper characterizes these videos as exhibiting uncontrolled recording conditions, rapid scene changes, diverse content styles, non-studio speech, overlapping sound sources, and noisy visuals, and uses those properties to argue that balanced audio-visual modeling is necessary for faithful caption generation (Wu et al., 15 Jul 2025).

A central conceptual distinction in the paper is between ordinary video captioning and what it calls “detailed omnimodal captioning.” In that framing, the task is not merely to summarize visible actions and objects. Rather, the target output is a coherent caption that integrates visual content, audio cues, OCR, temporal scene context, and overall video purpose. The benchmark is thus designed to probe whether a model can recover semantics that are available only from sound, such as speech content and style, the number and type of speakers, background music, ambient sounds, emotional tone, and sound effects, while also grounding visible content in the same output (Wu et al., 15 Jul 2025).

The paper positions UGC-VideoCap against prior captioning benchmarks such as VATEX, DREAM-1K, MSR-VTT, VDC, and VidCapBench, and states that UGC-VideoCap is the only listed benchmark in that comparison table with both audio in caption and visual in caption. A plausible implication is that the benchmark is intended not only as a new dataset but also as a critique of evaluation protocols that allow high scores while ignoring audio-grounded meaning (Wu et al., 15 Jul 2025).

2. Dataset composition and annotation pipeline

The benchmark is constructed through a three-stage human-in-the-loop annotation pipeline. For each video, annotators produce an audio-only annotation and caption, a visual-only annotation and caption, and a joint audio-visual final caption. The paper further states that annotators originally collect 15–20 audio/visual annotations per video as raw information and then select those most characteristic of UGC videos for benchmark construction (Wu et al., 15 Jul 2025).

The first stage, audio-only annotation, captures semantics available from sound without relying on frames. The paper explicitly lists number of speakers, voice type / gender, background music presence and type / genre, sound effects, broader phonetic features, and acoustic conditions. The associated audio-only caption summarizes voices, music, ambient sounds, and emotional tone. The second stage, visual-only annotation, captures information visible without listening, including OCR text, background transitions / background changes, motion dynamics, object types, scene dynamics, and object presence. The corresponding visual-only caption summarizes setting, visible actors and objects, actions, scene layout, and on-screen text. The third stage fuses the two streams into a final caption intended to capture the primary scene and setting, key characters or objects and actions, significant audio cues, OCR and its contextual role, and the overall theme or purpose of the video (Wu et al., 15 Jul 2025).

The paper reports that the full pipeline consumed over 350 hours of human effort, spanning filtering, annotation, standardization, and QA generation. Quality control is batch-based: every batch of 50 annotated videos is independently reviewed by two expert annotators and checked for accuracy, completeness, and clarity across audio attributes, visual descriptors, and audiovisual captions. Annotation errors are defined as factual inaccuracies, omissions of salient content, and inconsistencies with the video. A batch is rejected and sent back for re-annotation if the error rate exceeds 3%, and reviewer disagreements are resolved through arbitration or consensus review. The paper does not provide inter-annotator agreement numbers, annotator demographics, or compensation details (Wu et al., 15 Jul 2025).

This annotation design makes the benchmark diagnostically structured rather than caption-only. A plausible implication is that it supports failure analysis at the modality level, since a model may succeed on visual-only description while failing on speaker count, tone, or music-grounded narrative cues.

3. QA benchmark and evaluation design

In addition to captions, UGC-VideoCap includes a QA benchmark intended to measure both unimodal and multimodal understanding. The paper describes the resource as containing “around 4,000 high-quality QA pairs,” “over 4,000 manually curated open-ended and multiple-choice questions,” and a table value of 3,975 QA pairs. These QA pairs are divided into Visual QA, Audio QA, and Comprehensive QA. Visual QA covers scene dynamics, object presence, OCR content, and background changes. Audio QA covers speaker count, gender / voice attributes, music genre, and environmental sounds. Comprehensive QA corresponds to final captioning questions that require joint audio-visual understanding (Wu et al., 15 Jul 2025).

The benchmark therefore explicitly probes both unimodal understanding and cross-modal / integrated understanding. For multiple-choice questions, the paper uses exact-match Accuracy (ACC). For open-ended questions, it uses GPT-4o-2024-08-06 as an automatic judge, with the stated claim that the evaluated videos and texts are not in its training set. For OCR within visual-detail evaluation, the paper reports the average of BLEU-1, ROUGE-1, ROUGE-2, and ROUGE-L, scaled to a percentage-like score for averaging. The benchmark tables report columns for Voice Source, Tone, OCR, Background, Objects, Audio, Visual, Detail, and Average, although the paper does not define every column formula in detail (Wu et al., 15 Jul 2025).

The prompt used for benchmark caption generation is uniform across evaluated models. It asks for a detailed and coherent paragraph that integrates all modalities and explicitly requests five content types: the primary scene and background setting; key characters or objects and their actions or interactions; significant audio cues such as voices, background music, sound effects, and their emotional tone; on-screen text and its role in context; and the overall theme or purpose of the video. It also specifies that the output should be a fluent and objective paragraph rather than a bullet-point list. This prompt operationalizes the benchmark’s notion of omnimodal captioning and makes the expected output format explicit (Wu et al., 15 Jul 2025).

4. UGC-VideoCaptioner-3B architecture and training methodology

The paper presents UGC-VideoCaptioner-3B as a task-specialized adaptation of Qwen2.5-Omni-3B rather than a wholly new multimodal architecture. The concrete implementation details given are: multimodal input of audio + visual, 1 fps frame sampling, a maximum of 32 frames, a maximum of 100176 pixels per frame, a maximum prompt length of 8192, and training on 8× H200 144GB GPUs. The paper does not provide an internal block-level architecture description, a custom fusion module, explicit temporal encoder details, tokenizer formatting for audio/video streams, or whether any modality adapters are frozen or tuned (Wu et al., 15 Jul 2025).

A core methodological contribution is teacher-student distillation from Gemini-2.5 Flash. The authors use Gemini-2.5 Flash to auto-label 20,000 TikTok videos with detailed omni captions and then train the 3B student on those pseudo-labels. The paper gives the distillation objective as

$$\mathcal{L}_{\text{distill}(x_i)} = - \sum_{t=1}^{T} \log P_S\left(\hat{y}_{i,t} \mid \hat{y}_{i,<t}, x_i\right)$$

where $\hat{y}_i = T(x_i)$ is the teacher output for input $x_i$, and $P_S$ is the student model’s token probability. The overall objective is intended as

$\mathcal{L}_{\text{total} = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}_{\text{distill}(x_i)} .$

The paper notes that its printed LaTeX is malformed, but the intended meaning is the student’s imitation of teacher-generated omni captions (Wu et al., 15 Jul 2025).

The training pipeline has two stages: Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). The motivation is that, for a small 3B model, pure SFT appears to hit a performance ceiling quickly, whereas RL can improve detailed captioning even without chain-of-thought supervision. The GRPO objective is presented in partially corrupted form as

$J(\theta) = \mathbb{E}_{x}\!\biggl[ \sum_{k=1}^K w_k \,\log \pi_{\theta}\bigl(y_k \mid x\bigr) \biggr] - \beta \, D_{\mathrm{KL}\!\Bigl( \pi_{\theta}(\cdot \mid x) \,\Big\|\, \pi_{\mathrm{ref}(\cdot \mid x) \Bigr) ,$

where $y_1,\ldots,y_K$ are sampled candidate captions for input $x$, $w_k$ is a normalized reward-derived weight, and $\beta$ controls a KL penalty. The paper suggests a softmax-style normalization example,

$\hat{y}_i = T(x_i)$0

but does not fully specify the normalization procedure beyond that example (Wu et al., 15 Jul 2025).

The RL stage uses two rewards. The first is an LLM-based omni reward, in which Gemini-2.5 Flash scores a predicted caption against ground truth on five dimensions: scene_background, characters_objects, audio_cues, ocr_text, and theme_purpose. The judge returns an integer score

$\hat{y}_i = T(x_i)$1

with conservative rules: if there is hallucinated content contradicting the ground truth, the score must be at most 2, and borderline cases use the lower score. The second is a length-based reward designed to match the caption-length distribution of Gemini-2.5 Flash:

$\hat{y}_i = T(x_i)$2

The paper does not state exactly how $\hat{y}_i = T(x_i)$3 and $\hat{y}_i = T(x_i)$4 are combined into a single scalar reward (Wu et al., 15 Jul 2025).

5. Benchmark results and empirical findings

The benchmark evaluates proprietary and open-source omni-capable models including Gemini-2.5 Pro, Gemini-2.5 Flash, Gemma-3n-E4B-it, Gemma-3n-E2B-it, Qwen2.5-Omni-7B, Qwen2.5-Omni-3B, and MiniCPM-o-2.6-8B under the same prompt, with 1 fps frame sampling, a maximum of 32 frames, and both audio and video provided. The overall benchmark table reports Gemini-2.5 Flash at 76.73 average, Gemini-2.5 Pro at 73.78, MiniCPM-o-2.6-8B at 59.6, Qwen2.5-Omni-7B at 58.0, Gemma-3n-E4B-it at 53.3, Qwen2.5-Omni-3B at 52.2, and Gemma-3n-E2B-it at 51.3. For final caption columns, the paper reports Gemini-2.5 Flash at Audio 74.2, Visual 78.8, Detail 77.2; Gemini-2.5 Pro at 70.8 / 75.8 / 74.8; MiniCPM-o-2.6-8B at 46.0 / 70.4 / 62.4; Qwen2.5-Omni-7B at 57.6 / 59.4 / 57.0; and Qwen2.5-Omni-3B at 48.2 / 55.6 / 52.6 (Wu et al., 15 Jul 2025).

The paper emphasizes modality imbalance in open models. For example, Qwen2.5-Omni-3B scores 93.0 on Voice Source, but only 18.3 on Objects and 18.4 on Background. Qwen2.5-Omni-7B also has 86.6 on Voice Source but poor object and background scores. The authors interpret such results as evidence that current open-source omni models do not yet perform true audio-visual integration. This suggests that audio support at the input layer does not, by itself, guarantee balanced multimodal reasoning (Wu et al., 15 Jul 2025).

For the proposed model, the key training comparison is between the base model and different post-training strategies. The paper reports Qwen2.5-Omni-3B (base) at $\hat{y}_i = T(x_i)$5, $\hat{y}_i = T(x_i)$6, $\hat{y}_i = T(x_i)$7, Average 52.18; 1k SFT at 58.96 average; 10k SFT at 59.87; 20k SFT at 60.50; UGC-VideoCaptioner-3B-zero, 1k RL only at 55.40; and UGC-VideoCaptioner-3B, 1k SFT + 1k RL at 60.01. The paper explicitly draws four conclusions: SFT helps substantially even with only 1k samples; scaling SFT from 10k to 20k yields only 0.63 additional average points; RL-only is weaker than SFT-only; and 1k SFT + 1k RL outperforms 10k SFT and nearly matches 20k SFT, indicating strong data efficiency (Wu et al., 15 Jul 2025).

These findings are significant chiefly because the benchmark is built to separate auditory, visual, and fused caption competence. A plausible implication is that UGC-VideoCap functions as both an evaluation dataset and a modality-balance stress test for omni-capable MLLMs.

6. Relation to adjacent UGC research, limitations, and broader significance

UGC-VideoCap addresses semantic captioning and QA rather than perceptual quality assessment, but it is situated in a broader UGC literature that has increasingly emphasized authentic web-video conditions. The YouTube UGC Dataset introduced a large-scale raw-video corpus of 1500 video clips, each 20 seconds long, sampled from YouTube videos with the Creative Commons license, and organized around 15 categories; it contains no captions and no multimodal language annotations, but it established UGC as a distinct compression and quality domain rather than a pristine-video variant (Wang et al., 2019). The later BVI-UGC database extended this line of work to the UGC transcoding setting, with 60 pseudo-pristine source sequences, 60 non-pristine reference videos, 1,080 transcoded/distorted test videos, and a crowdsourced study involving more than 3,500 participants; it showed that both full-reference and no-reference video-quality metrics perform poorly on realistic UGC transcoding, with no tested metric reaching SROCC 0.6 on the overall DMOS task (Qi et al., 2024). The FineVQ line further introduced FineVD, a 6104-video UGC database with fine-grained quality scores and textual QA-style descriptions across six dimensions, thereby moving toward language-mediated description of UGC quality rather than captioning of scene semantics (Duan et al., 2024).

Against that background, UGC-VideoCap is distinctive because it targets short-form TikTok video, treats audio as first-class supervision, and pairs omnimodal captions with QA designed to probe unimodal and cross-modal understanding. At the same time, the paper identifies several limitations. The benchmark contains only 1,000 videos, the domain is limited to TikTok, official benchmark splits are not specified in the provided text, inter-annotator agreement statistics are not reported, and open-ended evaluation relies on LLM-as-a-judge rather than fully human scoring. On the model side, the contribution is primarily a training methodology rather than a new multimodal backbone, and the exact reward combination is underspecified in the text. The paper also points to future directions including automatic audio event detection, voice diarization, multilingual audio and text, and modality-aware attention mechanisms (Wu et al., 15 Jul 2025).

The broader significance of UGC-VideoCap lies in the way it reframes video captioning for real-world UGC. Earlier UGC datasets established that authentic user videos are noisy, heterogeneous, and often degraded by prior compression or transcoding (Wang et al., 2019, Qi et al., 2024). UGC-VideoCap adds the complementary claim that, even when semantic captioning is the primary task, faithful understanding of such videos requires explicit treatment of audio, OCR, scene dynamics, and purpose rather than a vision-only summary (Wu et al., 15 Jul 2025). This suggests a convergence between UGC quality research and UGC captioning research: both reject simplified assumptions derived from pristine, studio-like video, and both treat platform-native UGC as a domain with its own annotation requirements, evaluation criteria, and model failure modes.