---
title: 'UGC-VideoCap: Omnimodal Captioning Benchmark'
url: https://www.emergentmind.com/topics/ugc-videocap
type: topic
---

# UGC-VideoCap: Omnimodal Captioning Benchmark

UGC-VideoCap is a benchmark for detailed omnimodal captioning and question answering on short-form user-generated videos, introduced alongside the UGC-VideoCaptioner-3B model in 2025. In the terminology of the originating paper, **UGC-VideoCap** denotes the evaluation resource, whereas **UGC-VideoCaptioner-3B** denotes the corresponding 3B-parameter captioning model built on top of Qwen2.5-Omni-3B. The benchmark was motivated by the claim that existing video-captioning datasets and models are predominantly visual-centric and underrepresent audio-grounded semantics that are often indispensable in short-form UGC such as TikTok videos, where speech, music, sound effects, OCR, scene dynamics, and overall video purpose are tightly coupled [2507.11336].

## 1. Benchmark identity and problem formulation

UGC-VideoCap is explicitly defined as a benchmark for **detailed omnimodal captioning** of short-form user-generated videos. The benchmark is built from **1,000 TikTok videos**, each **under 60 seconds**, selected to contain **meaningful audio segments of at least 5 seconds**. Its domain is therefore short-form UGC rather than curated movies or studio footage. The paper characterizes these videos as exhibiting uncontrolled recording conditions, rapid scene changes, diverse content styles, non-studio speech, overlapping sound sources, and noisy visuals, and uses those properties to argue that balanced audio-visual modeling is necessary for faithful caption generation [2507.11336].

A central conceptual distinction in the paper is between ordinary video captioning and what it calls **“detailed omnimodal captioning.”** In that framing, the task is not merely to summarize visible actions and objects. Rather, the target output is a coherent caption that integrates **visual content, audio cues, OCR, temporal scene context, and overall video purpose**. The benchmark is thus designed to probe whether a model can recover semantics that are available only from sound, such as speech content and style, the number and type of speakers, background music, ambient sounds, emotional tone, and sound effects, while also grounding visible content in the same output [2507.11336].

The paper positions UGC-VideoCap against prior captioning benchmarks such as **VATEX**, **DREAM-1K**, **MSR-VTT**, **VDC**, and **VidCapBench**, and states that UGC-VideoCap is the only listed benchmark in that comparison table with both **audio in caption** and **visual in caption**. A plausible implication is that the benchmark is intended not only as a new dataset but also as a critique of evaluation protocols that allow high scores while ignoring audio-grounded meaning [2507.11336].

## 2. Dataset composition and annotation pipeline

The benchmark is constructed through a **three-stage human-in-the-loop annotation pipeline**. For each video, annotators produce **an audio-only annotation and caption**, **a visual-only annotation and caption**, and **a joint audio-visual final caption**. The paper further states that annotators originally collect **15–20 audio/visual annotations per video** as raw information and then select those most characteristic of UGC videos for benchmark construction [2507.11336].

The first stage, **audio-only annotation**, captures semantics available from sound without relying on frames. The paper explicitly lists **number of speakers**, **voice type / gender**, **background music presence and type / genre**, **sound effects**, broader **phonetic features**, and **acoustic conditions**. The associated audio-only caption summarizes voices, music, ambient sounds, and emotional tone. The second stage, **visual-only annotation**, captures information visible without listening, including **OCR text**, **background transitions / background changes**, **motion dynamics**, **object types**, **scene dynamics**, and **object presence**. The corresponding visual-only caption summarizes setting, visible actors and objects, actions, scene layout, and on-screen text. The third stage fuses the two streams into a final caption intended to capture the primary scene and setting, key characters or objects and actions, significant audio cues, OCR and its contextual role, and the overall theme or purpose of the video [2507.11336].

The paper reports that the full pipeline consumed **over 350 hours of human effort**, spanning filtering, annotation, standardization, and QA generation. Quality control is batch-based: every batch of **50 annotated videos** is independently reviewed by **two expert annotators** and checked for **accuracy, completeness, and clarity** across audio attributes, visual descriptors, and audiovisual captions. Annotation errors are defined as **factual inaccuracies**, **omissions of salient content**, and **inconsistencies with the video**. A batch is **rejected and sent back for re-annotation if the error rate exceeds 3%**, and reviewer disagreements are resolved through **arbitration or consensus review**. The paper does not provide inter-annotator agreement numbers, annotator demographics, or compensation details [2507.11336].

This annotation design makes the benchmark diagnostically structured rather than caption-only. A plausible implication is that it supports failure analysis at the modality level, since a model may succeed on visual-only description while failing on speaker count, tone, or music-grounded narrative cues.

## 3. QA benchmark and evaluation design

In addition to captions, UGC-VideoCap includes a QA benchmark intended to measure both unimodal and multimodal understanding. The paper describes the resource as containing “around **4,000** high-quality QA pairs,” “**over 4,000** manually curated open-ended and multiple-choice questions,” and a table value of **3,975 QA pairs**. These QA pairs are divided into **Visual QA**, **Audio QA**, and **Comprehensive QA**. Visual QA covers **scene dynamics**, **object presence**, **OCR content**, and **background changes**. Audio QA covers **speaker count**, **gender / voice attributes**, **music genre**, and **environmental sounds**. Comprehensive QA corresponds to final captioning questions that require joint audio-visual understanding [2507.11336].

The benchmark therefore explicitly probes both **unimodal understanding** and **cross-modal / integrated understanding**. For multiple-choice questions, the paper uses exact-match **Accuracy (ACC)**. For open-ended questions, it uses **GPT-4o-2024-08-06 as an automatic judge**, with the stated claim that the evaluated videos and texts are not in its training set. For OCR within visual-detail evaluation, the paper reports the average of **BLEU-1, ROUGE-1, ROUGE-2, and ROUGE-L**, scaled to a percentage-like score for averaging. The benchmark tables report columns for **Voice Source**, **Tone**, **OCR**, **Background**, **Objects**, **Audio**, **Visual**, **Detail**, and **Average**, although the paper does not define every column formula in detail [2507.11336].

The prompt used for benchmark caption generation is uniform across evaluated models. It asks for a detailed and coherent paragraph that integrates all modalities and explicitly requests five content types: the primary scene and background setting; key characters or objects and their actions or interactions; significant audio cues such as voices, background music, sound effects, and their emotional tone; on-screen text and its role in context; and the overall theme or purpose of the video. It also specifies that the output should be a fluent and objective paragraph rather than a bullet-point list. This prompt operationalizes the benchmark’s notion of omnimodal captioning and makes the expected output format explicit [2507.11336].

## 4. UGC-VideoCaptioner-3B architecture and training methodology

The paper presents **UGC-VideoCaptioner-3B** as a task-specialized adaptation of **Qwen2.5-Omni-3B** rather than a wholly new multimodal architecture. The concrete implementation details given are: multimodal input of **audio + visual**, **1 fps** frame sampling, a maximum of **32** frames, a maximum of **100176** pixels per frame, a maximum prompt length of **8192**, and training on **8× H200 144GB GPUs**. The paper does not provide an internal block-level architecture description, a custom fusion module, explicit temporal encoder details, tokenizer formatting for audio/video streams, or whether any modality adapters are frozen or tuned [2507.11336].

A core methodological contribution is **teacher-student distillation** from **Gemini-2.5 Flash**. The authors use Gemini-2.5 Flash to auto-label **20,000 TikTok videos** with detailed omni captions and then train the 3B student on those pseudo-labels. The paper gives the distillation objective as
$$
\mathcal{L}_{\text{distill}(x_i)} = - \sum_{t=1}^{T} \log P_S\left(\hat{y}_{i,t} \mid \hat{y}_{i,<t}, x_i\right)
$$
where $\hat{y}_i = T(x_i)$ is the teacher output for input $x_i$, and $P_S$ is the student model’s token probability. The overall objective is intended as
$$
\mathcal{L}_{\text{total} = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}_{\text{distill}(x_i)} .
$$
The paper notes that its printed LaTeX is malformed, but the intended meaning is the student’s imitation of teacher-generated omni captions [2507.11336].

The training pipeline has two stages: **Supervised Fine-Tuning (SFT)** and **Group Relative Policy Optimization (GRPO)**. The motivation is that, for a small 3B model, pure SFT appears to hit a performance ceiling quickly, whereas RL can improve detailed captioning even without chain-of-thought supervision. The GRPO objective is presented in partially corrupted form as
$$
J(\theta) = \mathbb{E}_{x}\!\biggl[ \sum_{k=1}^K w_k \,\log \pi_{\theta}\bigl(y_k \mid x\bigr) \biggr] - \beta \, D_{\mathrm{KL}\!\Bigl( \pi_{\theta}(\cdot \mid x) \,\Big\|\, \pi_{\mathrm{ref}(\cdot \mid x) \Bigr) ,
$$
where $y_1,\ldots,y_K$ are sampled candidate captions for input $x$, $w_k$ is a normalized reward-derived weight, and $\beta$ controls a KL penalty. The paper suggests a softmax-style normalization example,
$$
w_k \text{ could be } \frac{\exp(\tilde{r}_k/\tau)}{\sum_j \exp(\tilde{r}_j/\tau)} ,
$$
but does not fully specify the normalization procedure beyond that example [2507.11336].

The RL stage uses two rewards. The first is an **LLM-based omni reward**, in which **Gemini-2.5 Flash** scores a predicted caption against ground truth on five dimensions: `scene_background`, `characters_objects`, `audio_cues`, `ocr_text`, and `theme_purpose`. The judge returns an integer score
$$
R_{\text{LLM} \in \{1, 2, 3, 4, 5\}
$$
with conservative rules: if there is hallucinated content contradicting the ground truth, the score must be at most 2, and borderline cases use the lower score. The second is a **length-based reward** designed to match the caption-length distribution of Gemini-2.5 Flash:
$$
R_l = \begin{cases} 0 , & \text{if } len(completions) < 0.5*len(GT) \\ 0.5, & \text{if } 0.7*len(GT)> len(completions) > 0.5*len(GT) \\ 1.0 & len(completions) > 0.7*len(GT) \end{cases}
$$
The paper does not state exactly how $R_{\text{LLM}}$ and $R_l$ are combined into a single scalar reward [2507.11336].

## 5. Benchmark results and empirical findings

The benchmark evaluates proprietary and open-source omni-capable models including **Gemini-2.5 Pro**, **Gemini-2.5 Flash**, **Gemma-3n-E4B-it**, **Gemma-3n-E2B-it**, **Qwen2.5-Omni-7B**, **Qwen2.5-Omni-3B**, and **MiniCPM-o-2.6-8B** under the same prompt, with **1 fps** frame sampling, a maximum of **32 frames**, and both audio and video provided. The overall benchmark table reports **Gemini-2.5 Flash** at **76.73** average, **Gemini-2.5 Pro** at **73.78**, **MiniCPM-o-2.6-8B** at **59.6**, **Qwen2.5-Omni-7B** at **58.0**, **Gemma-3n-E4B-it** at **53.3**, **Qwen2.5-Omni-3B** at **52.2**, and **Gemma-3n-E2B-it** at **51.3**. For final caption columns, the paper reports **Gemini-2.5 Flash** at Audio **74.2**, Visual **78.8**, Detail **77.2**; **Gemini-2.5 Pro** at **70.8 / 75.8 / 74.8**; **MiniCPM-o-2.6-8B** at **46.0 / 70.4 / 62.4**; **Qwen2.5-Omni-7B** at **57.6 / 59.4 / 57.0**; and **Qwen2.5-Omni-3B** at **48.2 / 55.6 / 52.6** [2507.11336].

The paper emphasizes modality imbalance in open models. For example, **Qwen2.5-Omni-3B** scores **93.0** on **Voice Source**, but only **18.3** on **Objects** and **18.4** on **Background**. **Qwen2.5-Omni-7B** also has **86.6** on Voice Source but poor object and background scores. The authors interpret such results as evidence that current open-source omni models do not yet perform true audio-visual integration. This suggests that audio support at the input layer does not, by itself, guarantee balanced multimodal reasoning [2507.11336].

For the proposed model, the key training comparison is between the base model and different post-training strategies. The paper reports **Qwen2.5-Omni-3B (base)** at \(Caption_{\text{Audio} = 48.2\), \(Caption_{\text{Visual} = 55.6\), \(Caption_{\text{Details} = 52.6\), Average **52.18**; **1k SFT** at **58.96** average; **10k SFT** at **59.87**; **20k SFT** at **60.50**; **UGC-VideoCaptioner-3B-zero, 1k RL only** at **55.40**; and **UGC-VideoCaptioner-3B, 1k SFT + 1k RL** at **60.01**. The paper explicitly draws four conclusions: **SFT helps substantially** even with only 1k samples; scaling SFT from 10k to 20k yields only **0.63** additional average points; **RL-only is weaker than SFT-only**; and **1k SFT + 1k RL** outperforms **10k SFT** and nearly matches **20k SFT**, indicating strong **data efficiency** [2507.11336].

These findings are significant chiefly because the benchmark is built to separate auditory, visual, and fused caption competence. A plausible implication is that UGC-VideoCap functions as both an evaluation dataset and a modality-balance stress test for omni-capable MLLMs.

## 6. Relation to adjacent UGC research, limitations, and broader significance

UGC-VideoCap addresses semantic captioning and QA rather than perceptual quality assessment, but it is situated in a broader UGC literature that has increasingly emphasized authentic web-video conditions. The **YouTube UGC Dataset** introduced a large-scale raw-video corpus of **1500 video clips**, each **20 seconds** long, sampled from **YouTube** videos with the **Creative Commons license**, and organized around **15 categories**; it contains no captions and no multimodal language annotations, but it established UGC as a distinct compression and quality domain rather than a pristine-video variant [1904.06457]. The later **BVI-UGC** database extended this line of work to the **UGC transcoding setting**, with **60 pseudo-pristine source sequences**, **60 non-pristine reference videos**, **1,080 transcoded/distorted test videos**, and a crowdsourced study involving **more than 3,500 participants**; it showed that both full-reference and no-reference video-quality metrics perform poorly on realistic UGC transcoding, with no tested metric reaching **SROCC 0.6** on the overall DMOS task [2408.07171]. The **FineVQ** line further introduced **FineVD**, a **6104**-video UGC database with fine-grained quality scores and textual QA-style descriptions across six dimensions, thereby moving toward language-mediated description of UGC quality rather than captioning of scene semantics [2412.19238].

Against that background, UGC-VideoCap is distinctive because it targets **short-form TikTok video**, treats **audio as first-class supervision**, and pairs omnimodal captions with QA designed to probe unimodal and cross-modal understanding. At the same time, the paper identifies several limitations. The benchmark contains only **1,000 videos**, the domain is limited to **TikTok**, official benchmark splits are not specified in the provided text, inter-annotator agreement statistics are not reported, and open-ended evaluation relies on **LLM-as-a-judge** rather than fully human scoring. On the model side, the contribution is primarily a **training methodology** rather than a new multimodal backbone, and the exact reward combination is underspecified in the text. The paper also points to future directions including **automatic audio event detection**, **voice diarization**, **multilingual audio and text**, and **modality-aware attention mechanisms** [2507.11336].

The broader significance of UGC-VideoCap lies in the way it reframes video captioning for real-world UGC. Earlier UGC datasets established that authentic user videos are noisy, heterogeneous, and often degraded by prior compression or transcoding [1904.06457; 2408.07171]. UGC-VideoCap adds the complementary claim that, even when semantic captioning is the primary task, faithful understanding of such videos requires explicit treatment of audio, OCR, scene dynamics, and purpose rather than a vision-only summary [2507.11336]. This suggests a convergence between UGC quality research and UGC captioning research: both reject simplified assumptions derived from pristine, studio-like video, and both treat platform-native UGC as a domain with its own annotation requirements, evaluation criteria, and model failure modes.

Source: https://www.emergentmind.com/topics/ugc-videocap