OwlCap: Balanced Video Captioning MLLM

• OwlCap is a video-captioning multimodal LLM that addresses motion-detail imbalance by ensuring balanced representation of temporal dynamics and static visuals.
• It employs the HMD-270K dataset for balanced supervision and the Caption Set Equivalence Reward (CSER) to incentivize semantic correctness and completeness.
• OwlCap improves caption quality with significant benchmark gains, demonstrating robust performance in both motion-dominated and detail-rich scenarios.

OwlCap is a video-captioning multimodal LLM (MLLM) developed to address motion-detail imbalance, a failure mode in which a caption captures either temporal dynamics or static visual specifics but not both with comparable fidelity. The system is built from two coupled components: HMD-270K, a dataset construction strategy for motion-detail-balanced supervision, and Caption Set Equivalence Reward (CSER), a reinforcement-learning objective designed to reward both correctness and completeness at the semantic-unit level. In the reported experiments, OwlCap is obtained by supervised fine-tuning on HMD-270K followed by GRPO post-training with CSER, yielding gains on both detail-focused and motion-focused benchmarks (Zhong et al., 26 Aug 2025).

1. Problem formulation and design rationale

The central premise of OwlCap is that video captioning requires simultaneous modeling of two distinct information classes: motion / events, including actions, temporal transitions, camera movement, and interactions; and details / static attributes, including objects, scene layout, appearance, environment, and ambiance (Zhong et al., 26 Aug 2025). The paper argues that existing systems often overemphasize one side of this pair. When motion dominates, captions become event-heavy but visually sparse; when detail dominates, captions may enumerate scene content while omitting the temporal story.

Within this framing, motion-detail imbalance is treated not merely as a stylistic deficiency but as a source of incomplete captions. The paper further connects incomplete captioning to reduced consistency in both video understanding and video generation. A good video caption, under this view, must be both temporally faithful and descriptively rich, rather than optimized for only one axis.

OwlCap responds to this diagnosis with a two-part architecture. The first part operates at the data level, constructing training captions that are explicitly balanced between motion and detail. The second part operates at the optimization level, using a reward that evaluates whether a predicted caption matches ground truth in both directions: whether predicted facts are supported by the reference, and whether reference facts are covered by the prediction. This division of labor is important because the paper does not present reward design as sufficient on its own; the reported system depends on jointly improving supervision and post-training.

2. HMD-270K and motion-detail-balanced supervision

The dataset component is HMD-270K (“Harmonizing Motion-Detail 270K”), which contains 270K video-caption pairs intended to balance motion and detail more effectively than prior caption corpora (Zhong et al., 26 Aug 2025). The paper characterizes caption quality using two simple density statistics: words per second as a proxy for detail richness and verbs per second as a proxy for motion density. From these it defines the Motion-Detail Balance (MDB) metric:

$$\text{MDB} = \left(1 - \frac{w - v}{w + v}\right)\cdot \log(w+1)$$

where $w$ is words per second and $v$ is verbs per second. Higher MDB indicates captions that are both richer and more balanced. The paper reports that HMD-270K achieves much higher balance than Panda-70M, ShareGPT4Video, Vript, and Tarsier-585K.

HMD-270K is built with a two-stage pipeline. The first stage, Motion-Detail Fusion (MDF), is a caption completion process. A video is first given to the motion-oriented MLLM Tarsier, which produces a temporally focused caption. That motion-focused caption, together with the original video, is then provided to Qwen2.5-VL-72B, which is prompted to supplement static details. The result is a fused caption intended to contain both dynamics and fine-grained description.

The second stage, Fine-Grained Examination (FGE), verifies whether the fused caption is actually consistent with the video. The caption is decomposed into indivisible semantic units, and each unit is judged against the video using InternVL2.5-78B. The paper motivates this decomposition by noting that LLMs are better at judging short atomic statements than long free-form captions. A 90% unit accuracy threshold is used to keep a caption-video pair, with fault tolerance for subjective expressions such as “warm atmosphere” and ambiguous categories such as “speaking” versus “arguing.” The paper also mentions cross-model consistency checks and human re-evaluation of rejected samples.

The reported ablation attributes distinct roles to the two stages. MDF improves motion and detail scores relative to raw motion-oriented or detail-oriented generation baselines, while FGE substantially improves trustworthiness. In particular, MDF unit accuracy is 78.93, whereas FGE unit accuracy is 96.54. This suggests that HMD-270K is designed not only to enrich captions but also to remove semantically unreliable pairs before downstream training.

3. Caption Set Equivalence Reward

The optimization component of OwlCap is Caption Set Equivalence Reward (CSER), introduced as an alternative to coarser video-caption reinforcement-learning rewards such as VideoCap-R1’s Event Score (Zhong et al., 26 Aug 2025). CSER is intended to reward two properties simultaneously: correctness, meaning the caption should avoid unsupported content, and completeness, meaning it should not omit key facts.

Its formal basis is set equivalence over semantic units. The predicted caption is decomposed as

$$C_{\text{pred}} \rightarrow \{U_1, U_2, \dots, U_n\}$$

and the ground-truth caption as

$$C_{\text{gt}} \rightarrow \{F_1, F_2, \dots, F_m\}.$$

The correctness score checks whether each predicted unit is supported by the entire ground-truth caption:

$$S_{\text{correctness}} = \frac{\sum_{i=1}^{n}\mathbb{I}(U_i \in C_{\text{gt}})}{n}.$$

The completeness score checks whether each ground-truth fact is covered by the entire predicted caption:

$$S_{\text{completeness}} = \frac{\sum_{j=1}^{m}\mathbb{I}(F_j \in C_{\text{pred}})}{m}.$$

The paper’s interpretation is explicitly semantic rather than lexical: CSER does not enforce literal string equality. Instead, it encourages semantic equivalence between the predicted-caption set and the ground-truth-caption set in a Cantor-style sense, where meaningful elements in one set correspond to elements in the other. A common misconception would be to treat CSER as an exact-match reward; the paper argues the opposite, namely that paraphrase is acceptable provided content is preserved.

For post-training, CSER is used with Group Relative Policy Optimization (GRPO). GRPO is described as a PPO-like method that does not require a critic model. For a query $q_i$, a group of candidate outputs is sampled, rewards are computed, and they are normalized into group-relative advantages:

$$A_i = \frac{r_i - \text{mean}(\{r_1,\dots,r_n\})}{\text{std}(\{r_1,\dots,r_n\})}.$$

The paper states that the policy update uses a clipped surrogate objective and sets $\beta = 0.001$. The final reward used in GRPO is the sum of a format term and the two CSER terms:

$w$0

A plausible implication is that CSER shifts optimization away from one-sided event coverage and toward a bidirectional notion of semantic adequacy.

4. Model base and training pipeline

The main OwlCap instantiation uses Qwen2.5-VL-7B as the base MLLM, although the paper also reports compatibility with Qwen2-VL-7B and Qwen2.5-VL-3B (Zhong et al., 26 Aug 2025). The training pipeline has two stages.

In Stage 1, the base model undergoes supervised fine-tuning (SFT) on HMD-270K. This stage is intended to establish the model’s default captioning behavior under motion-detail-balanced supervision. The paper presents SFT as the foundational step rather than a lightweight warm start.

In Stage 2, the SFT model is further optimized by GRPO post-training with CSER. The reinforcement-learning setup is deliberately selective. The authors begin from HMD data filtered to 12K samples with 100% unit accuracy. Each sample is then generated 8 times to form a group. CSER variance across the group is used to identify hard or informative examples, and the top 50% highest-variance samples are retained, leaving 6K examples for the GRPO set. This concentrates post-training on cases where output variation is meaningful and reward-based discrimination can influence the policy.

The training ablation underscores the complementarity of the two stages. HMD SFT only already improves benchmark scores relative to the baseline, but SFT + GRPO with both CSER terms gives the best reported results. The paper also states that reinforcement learning without good SFT helps less. This suggests that OwlCap is not presented as a pure RL system; its gains are tied to the prior restructuring of the training distribution via HMD-270K.

5. Empirical performance

The principal evaluation is conducted on two benchmarks with different emphases: VDC (Video Detailed Captions), which focuses on detailed video description quality and reports metrics such as Acc, and DREAM-1K, which focuses on motion and event description and reports F1 and Recall (Zhong et al., 26 Aug 2025). With Qwen2.5-VL-7B as the base, OwlCap improves on both.

Benchmark	Baseline Qwen2.5-VL-7B	OwlCap
VDC Acc	42.7	46.9
DREAM-1K F1	30.1	34.7
DREAM-1K Recall	29.7	35.3

These correspond to +4.2 Acc on VDC, +4.6 F1 on DREAM-1K, and +5.6 Recall on DREAM-1K. The paper presents these gains as evidence that OwlCap improves both sides of the motion-detail tradeoff rather than merely shifting performance from one axis to the other.

The VDC comparison table is described as showing that OwlCap outperforms prior specialized methods including ShareGPT4Video, Vript, AuroraCap, Tarsier, and VideoCap-R1, and that it is the best overall model across all evaluated VDC categories in that table. Because the topic of VDC is detailed description, this result is used to support the claim that OwlCap’s motion-aware design does not come at the expense of static detail coverage.

The paper also reports a downstream text-to-video generation validation. The protocol uses 200 videos from Pixabay; captions are generated by competing models and then supplied to HunyuanVideo. Generated videos are compared with references using SSIM, PSNR, and FID. OwlCap achieves SSIM 0.33, PSNR 29.32, and FID 231.60, which are the best reported numbers in that evaluation. This suggests that the captions produced by OwlCap are more useful as generation prompts, not only as benchmark outputs.

6. Ablations, interpretation, and position within the literature

The ablations separate the effects of data construction and reward design (Zhong et al., 26 Aug 2025). On the data side, MDF improves both motion and detail scores, while FGE further improves them and especially raises unit accuracy. On the optimization side, HMD SFT provides a significant boost over the baseline, and adding reinforcement learning with the correctness and completeness reward terms yields further gains. The reported trajectory is: baseline VDC 42.7 / DREAM 30.1, HMD SFT only at 44.8 / 31.9, and full HMD + completeness + correctness at 46.9 / 34.7.

The comparison with Event Score is also structurally important. The paper states that Event Score gives only modest gains on VDC, whereas CSER improves both VDC and DREAM-1K more consistently. The methodological distinction is that Event Score is described as overly simplistic and one-sided, while CSER explicitly models caption quality as a two-way semantic match. This is not merely a metric substitution; it reflects a shift from event coverage alone toward simultaneous control of unsupported content and omitted content.

OwlCap’s relation to earlier datasets is equally central. The paper positions HMD-270K against Panda-70M, ShareGPT4Video, Vript, and Tarsier-585K by arguing that motion-detail balance is a measurable property of supervision rather than an incidental by-product of scale. Likewise, it positions OwlCap against systems such as Tarsier and VideoCap-R1 by claiming that a motion-specialized generator or a one-sided reward does not adequately solve the balance problem. This suggests a broader research claim: in video captioning, caption quality depends not only on model capacity but also on how motion and detail are jointly represented in the training target and in the optimization signal.

A common simplification would be to treat OwlCap as only a benchmark-improvement wrapper around Qwen2.5-VL-7B. The reported evidence argues against that reading. The paper shows compatibility with Qwen2-VL-7B, Qwen2.5-VL-3B, and Qwen2.5-VL-7B, and in all cases OwlCap improves both VDC and DREAM-1K. This suggests that the method is not tied to a single Qwen-family base, but to a particular combination of motion-detail-balanced supervision and unit-level bidirectional reward modeling.