HMD-270K: Balanced Video Captioning Dataset

• HMD-270K is a comprehensive video-caption dataset that provides balanced supervision by capturing both temporal dynamics and static visual details.
• It utilizes a two-stage pipeline—Motion-Detail Fusion and Fine-Grained Examination—to sequentially generate and validate enriched captions.
• The dataset demonstrates high performance with an MDB of 0.68 and significant improvements in motion and detail accuracy in downstream tasks.

Harmonizing Motion-Detail 270K (HMD-270K) is a large-scale video-caption dataset introduced for video captioning as part of “OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Reward” (Zhong et al., 26 Aug 2025). It contains 270K video-caption pairs and is designed to supply captions that are simultaneously motion-complete and detail-complete, addressing what the paper characterizes as a persistent motion-detail imbalance in existing video captioning data and models. Within the OwlCap framework, HMD-270K functions as the supervised data substrate for motion-detail balanced caption generation, while subsequent reinforcement learning is used to improve correctness and completeness (Zhong et al., 26 Aug 2025).

1. Definition and problem setting

HMD-270K is defined as a dataset for video captioning, not for motion capture, pose estimation, or head-mounted sensing. Its central design objective is to provide captions that jointly encode temporal dynamics, actions, and events together with static visual attributes, objects, environment, scene composition, and appearance (Zhong et al., 26 Aug 2025). The motivating claim is that existing datasets and models often overemphasize one semantic axis while underrepresenting the other, producing captions that are incomplete for both video understanding and downstream video generation tasks.

The paper formalizes this issue as motion-detail imbalance. In its diagnosis, some systems are biased toward motion-oriented description, while others are biased toward visual detail. HMD-270K is introduced specifically as a data-level intervention against that imbalance, rather than as a new backbone architecture. This makes the dataset unusual in that its novelty lies less in raw scale alone than in the explicit attempt to balance two captioning objectives that are often treated asymmetrically (Zhong et al., 26 Aug 2025).

The dataset is therefore not simply a generic synthetic caption corpus. Its stated purpose is to provide motion-detail balanced supervision. This framing is central to the paper’s argument: HMD-270K is intended to shape model behavior before any reward-based post-training is applied (Zhong et al., 26 Aug 2025).

2. Construction pipeline

HMD-270K is built through a two-stage pipeline consisting of Motion-Detail Fusion (MDF) and Fine-Grained Examination (FGE) (Zhong et al., 26 Aug 2025).

In the MDF stage, caption construction is explicitly sequential. A video is first processed by Tarsier, which is prompted to produce a motion-focused caption. That motion-oriented caption, together with the original video, is then provided to Qwen2.5-VL-72B, which is prompted to supplement static details. The resulting fused caption is intended to preserve the temporal scaffold supplied by Tarsier while enriching it with detail-oriented content from Qwen2.5-VL-72B. The paper describes this as using distinct MLLMs to separately extract motion and detail information and then integrating them into a fused caption (Zhong et al., 26 Aug 2025).

The FGE stage serves as a verification and filtering mechanism. The paper argues that direct holistic caption validation is ambiguous because a video caption attempts to summarize many elements at once. Its solution is to decompose each fused caption into minimal semantic units and validate those units individually against the video using InternVL2.5-78B. This produces a unit accuracy (Unit Acc) score for each video-caption pair, and the filtering rule retains only samples with at least 90% Unit Acc (Zhong et al., 26 Aug 2025).

The paper explicitly justifies the 90% threshold rather than 100% by citing two difficult cases: subjective descriptions, such as “exuding a warm atmosphere,” and ambiguous target descriptions, such as “speaking” versus “arguing.” This suggests that FGE is designed as a high-precision filter with limited fault tolerance rather than as an absolute semantic verifier.

The source videos are drawn from open-source datasets “following Tarsier2-Recap-585K,” including WebVid-10M, ActivityNet, and Ego4D, covering domains such as wildlife, movies, cooking, sports, news, and TV programs. The paper also states that videos used in the evaluation benchmarks are not included in HMD-270K (Zhong et al., 26 Aug 2025).

3. Annotation properties and balance statistics

The paper characterizes HMD-270K primarily through caption density, motion density, and a joint balance metric called Motion-Detail Balance (MDB) (Zhong et al., 26 Aug 2025). Caption length is reported to follow a normal distribution with a mean length of approximately 200 words, indicating that the dataset targets dense captioning rather than short-sentence description.

The principal dataset-level statistics reported for HMD-270K are:

• 13.9 words per second
• 2.0 verbs per second
• MDB = 0.68 (Zhong et al., 26 Aug 2025)

MDB is defined as

$$\text{MDB} = \left(1 - \frac{w - v}{w + v}\right) \cdot \log(w + 1),$$

where $w$ is the number of caption words per second and $v$ is the number of verbs per second (Zhong et al., 26 Aug 2025). In the paper’s interpretation, the first factor measures the proportion of motion-related words relative to total words, while the logarithmic term measures detail level.

The paper compares HMD-270K against Panda-70M, ShareGPT4Video, VRipt, and Tarsier-585K, reporting that HMD-270K attains the highest reported MDB among these datasets. Its comparison figures are 1.5 / 0.2 / 0.22 for Panda-70M, 10.7 / 1.3 / 0.54 for ShareGPT4Video, 13.0 / 1.4 / 0.50 for VRipt, 5.2 / 0.8 / 0.49 for Tarsier-585K, and 13.9 / 2.0 / 0.68 for HMD-270K, in the order words/sec / verbs/sec / MDB (Zhong et al., 26 Aug 2025). These values are the paper’s main quantitative evidence that HMD-270K is more balanced and comprehensive than the compared resources.

The paper does not state a train/val/test split for HMD-270K, nor does it give exact duration statistics, exact per-source counts, a license, or a vocabulary-size summary. It also does not report whether each video has multiple reference captions. These omissions are material for reproducibility and dataset governance, and they delimit what can be established directly from the paper (Zhong et al., 26 Aug 2025).

4. Data quality control and verification

The dataset’s quality-control logic is closely tied to its unit-based annotation philosophy. The paper reports an MDF/FGE ablation on 4K videos in which Tarsier alone achieves Motion Score 7.32 and Detail Score 6.15, while Qwen2.5-VL-72B alone achieves Motion Score 6.99 and Detail Score 6.36. After fusion, MDF reaches Motion Score 7.80, Detail Score 7.02, and Unit Acc 78.93; after examination, FGE reaches Motion Score 7.92, Detail Score 7.13, and Unit Acc 96.54 (Zhong et al., 26 Aug 2025).

These numbers indicate two distinct roles. MDF improves semantic richness relative to the source captioners, while FGE sharply improves factual reliability as measured by unit-level validation. The increase from 78.93 to 96.54 in Unit Acc is the clearest quantitative indicator that the filtering stage is doing more than cosmetic cleanup.

The paper also mentions three auxiliary quality checks in the appendix: a cross-model consistency check comparing InternVL2.5-78B and GPT-4o on 500 FGE samples, a threshold sensitivity analysis, and human re-evaluation of 100 automatically rejected samples (Zhong et al., 26 Aug 2025). However, the numerical outcomes of these checks are not reported in the summarized paper content. A plausible implication is that the authors considered judge-model bias and threshold robustness to be nontrivial concerns, but the available evidence in the main report remains centered on Unit Acc and benchmark performance.

5. Function within OwlCap training

HMD-270K is the dataset used for the supervised fine-tuning (SFT) stage of OwlCap, whose base model is Qwen2.5-VL-7B (Zhong et al., 26 Aug 2025). In this role, HMD-270K is intended to teach motion-detail balance before reinforcement learning is applied. The SFT setup reported in the paper uses 2 FPS frame sampling, Adam, learning rate 1e-5, global batch size 64, 1 epoch, and approximately 26 hours of training (Zhong et al., 26 Aug 2025).

After SFT, HMD-270K also serves as the source pool for post-training sample selection. The paper states that 12K samples are screened from HMD-270K, all with 100% unit accuracy; each sample is inferred 8 times by the stage-one model; Caption Set Equivalence Reward (CSER) is used to compute within-group variance; and the top 50% highest-variance samples are retained, yielding a final 6K-sample GRPO training set (Zhong et al., 26 Aug 2025). This means HMD-270K is not only the SFT corpus but also the basis for hard-example selection during post-training.

CSER is aligned with the same unit-based semantics used in FGE. It decomposes predicted captions into units $U_1,\dots,U_n$ and ground-truth captions into fact units $F_1,\dots,F_m$, using Qwen3-32B for unit decomposition and relevance assessment (Zhong et al., 26 Aug 2025). The paper defines a correctness score

$$S_{correctness} = \frac{\sum_{i=1}^{n} \mathbb{I}(U_i \in C_{gt})}{n},$$

and a completeness score

$$S_{completeness} = \frac{\sum_{j=1}^{m} \mathbb{I}(F_j \in C_{pred})}{m}.$$

The final reward is

$$\text{Reward} = S_{\text{format}} + S_{\text{correctness}} + S_{\text{completeness}}.$$

This establishes a direct continuity between HMD-270K’s filtering logic and OwlCap’s optimization logic: the same atomic semantic view used to curate the dataset is extended into reinforcement learning (Zhong et al., 26 Aug 2025).

6. Empirical significance, scope, and terminological disambiguation

The paper’s ablations indicate that HMD-270K is not an incidental component. In the component study, the baseline without HMD SFT or CSER obtains 42.7 VDC Acc, 2.21 VDC Score, 30.1 DREAM F1, and 29.7 DREAM Recall. Adding HMD SFT alone raises these to 44.8, 2.31, 31.9, and 32.1. The full configuration with HMD SFT, correctness, and completeness reaches 46.9, 2.43, 34.7, and 35.3 (Zhong et al., 26 Aug 2025). The paper therefore presents HMD-270K as the foundation on which CSER-based gains become most effective.

At the benchmark level, OwlCap is reported to improve over the base model by +4.2 Acc on the detail-focused VDC benchmark and +4.6 F1 on the motion-focused DREAM-1K benchmark (Zhong et al., 26 Aug 2025). Since the evaluation videos are explicitly excluded from HMD-270K, these gains are presented as evidence that the dataset transfers beyond its own training distribution.

A recurrent source of confusion is the acronym HMD. In HMD-270K, it abbreviates Harmonizing Motion-Detail (Zhong et al., 26 Aug 2025). In several adjacent literatures, however, HMD denotes head-mounted device. This is the case in HMD$^2$, which concerns environment-aware motion generation from a single head-mounted device and uses the Nymeria dataset rather than any resource called HMD-270K (Guzov et al., 2024); in EMHI, a multimodal egocentric human motion dataset with HMD and body-worn IMUs that does not mention HMD-270K (Fan et al., 2024); and in EventEgo3D++, where HMD likewise means head-mounted device and the introduced datasets are EE3D-S, EE3D-R, and EE3D-W, not HMD-270K (Millerdurai et al., 11 Feb 2025). HMD-270K thus belongs to the video-captioning literature, not the egocentric motion-capture literature, despite the acronym overlap.

The paper states that “The HMD-270K dataset and OwlCap model will be publicly released” (Zhong et al., 26 Aug 2025). It does not, however, specify a repository URL, release date, or license in the reported text. Accordingly, HMD-270K is best understood as a dataset whose conceptual contribution and headline statistics are explicit, while several practical release and governance details remain unspecified in the paper’s current description.