---
title: 'HMD-270K: Balanced Video Captioning Dataset'
url: https://www.emergentmind.com/topics/harmonizing-motion-detail-270k-hmd-270k
type: topic
---

# HMD-270K: Balanced Video Captioning Dataset

Harmonizing Motion-Detail 270K (HMD-270K) is a large-scale video-caption dataset introduced for video captioning as part of “OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Reward” [2508.18634]. It contains **270K video-caption pairs** and is designed to supply captions that are simultaneously **motion-complete** and **detail-complete**, addressing what the paper characterizes as a persistent **motion-detail imbalance** in existing video captioning data and models. Within the OwlCap framework, HMD-270K functions as the supervised data substrate for motion-detail balanced caption generation, while subsequent reinforcement learning is used to improve correctness and completeness [2508.18634].

## 1. Definition and problem setting

HMD-270K is defined as a dataset for **video captioning**, not for motion capture, pose estimation, or head-mounted sensing. Its central design objective is to provide captions that jointly encode **temporal dynamics, actions, and events** together with **static visual attributes, objects, environment, scene composition, and appearance** [2508.18634]. The motivating claim is that existing datasets and models often overemphasize one semantic axis while underrepresenting the other, producing captions that are incomplete for both video understanding and downstream video generation tasks.

The paper formalizes this issue as **motion-detail imbalance**. In its diagnosis, some systems are biased toward motion-oriented description, while others are biased toward visual detail. HMD-270K is introduced specifically as a data-level intervention against that imbalance, rather than as a new backbone architecture. This makes the dataset unusual in that its novelty lies less in raw scale alone than in the explicit attempt to balance two captioning objectives that are often treated asymmetrically [2508.18634].

The dataset is therefore not simply a generic synthetic caption corpus. Its stated purpose is to provide **motion-detail balanced supervision**. This framing is central to the paper’s argument: HMD-270K is intended to shape model behavior before any reward-based post-training is applied [2508.18634].

## 2. Construction pipeline

HMD-270K is built through a **two-stage pipeline** consisting of **Motion-Detail Fusion (MDF)** and **Fine-Grained Examination (FGE)** [2508.18634].

In the **MDF** stage, caption construction is explicitly sequential. A video is first processed by **Tarsier**, which is prompted to produce a **motion-focused caption**. That motion-oriented caption, together with the original video, is then provided to **Qwen2.5-VL-72B**, which is prompted to supplement **static details**. The resulting fused caption is intended to preserve the temporal scaffold supplied by Tarsier while enriching it with detail-oriented content from Qwen2.5-VL-72B. The paper describes this as using distinct MLLMs to separately extract motion and detail information and then integrating them into a fused caption [2508.18634].

The **FGE** stage serves as a verification and filtering mechanism. The paper argues that direct holistic caption validation is ambiguous because a video caption attempts to summarize many elements at once. Its solution is to decompose each fused caption into **minimal semantic units** and validate those units individually against the video using **InternVL2.5-78B**. This produces a **unit accuracy (Unit Acc)** score for each video-caption pair, and the filtering rule retains only samples with at least **90% Unit Acc** [2508.18634].

The paper explicitly justifies the **90%** threshold rather than **100%** by citing two difficult cases: **subjective descriptions**, such as “exuding a warm atmosphere,” and **ambiguous target descriptions**, such as “speaking” versus “arguing.” This suggests that FGE is designed as a high-precision filter with limited fault tolerance rather than as an absolute semantic verifier.

The source videos are drawn from open-source datasets “following Tarsier2-Recap-585K,” including **WebVid-10M**, **ActivityNet**, and **Ego4D**, covering domains such as **wildlife**, **movies**, **cooking**, **sports**, **news**, and **TV programs**. The paper also states that videos used in the evaluation benchmarks are **not included** in HMD-270K [2508.18634].

## 3. Annotation properties and balance statistics

The paper characterizes HMD-270K primarily through caption density, motion density, and a joint balance metric called **Motion-Detail Balance (MDB)** [2508.18634]. Caption length is reported to follow a normal distribution with a **mean length of approximately 200 words**, indicating that the dataset targets dense captioning rather than short-sentence description.

The principal dataset-level statistics reported for HMD-270K are:

- **13.9 words per second**
- **2.0 verbs per second**
- **MDB = 0.68** [2508.18634]

MDB is defined as

$$
\text{MDB} = \left(1 - \frac{w - v}{w + v}\right) \cdot \log(w + 1),
$$

where \(w\) is the number of caption words per second and \(v\) is the number of verbs per second [2508.18634]. In the paper’s interpretation, the first factor measures the proportion of motion-related words relative to total words, while the logarithmic term measures detail level.

The paper compares HMD-270K against **Panda-70M**, **ShareGPT4Video**, **VRipt**, and **Tarsier-585K**, reporting that HMD-270K attains the highest reported MDB among these datasets. Its comparison figures are **1.5 / 0.2 / 0.22** for Panda-70M, **10.7 / 1.3 / 0.54** for ShareGPT4Video, **13.0 / 1.4 / 0.50** for VRipt, **5.2 / 0.8 / 0.49** for Tarsier-585K, and **13.9 / 2.0 / 0.68** for HMD-270K, in the order **words/sec / verbs/sec / MDB** [2508.18634]. These values are the paper’s main quantitative evidence that HMD-270K is more balanced and comprehensive than the compared resources.

The paper does not state a **train/val/test split** for HMD-270K, nor does it give exact **duration statistics**, exact **per-source counts**, a **license**, or a vocabulary-size summary. It also does not report whether each video has multiple reference captions. These omissions are material for reproducibility and dataset governance, and they delimit what can be established directly from the paper [2508.18634].

## 4. Data quality control and verification

The dataset’s quality-control logic is closely tied to its unit-based annotation philosophy. The paper reports an MDF/FGE ablation on **4K videos** in which **Tarsier** alone achieves **Motion Score 7.32** and **Detail Score 6.15**, while **Qwen2.5-VL-72B** alone achieves **Motion Score 6.99** and **Detail Score 6.36**. After fusion, **MDF** reaches **Motion Score 7.80**, **Detail Score 7.02**, and **Unit Acc 78.93**; after examination, **FGE** reaches **Motion Score 7.92**, **Detail Score 7.13**, and **Unit Acc 96.54** [2508.18634].

These numbers indicate two distinct roles. MDF improves semantic richness relative to the source captioners, while FGE sharply improves factual reliability as measured by unit-level validation. The increase from **78.93** to **96.54** in Unit Acc is the clearest quantitative indicator that the filtering stage is doing more than cosmetic cleanup.

The paper also mentions three auxiliary quality checks in the appendix: a **cross-model consistency check** comparing InternVL2.5-78B and GPT-4o on **500 FGE samples**, a **threshold sensitivity analysis**, and **human re-evaluation of 100 automatically rejected samples** [2508.18634]. However, the numerical outcomes of these checks are not reported in the summarized paper content. A plausible implication is that the authors considered judge-model bias and threshold robustness to be nontrivial concerns, but the available evidence in the main report remains centered on Unit Acc and benchmark performance.

## 5. Function within OwlCap training

HMD-270K is the dataset used for the **supervised fine-tuning (SFT)** stage of OwlCap, whose base model is **Qwen2.5-VL-7B** [2508.18634]. In this role, HMD-270K is intended to teach motion-detail balance before reinforcement learning is applied. The SFT setup reported in the paper uses **2 FPS** frame sampling, **Adam**, learning rate **1e-5**, global batch size **64**, **1 epoch**, and approximately **26 hours** of training [2508.18634].

After SFT, HMD-270K also serves as the source pool for post-training sample selection. The paper states that **12K samples** are screened from HMD-270K, all with **100% unit accuracy**; each sample is inferred **8 times** by the stage-one model; **Caption Set Equivalence Reward (CSER)** is used to compute **within-group variance**; and the top **50% highest-variance samples** are retained, yielding a final **6K-sample** GRPO training set [2508.18634]. This means HMD-270K is not only the SFT corpus but also the basis for hard-example selection during post-training.

CSER is aligned with the same unit-based semantics used in FGE. It decomposes predicted captions into units \(U_1,\dots,U_n\) and ground-truth captions into fact units \(F_1,\dots,F_m\), using **Qwen3-32B** for unit decomposition and relevance assessment [2508.18634]. The paper defines a **correctness score**

$$
S_{correctness} = \frac{\sum_{i=1}^{n} \mathbb{I}(U_i \in C_{gt})}{n},
$$

and a **completeness score**

$$
S_{completeness} = \frac{\sum_{j=1}^{m} \mathbb{I}(F_j \in C_{pred})}{m}.
$$

The final reward is

$$
\text{Reward} = S_{\text{format}} + S_{\text{correctness}} + S_{\text{completeness}}.
$$

This establishes a direct continuity between HMD-270K’s filtering logic and OwlCap’s optimization logic: the same atomic semantic view used to curate the dataset is extended into reinforcement learning [2508.18634].

## 6. Empirical significance, scope, and terminological disambiguation

The paper’s ablations indicate that HMD-270K is not an incidental component. In the component study, the baseline without HMD SFT or CSER obtains **42.7** VDC Acc, **2.21** VDC Score, **30.1** DREAM F1, and **29.7** DREAM Recall. Adding **HMD SFT alone** raises these to **44.8**, **2.31**, **31.9**, and **32.1**. The full configuration with **HMD SFT**, **correctness**, and **completeness** reaches **46.9**, **2.43**, **34.7**, and **35.3** [2508.18634]. The paper therefore presents HMD-270K as the foundation on which CSER-based gains become most effective.

At the benchmark level, OwlCap is reported to improve over the base model by **+4.2 Acc** on the detail-focused **VDC** benchmark and **+4.6 F1** on the motion-focused **DREAM-1K** benchmark [2508.18634]. Since the evaluation videos are explicitly excluded from HMD-270K, these gains are presented as evidence that the dataset transfers beyond its own training distribution.

A recurrent source of confusion is the acronym **HMD**. In HMD-270K, it abbreviates **Harmonizing Motion-Detail** [2508.18634]. In several adjacent literatures, however, **HMD** denotes **head-mounted device**. This is the case in **HMD\(^2\)**, which concerns environment-aware motion generation from a single head-mounted device and uses the **Nymeria** dataset rather than any resource called HMD-270K [2409.13426]; in **EMHI**, a multimodal egocentric human motion dataset with HMD and body-worn IMUs that does not mention HMD-270K [2408.17168]; and in **EventEgo3D++**, where HMD likewise means head-mounted device and the introduced datasets are **EE3D-S**, **EE3D-R**, and **EE3D-W**, not HMD-270K [2502.07869]. HMD-270K thus belongs to the video-captioning literature, not the egocentric motion-capture literature, despite the acronym overlap.

The paper states that **“The HMD-270K dataset and OwlCap model will be publicly released”** [2508.18634]. It does not, however, specify a repository URL, release date, or license in the reported text. Accordingly, HMD-270K is best understood as a dataset whose conceptual contribution and headline statistics are explicit, while several practical release and governance details remain unspecified in the paper’s current description.

Source: https://www.emergentmind.com/topics/harmonizing-motion-detail-270k-hmd-270k