RealAction-5K: Real-World Video Dataset

• RealAction-5K is a compact, high-fidelity dataset featuring 5,000 annotated videos of daily human activities designed to improve contrastive preference learning.
• It employs a rigorous two-stage filtering process with automatic and manual curation to ensure natural motion, precise context, and detailed text-video alignment.
• The dataset powers RealDPO training by providing real vs. synthetic samples, leading to significant improvements in motion quality and text-to-video coherence.

RealAction-5K is a compact, high-fidelity dataset comprised of 5,000 real-world video clips depicting diverse daily human activities, created to serve as a "win" set for contrastive preference learning within the RealDPO alignment paradigm for video generative models. Prior work identified a persistent deficiency in motion quality—particularly naturalness, smoothness, and context alignment—in synthesized videos, motivating a shift from scale-oriented large benchmarks toward precise, scrutinized corpora. RealAction-5K operationalizes the “less is more” principle by offering a carefully curated set of short video exemplars, each paired with a rich textual description, to expose and correct model motion errors via real versus synthetic comparison (Cheng et al., 16 Oct 2025).

1. Motivation and Design Objectives

The principal impetus for RealAction-5K is to supply RealDPO with real data exemplars of human actions that possess precise, contextually grounded, smooth, and natural motion dynamics. The dataset counters the trend of assembling large but noisy video corpora by focusing on annotation precision and motion veracity. Design objectives include:

• Enabling accurate preference-based learning by providing "win" examples that contrast sharply with synthetic model errors.
• Covering a broad spectrum of common human actions within realistic, contextually rich settings.
• Ensuring text-video alignment by generating detailed, context-specific captions that encapsulate actors, objects, and nuanced motion aspects.

This design enables RealDPO to develop improved discrimination of motion realism and iteratively refine motion synthesis performances.

2. Data Collection Pipeline and Quality Assurance

RealAction-5K is sourced through keyword-based automated scraping of public-domain and royalty-free video repositories such as Pexels.com. Keyword queries target approximately 10 themes, including but not limited to sports, walking, eating/drinking, dancing, household chores, musical activity, pet interaction, and social gestures.

A two-stage filtering pipeline is deployed:

1. Automatic Filtering: Qwen2-VL (VideoLLM) is used to exclude low-resolution, shaky, off-topic, or otherwise deficient clips.
2. Manual Curation: Human annotators verify each retained clip for (1) thematic coherence, (2) absence of misleading content, and (3) visual clarity.

Captions are automatically generated for each selected video by LLaVA-Video. These textual annotations provide granular descriptions of the central action, associated actors, and contextual objects, forming a vital component of the metadata for downstream contrastive learning.

3. Dataset Composition and Statistics

A total of 5,000 video clips comprise RealAction-5K. The dataset does not report mean duration, frame rate, or spatial resolution; typical practice in the field is 3–10 seconds per clip, but the exact duration distribution is unspecified. Each video is assigned to an activity category, with approximately 10 categories represented, each constituting between 5–15% of the dataset. Figure 1(d) in (Cheng et al., 16 Oct 2025) visualizes this distribution with a bar chart. Caption length distribution and token frequency (word-cloud, histogram) are presented in Figures 3(e) and 3(c), respectively. No custom richness or motion-specific quantitative metrics are defined.

Attribute	Value or Method	Notes
Total clips	5,000	Exact figure given
Activity categories	≈10, 5–15% each	Examples only, full list not given
Annotation modality	Caption (LLaVA-Video, free-text)	No keypoint/skeleton labels

This suggests deliberate compactness and curatorial rigor take precedence over scale or breadth in class coverage.

4. Activity Taxonomy and Annotation Schema

The dataset is organized around broad "daily activity themes." Representative classes include:

• Sports (e.g., ball-kicking, frisbee-throwing)
• Eating/drinking (e.g., cutting food, sipping)
• Walking/running
• Dancing/jumping
• Playing musical instruments
• Interacting with pets (e.g., petting a dog)
• Household chores (e.g., sweeping, folding clothes)
• Social gestures (e.g., handshake, hug)
• Miscellaneous (e.g., reading, writing)

Annotations are restricted to free-text captions—no pose, keypoint, or skeleton labels are provided. Caption detail acts as an implicit motion annotation. The per-clip metadata schema is as follows:

{ "video_id": string, "filename": string, "duration_s": float (optional), "caption": string, "category": string, "filter_passed_by": "Qwen2-VL"/"manual", ... }

A plausible implication is that the annotation method can be adapted for future expansions with minimal overhead, provided high-quality filtering and captioning models are available.

5. Preprocessing, Format, and Data Accessibility

The pipeline incorporates both automated (Qwen2-VL) and manual curation, followed by captioning. Each video is encoded via the model's VAE encoder to extract the latent tensor $X_0$ for direct use in RealDPO training. Further preprocessing—such as cropping, normalization, or spatial–temporal alignment—is not specified.

Final on-disk format:

• Raw video files (.mp4, .avi)
• One JSON metadata file per video
• Directory containing captions as text files

This format is optimized for straightforward integration into machine learning pipelines and direct pairing with synthetic negatives for preference learning.

6. Use in Contrastive Alignment and Empirical Impact

Within RealDPO, RealAction-5K serves exclusively as the real ("win") data in training, with negative ("lose") samples produced by model inference under matched text conditions. These pairs ($X^w$, $X^l$) are input to a DPO loss designed for motion synthesis refinement:

$$L_{RealDPO} = -\mathbb{E} \left[ \log \sigma \left( -\beta T \omega(\lambda_t) \left( \| X^w - \hat X^w \|^2 - \| X^w - \tilde X^w \|^2 - (\| X^l - \hat X^l \|^2 - \| X^l - \tilde X^l \|^2 ) \right) \right) \right]$$

Replacing synthetic-vs-synthetic with real-vs-synthetic comparison leads to a stronger supervisory signal, facilitating more reliable correction of model behavior. Empirical results demonstrate marked improvements: a 73.33% human preference rate for motion quality versus 65–67% for other methods, and a +4–8 point gain in text, visual, and motion realism metrics. This suggests that small, high-fidelity, real-annotated datasets can exert a disproportionate influence on generative alignment quality compared to larger synthetic corpora.

7. Limitations, Extensions, and Prospects

The dataset does not provide standard train/validation/test splits; it is typically used in toto as the positive class, with separate held-out data (RealAction-TestBench) reserved for model evaluation. Absence of keypoint, skeleton, or frame-level atomic annotations restricts utility for research in pose estimation or fine-grained motion parsing, focusing its value exclusively on preference-based contrastive learning. No explicit quantification of dataset "richness" or motion smoothness via numeric scores is attempted; visualization remains the primary quality assurance mechanism.

A plausible implication is that extensions or augmentations of RealAction-5K—with further activity classes, fine-grained labels, or higher-resolution channel-specific data—could enhance its applicability, provided the guiding principle of annotation quality over scale is retained.

---

RealAction-5K represents a paradigmatic shift in video generative model training, favoring compact, curated, real-data-centric corpora to drive preference-based contrastive alignment workflows, underpinning the RealDPO method’s gains in motion realism, smoothness, and text-to-video coherence (Cheng et al., 16 Oct 2025).