VideoMind: Omni-Modal Video Dataset

• VideoMind is an omni-modal video dataset with 103K clips that integrates multi-layer textual annotations (factual, abstract, and intent) to enable deep-cognitive video understanding.
• It employs a hierarchical annotation model using a 6W schema and Chain-of-Thought prompting to combine observable features with inferred motivations.
• The benchmark supports hybrid-cognitive retrieval tasks, revealing performance differences across factual, abstract, and intent-based queries in state-of-the-art video-language models.

VideoMind denotes a 2025 line of research in video understanding whose primary referent is an omni-modal video dataset with intent grounding for deep-cognitive video understanding, while the same name is also used by a separate Chain-of-LoRA video-language agent for long video reasoning; a related but distinct framework, VideoMindPalace, addresses long video analysis through environment-grounded semantic graphs (Yang et al., 24 Jul 2025, Liu et al., 17 Mar 2025, Huang et al., 8 Jan 2025). In its dataset sense, VideoMind is a video-centric omni-modal corpus comprising 103,000 video clips, including a 3,000-sample gold-standard test set, with layered textual descriptions that progress from factual observation to abstract summarization and finally to intent expression. Its central contribution is to move beyond surface-level captioning by introducing intent expressions that require contextual integration across the entire video and are not directly observable (Yang et al., 24 Jul 2025).

1. Corpus composition and scope

The dataset "VideoMind: An Omni-Modal Video Dataset with Intent Grounding for Deep-Cognitive Video Understanding" is organized around 103,000 video samples, of which 100,000 constitute the training pool and 3,000 form the held-out gold-standard test set, "VideoMind-3K" (Yang et al., 24 Jul 2025). Each clip is accompanied by raw video with twelve sampled key frames, an audio track, and multi-perspective text streams: ASR transcript, OCR text, the uploader’s manual text if any, and Qwen2.5-Omni–generated factual, abstract, and intent layers. In addition, each sample carries semantic "6W" tags: who, where, when, what, how, and why.

The corpus contains over 22 million words, averaging approximately 225 words per sample, which situates it as a densely annotated resource rather than a caption-only benchmark (Yang et al., 24 Jul 2025). The paper describes it, to its knowledge, as the first large-scale omni-modal video corpus whose annotations go well beyond surface-level captioning to include deep-cognitive intent expressions. The absence of a separate released validation split is explicit: researchers typically hold out a portion of the 100,000 training clips if needed.

This scope matters because the dataset is designed not merely for generic video-text alignment, but for deep video content cognition and enhanced multi-modal feature representation. A plausible implication is that VideoMind is intended to pressure-test whether video-language systems can align observable evidence, fused summaries, and inferred motivations within a single benchmark regime.

2. Hierarchical annotation model and the 6W schema

VideoMind structures text into three hierarchical layers of increasing cognitive depth: factual, abstract, and intent (Yang et al., 24 Jul 2025). All text is generated from an mLLM through a Chain-of-Thought protocol. The factual layer records directly observable or extractable modalities without inference. It contains five non-overlapping sub-fields: visual, audio, OCR, ASR, and raw text. Its typical length is approximately 143 words. In effect, this layer separates what is seen, what is heard, what text appears on screen, what is spoken, and what metadata the uploader supplied.

The abstract layer fuses those five streams into a concise narrative summary, formatted as a single paragraph of approximately 38 words covering subject, place, time, and event (Yang et al., 24 Jul 2025). It is therefore the first stage at which cross-stream integration becomes explicit. The paper presents this layer as a compressed semantic synthesis rather than a direct transcription of any one modality.

The intent layer speculates on the underlying motivation of both the uploader and the on-screen character. Its expression rule is fixed: $[\text{subject}]$ aims to $[\text{intent}]$ by $[\text{action}]$. The paper gives the example, "The blacksmith aims to demonstrate traditional forging techniques by hammering red-hot iron on an anvil." The combined length of the two intent statements is typically about 43 words. This layer is the defining feature of the dataset because intent is treated as a deep-cognitive expression that must be inferred from context rather than directly observed (Yang et al., 24 Jul 2025).

The 6W schema spans all three layers. Subject, place, time, and event are automatically extracted from the abstract layer by prompting Qwen2.5-vl to highlight nouns of each category, while action and intent are parsed from the fixed-format intent sentences. This gives the dataset a regularized semantic interface for downstream recognition tasks.

A recurrent misconception is to treat the intent layer as ordinary captioning. The paper explicitly positions it otherwise: intent expressions require contextual integration across the entire video and are not directly observable (Yang et al., 24 Jul 2025). That distinction is foundational to how the benchmark is constructed and interpreted.

3. Chain-of-Thought prompting and intent grounding

The generation pipeline proceeds in staged form under prompting guidance with Qwen2.5-Omni and Qwen2.5-vl (Yang et al., 24 Jul 2025). First, the system generates the five sub-fields of the factual layer. Second, it produces the abstract summary while simultaneously extracting the 6W tags. Third, it triggers intent speculation in two turns, one from the uploader’s perspective and one from the character’s perspective, enforcing the fixed schema "$[\text{subject}]$ aims to $[\text{intent}]$ by $[\text{action}]$."

The dual prompts are central to the definition of intent. Role A asks, "Why did I post this video?" from the uploader’s perspective. Role B asks, "Why do I, as the blacksmith, perform this action?" from the character’s perspective (Yang et al., 24 Jul 2025). This division makes intent a two-perspective construct rather than a single latent variable.

The paper also describes a pre- and post-validation procedure. In pre-validation, a second mLLM re-speculates intent, and the two intent keywords are compared via cosine similarity of their embeddings; they must exceed a threshold $\theta$ to pass. In post-validation, each intent text is converted back into a 10 sec video using Wan2.1, and two human experts judge whether the generated clip reasonably matches the claimed intent (Yang et al., 24 Jul 2025). No explicit LaTeX formula appears in the paper beyond the template rule, though the rule is formalized in the details as

$$\text{intent\_text} = \langle \text{subject} \rangle \oplus \text{“aims to”} \oplus \langle \text{intent} \rangle \oplus \text{“by”} \oplus \langle \text{action} \rangle.$$

These procedures clarify a second common misunderstanding. VideoMind does not claim that intent is directly annotated from visual evidence alone. Rather, intent is generated through step-by-step reasoning and then filtered through model-based and human validation. This suggests a benchmark philosophy in which higher-level semantics are operationalized through controlled inference rather than simple perceptual labeling.

4. Benchmark design and hybrid-cognitive retrieval

VideoMind-3K is the evaluation substrate for hybrid-cognitive retrieval at all three cognitive depths: factual, abstract, and intent (Yang et al., 24 Jul 2025). Two retrieval scenarios are defined. In text-to-video retrieval, queries may be factual, abstract, intent, or "any," where the system chooses the best matching layer. For $N=3000$, the metrics are

$$R@K = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}(\text{rank}_i \le K),$$

and

$[\text{intent}]$0

In video-to-text retrieval, the ground truth for each clip is the triplet of factual, abstract, and intent texts. The benchmark defines "Hit any"@K, meaning at least one correct layer appears in the top-$[\text{intent}]$1 retrievals, and "Hit all"@K, meaning all three layers appear within top-$[\text{intent}]$2 with $[\text{intent}]$3 (Yang et al., 24 Jul 2025). It also defines

$[\text{intent}]$4

$[\text{intent}]$5

and

$[\text{intent}]$6

This benchmark design is technically significant because it avoids collapsing all text supervision into a single caption channel. Instead, it tests whether embeddings preserve correspondence across observable details, compressed semantic abstraction, and inferred motivation. The "any" setting further probes whether a model can opportunistically align at whichever cognitive layer is most retrievable.

5. Empirical results and what they show

The released evaluations cover five video-language foundations on VideoMind-3K, including InternVideo, UMT-L, CLIP-VIP, mPLUG-2, and VAST, with no test samples used during training (Yang et al., 24 Jul 2025). The clearest reported pattern is depth-sensitive degradation. For InternVideo in text-to-video retrieval, factual-to-video yields $[\text{intent}]$7, $[\text{intent}]$8, $[\text{intent}]$9, and $[\text{action}]$0; abstract-to-video yields $[\text{action}]$1 and $[\text{action}]$2; intent-to-video drops to $[\text{action}]$3, $[\text{action}]$4, $[\text{action}]$5, and $[\text{action}]$6; and the "any" setting gives $[\text{action}]$7 with $[\text{action}]$8 (Yang et al., 24 Jul 2025).

The broader trend across UMT-L, CLIP-VIP, mPLUG-2, and VAST is reported as strong retrieval on factual queries, moderate retrieval on abstract queries, and collapse on intent queries. In summary form, the paper characterizes these models as having greater than $[\text{action}]$9 $[\text{subject}]$0 with $[\text{subject}]$1 on factual retrieval, around $[\text{subject}]$2 $[\text{subject}]$3 with $[\text{subject}]$4 to $[\text{subject}]$5 on abstract retrieval, and less than $[\text{subject}]$6 $[\text{subject}]$7 with $[\text{subject}]$8 on intent retrieval (Yang et al., 24 Jul 2025).

For video-to-text retrieval, InternVideo reaches $[\text{subject}]$9, $[\text{intent}]$0, $[\text{intent}]$1, $[\text{intent}]$2, and $[\text{intent}]$3 (Yang et al., 24 Jul 2025). The paper emphasizes that even when one layer is found, the others rank poorly, with LowestR often greater than 200 for current models. This suggests that existing embeddings can anchor onto at least one semantic surface of a clip, but do not reliably maintain coherent alignment across the full cognitive stack.

The corpus-level analysis also includes word-cloud visualizations of the most frequent intents, actions, subjects, places, and audio styles. Together with Tables 2 and 3, these visualizations frame the benchmark as a diagnostic for depth of understanding rather than a single-score leaderboard.

6. Nomenclature, related systems, and distinctions

The name "VideoMind" is not unique to the dataset. In March 2025, "VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning" introduced a video-language agent for temporal-grounded video understanding with a role-based workflow consisting of Planner, Grounder, Verifier, and Answerer, all implemented through role-specific LoRA adapters on a Qwen2-VL backbone (Liu et al., 17 Mar 2025). That system targets answering free-form queries while identifying the temporal segment $[\text{intent}]$4 supporting the answer, and it reports experiments on 14 public benchmarks spanning Grounded VideoQA, video temporal grounding, and general VideoQA (Liu et al., 17 Mar 2025).

A further related work is "Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMs," which introduces VideoMindPalace rather than VideoMind proper (Huang et al., 8 Jan 2025). VideoMindPalace constructs a three-layered semantic graph from hand-object interaction, clustered activity zones, and environment layout mapping, and evaluates on its own Video MindPalace Benchmark as well as EgoSchema, NExT-QA, IntentQA, and the Active Memories Benchmark (Huang et al., 8 Jan 2025).

Work	Core object	Distinguishing elements
VideoMind (Yang et al., 24 Jul 2025)	Omni-modal video dataset	103K clips, three text layers, 6W tags, VideoMind-3K
VideoMind (Liu et al., 17 Mar 2025)	Chain-of-LoRA agent	Planner, Grounder, Verifier, Answerer
VideoMindPalace (Huang et al., 8 Jan 2025)	Long-video graph framework	semantic graph, activity zones, room layout

This naming overlap can create bibliographic confusion. In current usage, the dataset paper is the source of the "omni-modal video dataset with intent grounding" formulation, whereas the agent paper uses the same name for a role-based temporal-grounded reasoning architecture (Yang et al., 24 Jul 2025, Liu et al., 17 Mar 2025). The two works operate at different levels: one defines a benchmark and corpus for deep-cognitive video understanding; the other defines an inference architecture for long-form temporal reasoning. VideoMindPalace is related in ambition but distinct in representation, replacing layered captions and intents with a topologically structured semantic graph (Huang et al., 8 Jan 2025).

Across these works, a shared research direction is evident: long-form and multi-modal video understanding increasingly requires intermediate structure beyond flat captioning, whether in the form of intent-grounded text layers, role-specialized agents, or hierarchical semantic graphs.