--- title: Streamo-Instruct-465K Video Dataset url: https://www.emergentmind.com/topics/streamo-instruct-465k type: topic --- # Streamo-Instruct-465K Video Dataset Streamo-Instruct-465K is a large-scale, multi-task instruction tuning dataset specifically constructed for continuous, streaming video understanding. Comprising 465,000 instruction–video pairs drawn from 135,875 public videos, it provides unified multi-task supervision tailored for real-time, interactive video models. Streamo-Instruct-465K enables the training of video language models (VLMs) capable of temporally grounded narration, action and event captioning, temporal event grounding, and time-sensitive question answering, thus supporting a broad set of streaming video applications and dynamic user interactions [2512.21334]. ## 1. Dataset Construction and Structure Streamo-Instruct-465K sources videos from eight open-access corpora—Koala, LLaVA-Video, ActivityNet, QVHighlight, YouCook2, HACS, EgoTimeQA, DiDeMo, and COIN—covering domains such as cooking, procedural instruction, egocentric perspective, sports, and general human activity. The dataset encompasses a combined ~1,150 hours of video, with individual clips ranging from a few seconds to several minutes (median length ~30 s). A unified annotation pipeline leverages large language models for automatic instruction generation, segmenting each video into 1-second windows (1 fps). Diverse tasks are formulated: - **Real-time Narration:** Per-second commentary on frame-to-frame changes using multi-window context, post-processed for redundancy and coherence by secondary LLMs. - **Event Caption:** Segment-level captioning via temporally grounded sampling, retaining only those samples where segment spans overlap consistently. - **Action Caption:** Filtering the event caption pipeline for action-centric prompts, generating step-level procedural annotations. - **Event Grounding:** Conversion of event captions into temporal grounding prompts, labeled with precise onset and offset timestamps. - **Time-Sensitive QA (TSQA):** Question-answer pairs where the answer changes at annotated temporal landmarks, supporting dynamic, evolving Q&A grounded in temporal video events. Each task is represented using a consistent prompt template and temporal marker format (``), ensuring alignment between instructions and annotated video segments. ## 2. Task Taxonomy and Sample Distribution Streamo-Instruct-465K’s instruction–response pairs are distributed approximately as follows: | Task Type | Number of Samples | Percentage of Total | |--------------------------|-------------------|--------------------| | Real-time Narration | 140,000 | ~30% | | Event Caption | 110,000 | ~24% | | Action Caption | 85,000 | ~18% | | Event Grounding | 70,000 | ~15% | | Time-Sensitive QA | 60,000 | ~13% | | Offline QA (LLaVA-Video) | 65,000 | — | The class distribution for streaming decision-state labels (::) is heavily imbalanced at roughly 12:3:2. Instructions cover a spectrum from brief observation to complex temporal reasoning, as illustrated by the example: > USER: “Notify me when the light turns green.” > ASSISTANT (frames 4s–5s): ` The light just turned green.` Multiple streaming-style tasks are annotated per video, enabling unified multi-task training with diverse supervision signals [2512.21334]. ## 3. Annotation Methodology and Quality Control The annotation pipeline is fully automated and LLM-driven, without manual annotators or inter-annotator disagreement management. Each annotation task employs task-specific prompting and robust post-processing: - Templates and temporal markers ensure instruction consistency. - GLM-4.5 is used for deduplication and smoothing of real-time narration. - Consistent overlapping of event segments is required for event and action captions. - Automated cross-model validation and redundancy filtering serve as primary quality filters. No reported human-annotator disagreements; quality is instead enforced through heuristics, span consistency checks, and multi-model comparison. ## 4. Integration into Model Training and Mathematical Formulations All samples are formatted as interleaved, multi-turn dialogue with 1-second response granularity. Temporal tags are prepended to video-feature tokens. The model training pipeline consists of: - Vision encoder kept frozen; connector and LLM components are fine-tuned end-to-end. - A single epoch over the entire dataset (batch size 512, learning rate 1e-5). - Unified multi-task optimization through exhaustive shuffling; no explicit task curriculum. The decision-state prediction leverages a focal cross-entropy loss to rebalance sparse labels: $$ w_\text{focal}(x_i) = (1 - p_{c_i})^\gamma, \; \gamma=2 $$ $$ \alpha_k = \frac{1}{|S|}\left(\sum_{j\in S} \frac{n_j}{n_k}\right) $$ $$ L_i = \alpha_{t_i}\cdot w_\text{focal}(i)\cdot \mathrm{CE}(z_i, t_i), \;\text{if } t_i \in S $$ $$ L_\text{total} = \frac{1}{|M|} \sum_{i\in M} L_i $$ Temporal grounding is quantitatively evaluated using mean Intersection over Union (mIoU) between predicted and ground-truth event intervals: $$ \text{IoU}_i = \frac{\operatorname{length}(\operatorname{intersection}(t_i^{\text{pred}}, t_i^{\text{gt}}))}{\operatorname{length}(\operatorname{union}(...))} $$ $$ \text{mIoU} = \frac{1}{N} \sum_{i=1}^{N} \text{IoU}_i $$ Ablation studies confirm that focal loss is critical for recovering rare response states in real-time predictions [2512.21334]. ## 5. Benchmark Impact and Use Cases Substituting Streamo-Instruct-465K for prior benchmarks (e.g., ET-Instruct-164K) on OVO-Bench yields +11.8% overall and +7.1% forward-responding task improvements. Models trained on Streamo-Instruct-465K achieve 2–4% absolute accuracy gains on offline video understanding suites such as MVBench, TempCompass, VideoMME, and LongVideoBench. These results indicate substantive improvements in temporal reasoning and multi-task generalization across streaming and traditional video benchmarks. Typical application scenarios include: - Live narration and commentary of streaming video content - Real-time detection and reporting of temporally grounded events - On-the-fly, evolving question answering as video context changes - Interactive assistants that can anchor instructions and queries in dynamic visual streams A plausible implication is that Streamo-Instruct-465K enables unified, interactive multimodal assistants that more closely approximate continuous, intelligent video understanding [2512.21334]. ## 6. Limitations and Future Directions Limitations of Streamo-Instruct-465K include strong dependence on LLM-based annotation pipelines, which may variably capture domain diversity across video types. Average streaming context lengths per sample remain modest (~30 s), suggesting future work on scaling to longer or lower-latency streams. The dataset's heavy task and label imbalance poses challenges for less frequent real-time response states, which are partially mitigated by loss reweighting. Future directions indicated in experimental reports involve: - Expanding coverage to more complex or longer streaming scenarios - Optimizing for lower-latency, higher-throughput deployment - Exploring human-in-the-loop and hybrid annotation pipelines for broader generalization - Integrating planning or adaptive prompt mechanisms for improved real-time performance Streamo-Instruct-465K represents the first large-scale, unified temporal instruction-following resource for streaming video understanding, providing the baseline for end-to-end LLM training in interactive, temporally annotated video domains [2512.21334]. Source: https://www.emergentmind.com/topics/streamo-instruct-465k