SLVMBench: Skill Learning from Video Memory

Abstract: We introduce Skill Learning from Video Memory (SLVMBench), the first benchmark that jointly evaluates whether video LLMs (video-LLMs) can learn skills from long video memory and apply them to real-time tasks. SLVMBench presents models with 2-3 hour video streams that contain a tutorial video embedded in a stream of arbitrary irrelevant videos, resembling real-world human learning practices. Video-LLMs are asked to apply the acquired skill to answer real-time questions about an ongoing video. Unlike long-video understanding benchmarks that emphasize passive comprehension and skill-learning benchmarks that rely on short, immediate demonstrations, SLVMBench tests the full pipeline of memorizing and extracting procedural knowledge, as well as transferring it to real-time tasks. Moreover, rigorous human annotations feature sub-second-level temporal calibration, manually engineered questions eliminating common-sense guessing, and collated tutorials to ensure coverage of the required skills. Evaluations on state-of-the-art proprietary and open-source video LLMs show that video-LLMs struggle substantially with learning and applying skill knowledge from videos. Moreover, performance degrades markedly when the skill knowledge is placed within a long video memory. These results reveal a key limitation of existing video LLMs and position SLVMBench as the first benchmark for studying real-time skill acquisition and application from long-context video memory.

• The paper introduces SLVMBench, a benchmark for evaluating long-term procedural skill learning from video memory by simulating realistic tutorial and distractor scenarios.
• It employs a rigorous multi-stage pipeline combining LLM-driven query generation with human validation to create 2,261 QA pairs across diverse domains and task types.
• Empirical results reveal significant memory decay in non-streaming models and improved retention in streaming architectures, underlining critical challenges for future multimodal AI systems.

SLVMBench: Evaluating Skill Learning from Long-Horizon Video Memory in Multimodal LLMs

Motivation and Benchmark Design

SLVMBench is introduced as a rigorous evaluation framework to address a critical, underexplored capability in contemporary video-LLMs: the acquisition, long-term retention, and transfer of procedural skill knowledge from extended video streams. Unlike prior benchmarks that predominantly target either real-time streaming perception or passive comprehension of long-form content, SLVMBench explicitly simulates realistic human learning scenarios. This involves models needing to observe a tutorial video, retain its procedural information across potentially hours-long distractor content, and adapt the learned skill to answer predictive questions about an ongoing task video at precise moments.

Figure 1: The SLVMBench task structure requires procedural learning, maintenance amidst distractor streams, and predictive reasoning at controlled timestamps.

The formalized evaluation comprises triplets: (i) a curated tutorial video, (ii) up to 2 hours of distractor videos, and (iii) a target video paused at a critical cutoff. Models are tasked with answering multiple-choice questions that require cross-video knowledge transfer—not trivially solvable from visual context or common sense alone.

Task Taxonomy and Dataset Organization

SLVMBench's taxonomy spans 11 real-world domains and 11 distinct task types, covering procedural mastery (e.g., next-step prediction, step ordering), constraint/tool logic (e.g., parameter recall, tool configuration, safety checks), and diagnostic/adaptation reasoning (e.g., mistake detection, conditional branching). The challenge is multi-modal and multi-dimensional: questions require simultaneous integration of verbal cues, visual evidence, and precise temporal state tracking.

Figure 2: SLVMBench’s fully hierarchical taxonomy enables coverage of procedural, constraint, and diagnostic reasoning tasks.

The dataset consists of 2,261 manually validated QA pairs across 1,220 video instances. Each instance is annotated with sub-second temporal calibration for cutoffs, engineered distractor options to preclude guessing, and thorough quality control to ensure logical rigor and tutorial coverage. Diversity and distribution are balanced for both task types and video durations, with empirical coverage from 10 minutes to 120 minutes, directly targeting the limits of long-horizon memory in current architectures.

Data Construction Pipeline

SLVMBench's five-stage pipeline leverages LLMs (GPT-4o and Gemini-2.5-Pro) for procedural query generation and initial QA synthesis, followed by intensive human validation to guarantee high-fidelity mapping between tutorial-target pairs. Key stages include post-2025 video filtering to minimize training data contamination, logic audits to ensure non-trivial knowledge transfer, and distractor engineering to ensure plausible but incorrect options.

Figure 3: Multi-stage SLVMBench pipeline combines LLM-driven content acquisition, automated QA synthesis, and expert human annotation for logical and temporal precision.

This process ensures that episodic memory and skill transfer are directly evaluated, not confounded by prior memorization or common-sense cues.

Evaluation Paradigms and Model Analysis

SLVMBench introduces three distinct paradigms for comparative evaluation: (1) baseline (target video only), (2) immediate tutorial transfer (tutorial followed directly by target), (3) long-context episodic transfer (tutorial separated from target by distractors up to 2 hours). Accuracy is the primary metric, with deterministic inference and Wilson confidence intervals reported for statistical reliability.

Strong experimental results evidence the validity and challenge of SLVMBench:

• Immediate tutorial knowledge yields substantial performance gains across all models (e.g., Gemini 3.1 Pro +29.47%, video-SALMONN2+ +29.72%) compared to baseline, confirming the necessity of procedural learning.
• Memory decay is pronounced in the long-context paradigm: For most non-streaming models, performance degrades sharply, with $\Delta_L$ (long-form episodic gain) shrinking to near baseline (e.g., GPT-5.2 +2.21%, VideoLLaMA3 +0.64%). Streaming and memory-augmented architectures (video-SALMONN-S, video-SALMONN2+) achieve higher retention ($\Delta_L$ up to +15.79%), but still fall short of reliable retrieval after extended distractor sequences.

Fine-grained analysis reveals that logic-intensive tasks (diagnostic branching, mistake detection) depend heavily on continuous temporal tracking, with streaming architectures achieving up to 73.16% in conditional branching, while non-streaming models collapse to $\approx$16.09%.

Memory Decay and Architectural Implications

SLVMBench empirically demonstrates that all architectures suffer significant accuracy decay as the temporal gap increases. The memory gap between tutorial and target yields a clear cognitive bottleneck, with open-source models exhibiting abrupt collapse after 30 minutes and proprietary models (Gemini 3.1 Pro) plateauing before a final steep drop.

Figure 1: Memory decay curves highlight rapid deterioration in episodic retrieval as the distractor window lengthens.

These results underscore the urgent need for architectural innovations in selective retrieval and information noise filtering for practical, agentic AI systems facing real-world long-horizon tasks.

Practical and Theoretical Implications

SLVMBench establishes a new benchmark paradigm: realistic, high-fidelity episodic memory and skill transfer in multimodal LLMs. Empirical findings substantiate several key claims:

• Procedural learning and adaptation from videos are not reliably solvable by current MLLMs, especially in long-context settings.
• Audio integration is critical; the absence of verbal cues leads to significant performance deficits, especially in ambiguous procedural steps.
• Higher frame densities and continuous streaming architectures demonstrably bolster memory retention and reasoning fidelity, but naive input window extension or keyframe sampling is insufficient.
• Diagnostic reasoning, error detection, and adaptive branching remain unsolved in non-streaming models, limiting applicability in embodied or interactive AI agent scenarios.

SLVMBench provides a foundation for future advances in memory architectures, streaming retrieval protocols, and multi-modal fusion, with direct relevance to real-time skill acquisition and deployment in agentic and robotic settings. The open annotation pipeline and rigorous quality control facilitate reproducibility and transparency for continued benchmarking and research.

Conclusion

SLVMBench rigorously characterizes the episodic skill learning and practical memory limits in state-of-the-art video-LLMs. By exposing severe memory decay and transfer deficits, it frames the next frontier in multimodal AI: architectures capable of high-fidelity long-term procedural reasoning under distractor-rich, real-world streaming conditions. The findings motivate focused research into scalable memory augmentation, selective retrieval, and robust multimodal integration for agentic skill learning and deployment.