Watch, Remember, Reason: Human-View Video Understanding with MLLMs

Abstract: Video understanding is being rapidly transformed by multimodal LLMs (MLLMs), as research moves from short clips to long, multimodal, and knowledge-intensive video scenarios. These scenarios require models to handle sparse evidence, long-range dependencies, multimodal alignment, and reliable inference under limited computational budgets. This work presents a human-view perspective on LLM-based video understanding, organized around three functional abilities: watching, remembering, and reasoning. Rather than treating video tasks as isolated benchmarks, this view provides a unified structure for analyzing how video MLLMs acquire evidence, preserve context, and produce grounded outputs. We introduce a formulation that characterizes video understanding systems by their perceptual representations, memory states, reasoning traces, and final predictions. Based on this formulation, we identify challenges in spatio-temporal perception, efficient long-video processing, memory modeling, streaming understanding, and faithful reasoning. Representative methods are organized by their roles in video MLLM systems. Watching covers fine-grained, comprehensive, audio-visual, and efficient perception. Remembering includes offline and streaming memory, while reasoning covers text-only reasoning and thinking with videos. We further examine application domains such as egocentric, sports, instructional, medical, and narrative videos, and cover training datasets and evaluation benchmarks across task types, supervision formats, modalities, and capability dimensions. Finally, we outline open problems and future directions for scalable, memory-aware, and evidence-grounded video intelligence. Related works will be continuously traced at https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.

• The paper introduces a human-inspired taxonomy ('watch, remember, reason') that reorganizes video MLLM design around perception, memory, and inference.
• It details diverse methodologies including fine-grained visual evidence acquisition, adaptive memory modeling, and grounded reasoning to enhance accuracy and reduce hallucinations.
• The findings emphasize practical tradeoffs and system innovations such as agentic pipelines and efficient memory control for scalable, long-form video analysis.

Watch, Remember, Reason: A Functional Taxonomy for MLLM-Based Video Understanding

Abstract and Survey Scope

"Watch, Remember, Reason: Human-View Video Understanding with MLLMs" (2606.07433) offers a systematic survey centered around a human-inspired functional taxonomy for multimodal LLM (MLLM) video understanding. It reframes the design and analysis of video MLLMs along three core axes—perception ("watch"), memory ("remember"), and inference ("reason")—thereby structuring a diverse landscape of contemporary techniques, data organization, and evaluation for scalable, efficient, and evidence-grounded video intelligence. The work addresses key issues in handling long, multimodal, and knowledge-intensive video scenarios, ranging from query-aware evidence selection to streaming inference, grounded reasoning, and domain-specific applications.

Figure 1: Survey pipeline and Watch--Remember--Reason taxonomy, covering fine-grained grounding, efficient processing, memory mechanisms, and agentic/grounded reasoning.

Unified Functional Decomposition: Watch, Remember, Reason

The survey argues that traditional task- or architecture-centric perspectives are insufficient for real-world, long-form video scenarios, where evidence is sparse, distributed, and multi-modal. Instead, it advocates organizing progress and challenges through three human-analogue functional lenses:

• Watching—selective multimodal perception, including spatio-temporal, auditory, and scene abstraction, while minimizing redundancy.
• Remembering—construction and adaptive retrieval of multi-scale memory representations, enabling long-range dependencies and cross-event reasoning under restricted computational budgets.
• Reasoning—grounded, interpretable inference, including both text-centric chains-of-thought (CoT) and agentic, evidence-seeking processes.

This functional decomposition serves as a scaffold for formulating system design tradeoffs, current algorithmic solutions, and open research challenges in video MLLMs.

Methods and Technical Design: Functional Analysis

1. How to Watch? Perceptual Evidence Acquisition

The review systematically categorizes perceptual approaches:

• Fine-grained watching: Models move beyond naive frame sampling, using timestamp-aware encoders, query-conditioned frame selection, and explicit spatio-temporal grounding. Recent architectures encode time as a first-class token, incorporate multi-scale encoders, and deploy RL-based verifiable reward post-training for precise temporal boundaries.
• Comprehensive watching: Advances in captioning extend from whole-video summaries to event-dense, region-level, and script-like hierarchical descriptions. Large recaptioned corpora and preference-optimized MLLMs enable high-coverage, fine-grained semantic supervision, while control methods allow user-conditioned generation.
• Audio-visual watching: Omni-modal architectures align vision and audio streams via explicit temporal tokenization (e.g., TMRoPE), Mixture-of-Experts routers, and contrastive/CTC-based feature supervision. These enable both offline analysis and low-latency, real-time human–computer interaction.
• Efficient watching: To scale for long videos, methods aggressively prune redundancy through query-aware frame selection, token compression, and adaptive model computation on both the frame- and token-level—coupled with model-side innovations like sparse attention and KV cache optimization.

Figure 2: Diverse perceptual strategies: fine-grained watching grounds events, comprehensive watching enables abstraction, audio-visual watching aligns modalities, and efficiency modules reduce input size.

2. How to Remember? Memory Modeling

The memory section delineates between offline and streaming regimes:

• Offline memory: The spectrum runs from agentic, tool-augmented external memory and adaptive retrieval to non-agentic staged pipelines (dynamic selection/compression, hierarchical memory, event graphs). Agentic paradigms exhibit flexible planning but high overhead, while non-agentic models emphasize deterministic, single-pass efficiency.
• Streaming memory: This regime targets unbounded video via rolling short-term/long-term memory banks, hierarchical event consolidation, and KV cache pruning. Hybrid architectures maintain constant or sublinear memory usage, with real-time updates and hierarchical retrieval to support both recency and long-range recall.

Figure 3: Agentic offline memory (LLM-controlled), non-agentic structured memory, and streaming memory (sliding windows, hierarchical storage) pipelines.

3. How to Reason? Grounded Inference Paradigms

The reasoning landscape is broken into:

• Text-only reasoning: Early methods rely on autoregressive CoT in language space, with subsequent reinforcement learning and preference optimization to enhance temporal/spatial robustness and reduce hallucination. GRPO-style RL and hierarchical DPO optimize both answer and reasoning trace quality.
• Thinking with videos: Inspired by OpenAI's o3, models increasingly interleave reasoning with explicit evidence gathering—both via agentic (tool-use, active zoom, hierarchical search) and non-agentic (structured, verifiable trace with timestamps/boxes) paradigms. These approaches exhibit improved faithfulness, traceability, and allow more interpretable multi-step visual reasoning.

Figure 4: Agentic reasoning (modular step decompositions, memory/tool use) versus non-agentic pipelines (single-pass, CoT-like), and agentic/non-agentic grounded reasoning where models directly interact with or reference visual evidence.

Domain-Specific Subfields and Data Infrastructure

The taxonomy is extended to specialized domains:

• Egocentric, sports, instructional, and medical video understanding demand varying configurations of perception (fine-grained, knowledge-intensive), memory persistence, and reasoning complexity.
• Training data and benchmarks: There is an explosion in scale and diversity—datasets now cover instruction tuning, chain-of-thought, multi-modal scripts, explicit event grounding, long-context streaming, and domain-specific annotations.
• Benchmarks now not only assess accuracy but test interpretability, faithfulness, evidence grounding, and long-context generalization.

Implications, Strong Claims, and Open Directions

• Strong numerical outcomes: RL-based post-training and agentic pipelines consistently outperform SFT-only models on benchmarks with verifiable evidence (temporal IoU, grounded multi-step QA). E.g., agentic and grounded reasoning systematically reduce hallucination rates and improve temporal/spatial localization—often with significant sample efficiency improvements over prior approaches.
• Bold/contradictory claims: The survey asserts that simple scaling of context or model size is insufficient for robust, long-horizon video understanding; explicit memory control and evidence-centric reasoning are mandatory for further progress.
• Practical implications: Real-time video agents (assistants, autonomous agents) and hour-long streaming scenarios become feasible only with compact, hierarchical, agentic memory and grounded reasoning.
• Theoretical implications: Separating perception, memory, and reasoning clarifies the decomposition of cognitive video understanding, exposing roles for system-level control (agentic retrievers) and hierarchical abstraction.

Future Research Directions

The work identifies open problems and future research avenues:

• Spatial reasoning: Bridging the gap between fine-grained object tracking and holistic scene modeling remains unsolved; deeper geometric and object-level temporal memory and explicit scene representations are required.
• Multi-segment/multi-video grounding: Real-world settings require cross-segment, set-based retrieval and edit-aware reasoning, where temporal and narrative boundaries are ambiguous or non-contiguous.
• Hour-scale video with structured memory: Scalable, evidence-pointer-based multi-level memory will be necessary for tasks demanding both needle-in-haystack recovery and extended narrative or procedural recall.
• Verifiable, efficient reasoning: Future systems must optimize not only answer accuracy but also evidence trace compactness and relative informativeness, pushing toward uncertainty-aware and budgeted evidence acquisition as first-class objectives.
• Streaming/embodied scenarios: Stateful, goal-conditioned streaming memory with proactive retrieval and event-triggered writing are essential for interactive and embodied cognition.

Conclusion

This survey establishes the "Watch, Remember, Reason" taxonomy as the primary analytical framework for MLLM-based video understanding. By emphasizing the co-design of perception, memory, and grounded, interpretable reasoning, it details the transition from monolithic end-to-end models to modular, human-centric video intelligence—outlining both practical recipes and theoretical challenges which will shape future developments in truly capable, evidence-grounded video MLLMs.