M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

Abstract: As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability. Despite substantial efforts in developing video datasets and benchmarks, existing works primarily focus on perception and reasoning, without systematically evaluating memory: what models retain, how faithfully information is preserved, and how robust memory remains under interference. To address this gap, we introduce M$^3$Eval, the first comprehensive evaluation framework and benchmark for probing different memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks that isolate key aspects of memory. Leveraging M$^3$Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors. We find that models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns differing substantially from those observed in human memory, ground memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic memory. Collectively, our benchmark provides a valuable resource for future research, while our findings highlight memory as a fundamental yet underexplored capability and offer insights for designing more effective memory mechanisms in multi-modal models. Our code and dataset are available at https://pku-value-lab.github.io/m3eval-homepage.

• The paper introduces M³Eval, a benchmark leveraging cognitive psychology tasks to assess long-form video memory in multi-modal models.
• It employs four paradigms—divided attention, memory interference, interleaved events, and N-Back tests—to isolate memory-specific failures.
• Results reveal that MMMs struggle with parallel processing, interference management, and temporal source tracking compared to human performance.

Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks: An Expert Analysis of M$^3$Eval

Introduction and Motivation

M$^3$Eval presents a rigorous benchmark for dissecting and systematically evaluating the memory capabilities of multi-modal models (MMMs) in long-form video understanding. While significant progress has been observed in context window scaling and perceptual or reasoning benchmarks, the capacity of such models for encoding, retaining, and retrieving information over extended temporal horizons is largely uncharacterized. Existing video or multi-modal evaluation protocols either confound memory with perception/reasoning or probe memory implicitly, eschewing principled evaluation along memory-theoretic dimensions. M$^3$Eval explicitly fills this gap via cognitively-motivated tasks probing discrete facets of memory—paralleling the analytical scaffolding found in human cognitive psychology.

Figure 1: M$^3$Eval, a principled framework and benchmark for evaluating memory capabilities of multi-modal models, grounded in psychological theory via split-screen video scenarios and multi-category memory probing.

Cognitively-Grounded Task Design

The evaluation axes in M$^3$Eval derive directly from well-established lines of cognitive psychology, yielding four distinct paradigms:

1. Divided Attention: Models encode split-screen, parallel video streams. Variants include static spatial assignments and dynamic frame swaps, isolating failures in stream disentanglement and attention competition.
2. Memory Interference: Sequenced presentation of semantically similar clips probes proactive vs. retroactive interference, mirroring classic human memory asymmetries. Multiple-choice questions quantify recall fidelity and intrusion from distractor content.
3. Interleaved Events: Fine-grained temporal interspersion of two storylines tests the ability to reconstruct and source event sequences, challenging both order recovery and source discrimination.
4. N-Back: Symbolic abstraction over sequence memory, operationalized via scene/action category matching tasks at varying distances (N) and sequence lengths (K), targeting working memory capacity and selective information retention.

This cognitive scaffolding allows isolation of failure modes and capacity constraints not confounded by general perceptual or reasoning performance.

Figure 3: Divided Attention is operationalized through a split-screen setup, optionally with periodic spatial swaps to increase attentional complexity.

Figure 2: Memory Interference manipulates presentation order to distinguish proactive from retroactive interference effects.

Figure 4: Interleaved Events; alternating segments from two sources demand event disentanglement and temporal source assignment.

Figure 5: N-Back abstraction: models decide whether the final clip matches the one N steps earlier, on either scene or action attributes.

Experimental Protocol

The benchmark consists of 2,403 high-quality, manually verified multiple-choice questions over 451 videos, curated from high-diversity datasets (e.g., HourVideo, Video-MME, LVBench, InfiniBench, CrossVid). Automatic generation using advanced LLMs is followed by stringent manual curation. Models evaluated encompass state-of-the-art proprietary systems (e.g., Gemini-3.1-Pro-Preview, GPT-5.4), open-weight MMMs (Qwen3-VL, InternVL3.5), and agentic pipelines (VideoLucy, M3-Agent). Human performance provides a critical gold standard.

Empirical Findings

Divided Attention

Across all categories (source identification, order understanding, content retention), MMMs exhibit substantial performance deficits compared to humans, often barely exceeding random baseline. Notably, spatial frame swapping sharply degrades source identification but not sequence or content memory, implicating an attention diffusion and stream confusion failure in parallel input processing.

Figure 6: Attention visualizations demonstrate split-screen-induced interference; model focus degrades under parallel stream intrusion.

Memory Interference

A robust retroactive-vs-proactive interference asymmetry is observed in human subjects—retroactive interference (later content intruding on earlier recall) far exceeds proactive effects, matching the literature. MMMs, in contrast, display near-symmetric interference effects, with no pronounced retroactive dominance. Strikingly, repetition of either target or interfering content ameliorates interference weaknesses, suggesting representational blending rather than canonical overwriting.

Interleaved Events

Task difficulty increases substantially in temporally-interleaved conditions. All MMMs score dramatically below human baselines on source identification, order recovery, and discrimination of false memories (with frequent below-random performance in the latter). Further, memory source assignment is more robust along the spatial domain (as in split-screen) than the temporal domain (as in interleaved events), for both models and humans, underlying the increased computational burden of temporal source tracking.

N-Back and Symbolic Memory

On abstracted N-Back tasks, MMMs demonstrate a pronounced limitation. Human working memory decays gracefully with increasing N—reflecting plausible bounded capacities—but retains high accuracy by filtering out task-irrelevant information when the sequence length increases with fixed N. MMMs, conversely, do not exhibit temporal decay with N (benefiting from unrestricted Transformer attention), but their accuracy degrades severely as sequence length grows, exposing a lack of efficient irrelevant information suppression and selective forget mechanisms.

Figure 7: Overall accuracy on the N-Back symbolic matching task; humans outperform all MMMs, which show no working-memory capacity profile.

Implications and Research Directions

M$^3$Eval delivers several robust, empirically supported findings:

• Parallel Stream Processing Deficit: Current MMMs, even those with exceptional single-stream competence, lack a robust mechanism for attentional disentanglement under concurrent visual load, unlike humans.
• Interference Profile Divergence: Humans and MMMs implement fundamentally distinct memory update strategies. Transformer's universal attention erases human-like retroactive vulnerability and instead induces uniform blending/interference.
• Temporal Organization Bottleneck: Representing and reconstructing temporally interleaved event streams is a key unsolved problem in MMMs; spatial cues provide anchoring, but temporal cues require new architectural or training innovations.
• Symbolic Memory and Selective Forgetting: The absence of natural memory decay or selective abstraction translates into sharply reduced filtering and abstraction performance when the memory load increases, a critical block for compositional or generalizable reasoning.

Conclusion

M$^3$Eval systematically characterizes the boundaries of multi-modal video memory via cognitivist paradigms, revealing stark gaps between current MMMs and human-level memory. It establishes diagnostic axes for future model development, highlighting the need for architectural or algorithmic rethinking in attentional routing, dynamic interference handling, temporal ordering, and symbolic abstraction. Integration of cognitive principles—such as attention gating, structured forgetting mechanisms, and explicit temporal encoding—may be mandatory for closing these fundamental gaps, enabling the design of MMMs that match human performance not only in perception and reasoning, but in the robustness and fidelity of memory (2606.05008).