MemEye Framework: Visual Memory Diagnostics

• MemEye is a visual-centric framework that systematically diagnoses multimodal agents’ memory by assessing fine-grained visual evidence and temporal state reasoning.
• It organizes evaluations along two axes—visual evidence granularity (from scene-level to pixel-level) and reasoning depth (from atomic recall to evolutionary synthesis).
• The benchmark uses 371 questions across eight real-world tasks to expose gaps in current architectures relating to detailed storage and up-to-date evidence integration.

MemEye is a visual-centric evaluation framework for systematically diagnosing and benchmarking the memory capabilities of multimodal agents. The framework is designed to probe whether agent systems preserve, retrieve, and reason over fine-grained visual evidence—beyond coarse scene-level representations—and whether they can synthesize up-to-date beliefs from temporally evolving multimodal experiences. MemEye introduces orthogonal dimensions of evaluation spanning visual evidence granularity and reasoning depth. It is accompanied by a curated benchmark targeting a spectrum of real-world life-scenario tasks that expose distinct technical challenges in long-term multimodal memory (Guo et al., 14 May 2026).

1. Motivation and Limitations of Prior Work

Long-term memory in agent systems has rapidly evolved toward multimodality, incorporating images, text, and interaction histories. However, established benchmarks such as VisDial and CLEVR-Dialog focus on short-context dialogue or allow image content to be replaced by captions, failing to stress-test visual preservation. Other long-memory testbeds, including LoCoMo and LongMemEval, are predominantly text-oriented and not visual‐centric. In these settings, agent failures remain underdiagnosed, as successful question answering may occur via shortcutting (leveraging captions rather than genuine visual state) or by reasoning over outdated/stale state.

Critical open problems highlighted by MemEye include:

• Preservation of Fine-Grained Visual Evidence: Effective memory architectures must retain region layouts, object identities, and pixel-level cues (color, small text, texture, OCR), not just global scene summaries.
• Reasoning Over Temporal State Changes: Agents must resolve conflicts, link clues across sessions/modalities, and synthesize non-monotonic updates to accurately infer current world state.

2. Organizational Axes and Formal Framework

MemEye evaluates agent memory via a structured grid defined by two axes:

• Visual Evidence Granularity (denoted $X$): Ranges from $X_1$ (scene-level gist recoverable from captions), through $X_2$ (region-level groups), $X_3$ (instance-level object/person identity), to $X_4$ (pixel-level detail: color, OCR, texture).
• Memory-Reasoning Depth (denoted $Y$): Spans $Y_1$ (atomic retrieval of a single evidence unit), $Y_2$ (linking multiple nonconflicting clues across modal/session divides), to $Y_3$ (evolutionary synthesis: reconciling conflicting/updated evidence to infer the current state).

Each benchmark question $q$ is annotated as $X_1$0, with $X_1$1 (evidence granularity) and $X_1$2 (reasoning complexity). This bidimensional taxonomy enables granular traceback of failure modes attributable to either detail loss or synthesis breakdown.

3. Benchmark Construction and Validation Methodology

The MemEye benchmark comprises 371 questions curated across eight "life-scenario" tasks:

Domain	Task Name	$X_1$3-Range	$X_1$4-Range
Leisure	Card Playlog	$X_1$5	$X_1$6
	Cartoon Entertainment	$X_1$7	$X_1$8
Domestic	Home Renovation	$X_1$9	$X_2$0
	Outdoor Navigation	$X_2$1	$X_2$2
Professional	Brand Memory	$X_2$3	$X_2$4
	CrossScene Memory	$X_2$5	$X_2$6
Personal	Health Care	$X_2$7	$X_2$8
	Social Chat	$X_2$9	$X_3$0

Each question is instantiated in both multiple-choice (with cyclic option rotations to control for positional bias) and open-ended forms.

Ablation-Driven Validation Gates:

• Answerability Gate: Excludes items unsolvable by foundation models (rotation-averaged EM ≤0.25) in gold-clue rounds, identifying base model rather than memory deficits.
• Shortcut-Resistance Gates: Probes model performance under (a) options alone, (b) context without images, (c) minimal captions. Items with EM=1.0 in these views are flagged as insufficiently visual.
• Visual Necessity Gate: Uses dense GPT-5.2 captions in place of images; if answer accuracy remains perfect, item is revised/removed as text-bypassable.
• Reasoning-Structure Audit: Ensures that the intended reasoning structure is present (single clue for $X_3$1; at least two for $X_3$2; two or more temporally ordered clues and conflict resolution for $X_3$3).

4. Evaluation Protocol and Diagnostic Metrics

MemEye evaluates 13 memory architectures (seven text-only, six multimodal) across four VLM backbones: Qwen3-VL-8B-Instruct, GPT-4.1-nano, GPT-5.4-mini, and Gemini-2.5-flash-lite.

• Multiple-Choice: Accuracy computed as Exact Match (EM), rotation-averaged.
• Open-Ended: Output evaluated by GPT-5.2 as LLM-as-a-Judge ($X_3$4), with auxiliary BLEU-1 reporting. Human judge agreement is high (Cohen’s $X_3$5).
• Aggregate Scores: EM and LLM-Judge reported per $X_3$6 cell, with macro-averages for holistic view.
• Retrieval Diagnostics (Retrieval-based Methods):
  • Any-Clue Recall@K: Fraction of items for which ≥1 gold clue is retrieved.
  • Coverage@K: Average fraction of gold clues recalled.
  • Full-Clue Recall@K: Fraction of items retrieving the full gold clue set.
  • Latest-Clue Recall@K (for $X_3$7): Sensitivity to recency, fraction retrieving current valid state.
  • Stale-Dominance: Proportion where a stale clue is ranked above the latest.

Caption-Proof Validation: Computes score differential $X_3$8 between native images and dense captions.

Recency Re-Ranking Probe: Adjusts retrieval ranking via $X_3$9, quantitatively probing stale-dominance and its mitigation.

5. Experimental Results and Observed Failure Modes

Performance (GPT-5.4-mini):

Architecture	LLM-Judge (Best)	EM (Best)
SRAG(T)	≈0.391	≈0.548
SRAG(V)	≈0.494	≈0.618
FC(V)	≈0.439	≈0.604
Mean	≈0.358	≈0.500

Key Findings:

• Fine-Grained Detail Exposes Robustness Gaps: Caption-Proof gains ($X_4$0) are marginal at scene level ($X_4$1, +0.03), but expand at instance/pixel levels ($X_4$2, +0.28), revealing that many architectures rely on caption-like abstractions and miss visual granularity.
• Textual vs. Image-Based Memory Trade-Offs: Text memory excels at tracking evolving state chains ($X_4$3) but loses object/pixel precision ($X_4$4). Image memory maintains detail but is vulnerable to staleness and partial evidence when reasoning over updates.
• Retrieval at $X_4$5 (SRAG(V), K=10): Any-Clue Recall ≈0.75, Coverage ≈0.55, Full-Clue ≈0.37, Latest-Clue ≈0.53, Stale-Dominance ≈0.77. High Stale-Dominance signals frequent retrieval of outdated visual states.
• Recency Re-Ranking: With $X_4$6, Stale-Dominance is reduced by ~8%, but latest clues are still frequently missed; answer improvement is inconsistent.
• Evolving Visual-State Probe: Oracle all-clue Judge ≈0.73, latest-only ≈0.71, stale-only ≈0.59; memory systems achieve only 0.18–0.39, indicating persistent challenges in integrating up-to-date evidence.
• Cross-Topic Scaling: FC(V) experiences >10 EM point drop as topic-misaligned history accrues; SRAG(V) and MMA are more robust, with only 2–3 point variation.

Case studies highlight frequent failures of caption-based retrieval at higher granularity and reasoning levels ($X_4$7), and retrieval traps in evolving narratives ($X_4$8), such as fossilized outdated facts or narrative misalignment.

6. Design Principles and Diagnostic Utility

MemEye demonstrates that robust long-term multimodal memory must integrate:

1. Raw Image Evidence Storage: Essential for $X_4$9 (instance/pixel) tasks that cannot be resolved from captions alone.
2. Structured State Update Logics: Textual or symbolic chains facilitate accurate evolutionary synthesis ($Y$0), mitigating staleness and enabling valid-current-state inference.
3. Recency- and Authority-Aware Retrieval: Temporal attributes must be explicitly modeled and integrated into evidence ranking; otherwise, noisy histories dominate and corrupt final answers.

MemEye offers a diagnostic taxonomy for the design and evaluation of new memory architectures, enabling precise attribution of failure and success to evidence granularity and reasoning depth.

7. Implications and Prospective Directions

MemEye’s multidimensional diagnostic lens reveals persistent trade-offs and design gaps in leading multimodal agent architectures. It underscores the inadequacy of caption-replacement or naive retrieval for tasks requiring detailed, time-sensitive visual synthesis.

Future system designs are informed by the following priorities:

• Image Buffers: Dedicated raw storage of visual events for downstream retrieval.
• State Change Logs: Textual or structured representations supporting update- and conflict-oriented queries.
• Temporal Authority Modules: Explicit modeling of recency and versioning for evidence selection.
• Fine-Grained, Temporally-Aware Retrieval: Ranking functions that conjoin semantic similarity with temporal prioritization.

MemEye’s benchmark and methodology offer a foundation for convergence toward memory systems that jointly optimize for visual detail preservation and coherent reasoning over evolving multimodal agent experiences (Guo et al., 14 May 2026).