MemEye: Visual-Centric Multimodal Memory Evaluation

• MemEye is a visual-centric multimodal memory framework that tests agents’ ability to preserve and use fine-grained visual evidence over time.
• It employs a two-dimensional taxonomy of visual granularity and reasoning depth to categorize memory challenges across 8 realistic life-scenario tasks.
• The benchmark combines multiple-choice and open-ended formats with rigorous validation gates to expose shortcomings in evidence retention, temporal tracking, and detail extraction.

MemEye is a visual-centric evaluation framework for long-term multimodal agent memory that tests whether agents preserve the visual evidence required for later reasoning rather than relying on captions or textual traces. It combines a two-dimensional taxonomy of memory difficulty with a curated benchmark of 371 questions across 221 sessions, 848 dialog rounds, and 438 images in 8 life-scenario tasks, and it evaluates systems in both multiple-choice and mirrored open-ended formats (Guo et al., 14 May 2026).

1. Conceptual scope and motivation

MemEye was introduced to address a specific failure mode in multimodal memory evaluation: many visually grounded questions in prior work can be answered from captions or dialogue alone, so the benchmark does not actually verify whether a system retained decisive visual evidence. Harder cases involving reasoning over changing visual states are also largely absent. MemEye is therefore designed to make the visual channel the bottleneck and to test not only recall of visual content but also reasoning over how that content evolves over time (Guo et al., 14 May 2026).

A central claim of the framework is that it is not “just another VQA dataset.” The benchmark is organized around long-term memory across sessions and rounds, and its questions are explicitly filtered to be “caption-proof”: they are intended to resist solution by reasonable captions, local textual clues, or answer-option priors. This focus distinguishes MemEye from short-context visual dialogue benchmarks and from long-context memory benchmarks whose conflicts and updates are primarily textual rather than visual (Guo et al., 14 May 2026).

The broader significance of this design is methodological. MemEye separates two issues that are often conflated in agent evaluation: whether a system preserved sufficiently fine-grained evidence, and whether it can correctly use preserved evidence to answer temporally structured questions. This suggests a more diagnostic view of multimodal memory, in which retrieval quality, temporal authority, and visual granularity are independently stress-tested.

2. Two-dimensional taxonomy

MemEye’s organizing principle is a coordinate system over visual evidence granularity and memory-reasoning depth. Every question is annotated with a coordinate $(X_i, Y_j)$.

Level	Category	Definition
$X_1$	Scene-level	“Scene type, activity, and global semantic gist.”
$X_2$	Region-level	“Semantically coherent subregions” and local context.
$X_3$	Instance-level	“Localize and distinguish specific object or person instances… preserving entity identity when multiple similar candidates exist.”
$X_4$	Pixel-level	“Fine-grained details such as color, texture, or small text… often absent from text, reflecting pixel-level necessity.”
$Y_1$	Atomic Retrieval	“Retrieve a single fact… no cross-session reasoning.”
$Y_2$	Relational Association	“Associate distributed evidence across sessions and modalities” in a monotonic way.
$Y_3$	Evolutionary Synthesis	“Non-monotonic synthesis over evolving memory… updates, conflicts, and overrides, maintaining a coherent world state under revision.”

Questions are labeled by the finest evidence level actually required under a “highest-bottleneck rule.” The two axes are treated as orthogonal: a question may require pixel-level evidence with only atomic retrieval, or scene-level evidence with evolutionary synthesis. This separation is fundamental to the benchmark’s interpretability because failure at high $X$ implicates evidence preservation, whereas failure at high $Y$ can persist even under oracle retrieval and therefore implicates downstream reasoning (Guo et al., 14 May 2026).

The distribution of questions is intentionally skewed toward fine-grained visual dependence. Of the 371 questions, 144 fall in $X_1$0 and 136 in $X_1$1, while 198 fall in $X_1$2 and 60 in $X_1$3. This suggests that MemEye was engineered less as a balanced survey corpus than as a stress test for the high-granularity, temporally structured regime that caption-based memory often fails to support.

3. Benchmark construction and validation gates

MemEye constructs each item in two forms: one multiple-choice question with 4 options and answer rotations, and one mirrored open-ended question. The benchmark then applies ablation-driven validation gates intended to eliminate shortcut-solvable or under-specified items (Guo et al., 14 May 2026).

The first gate is Answerability (Oracle Answerability Gate). Systems are given the gold clue rounds and the original images. If strong VLMs still perform at or below chance on the multiple-choice version, the item is treated as visually ambiguous, underspecified, or too hard for the backbone, and it is removed or revised.

The second gate is Shortcut resistance. An Option-only gate tests whether question and options alone leak the answer; a Text-only clue gate tests whether clue-round text without images makes the answer recoverable. Items that are consistently solvable under these conditions are removed or revised.

The third gate is Visual necessity and caption-proofness. Images are replaced with minimal captions such as “A room photo,” “A game board screenshot,” or “A cartoon panel.” If the item remains reliably solvable, it is rejected because detailed visual content was not actually necessary.

The fourth gate is Reasoning structure alignment (Taxonomy Audit). This verifies that a purported $X_1$4 item truly collapses to a single evidence unit, that a $X_1$5 item genuinely requires distributed non-redundant clues, and that a $X_1$6 item genuinely depends on updates or overrides rather than simple aggregation.

These gates are not auxiliary bookkeeping. They are the mechanism by which MemEye operationalizes “visual necessity” and “reasoning structure” as measurable design constraints rather than post hoc interpretations. A plausible implication is that MemEye treats dataset curation itself as part of the evaluation protocol, not merely as data collection.

4. Task domains and benchmark content

The benchmark spans 8 life-scenario tasks grouped into 4 domains.

Domain	Task	Core challenge
Leisure	Card Playlog	Card faces, counts, hand-state changes
Leisure	Cartoon Entertainment	Narrative progression, character identity, fine attributes
Domestic	Home Renovation	Design revisions, sample matching, final-vs-tested states
Domestic	Outdoor Navigation	Landmarks, route segments, prior-later associations
Professional	Brand Memory	Logo variants, slogans, layout recall
Professional	CrossScene Memory	Object migration, tag replacement, latest valid state
Personal	Health Care	Dashboard updates, doctor-message overrides, small UI text
Personal	Social Chat	Person details across chat sessions

Several tasks are designed around explicit visual state evolution. In CrossScene Memory, tools and containers move across benches, shelves, and desks, and tags are replaced. In Home Renovation, paint tests and cabinet samples are compared against later final-room images. In Health Care, portal fields and doctor guidance change over time, including patterns described as $X_1$7. In Card Playlog, the benchmark uses UNO-like board screenshots in which exact card colors, values, and counts matter (Guo et al., 14 May 2026).

The benchmark’s content statistics reinforce this emphasis. It contains 49 $X_1$8 questions, 42 $X_1$9 questions, 144 $X_2$0 questions, and 136 $X_2$1 questions. Along the reasoning axis, it contains 113 $X_2$2 questions, 198 $X_2$3 questions, and 60 $X_2$4 questions. The high count of $X_2$5 and $X_2$6 items means that the benchmark disproportionately targets cases where dense captions and textual summaries are known to omit decisive evidence.

5. Evaluation protocol, metrics, and empirical results

MemEye evaluates 13 memory methods across 4 VLM backbones: Qwen3-VL-8B-Instruct, GPT-4.1-nano, GPT-5.4-mini, and Gemini-2.5-flash-lite. The memory methods include caption-based full context and retrieval systems, text-centric structured agents such as Reflexion, Generative Agents, MemoryOS, and A-Mem, and multimodal methods such as FC(V), SRAG(V), MIRIX, MMA, M2A, and SimpleMem(V) (Guo et al., 14 May 2026).

The benchmark reports EM (Exact Match) for multiple-choice evaluation and LLM-as-a-Judge for open-ended evaluation. The judge is GPT-5.2 with a rubric scoring in $X_2$7, and human agreement on a 72-item sample is reported as Cohen’s $X_2$8. BLEU-1 is included as an auxiliary lexical metric.

Under GPT-5.4-mini, the best overall open-ended result is SRAG(V) with average LLM-Judge 0.4937 and average EM 0.6177. Full-context baselines are weaker: FC(T) obtains EM 0.5670 and LLM-Judge 0.4280, while FC(V) obtains EM 0.4651 and LLM-Judge 0.4391. The benchmark therefore shows that even the best reported systems remain far from ceiling performance.

MemEye also includes direct image-versus-caption diagnostics. Visual gain is defined as

$X_2$9

At low $X_3$0, $X_3$1 is small, indicating that captions are often sufficient for scene-level and some region-level questions. At high $X_3$2, the gain becomes clearly positive. In oracle evidence diagnostics, the reported visual-text open-ended gaps are +0.122 for $X_3$3, +0.262 for $X_3$4, +0.264 for $X_3$5, and +0.298 for $X_3$6. Even with task-aware dense captions, the gap remains positive for $X_3$7, supporting the benchmark’s claim of pixel-level irreducibility (Guo et al., 14 May 2026).

Reasoning depth remains difficult even when retrieval is perfect. Under oracle evidence with GPT-5.4-mini, LLM-Judge scores by reasoning level are 0.673 for $X_3$8, 0.601 for $X_3$9, and 0.558 for $X_4$0. This demonstrates that the $X_4$1 axis is not reducible to retrieval difficulty alone. To probe stale-evidence failures, the paper studies a recency-aware reranking rule,

$X_4$2

which reduces stale-dominance and rank-inversion but does not fully solve answer quality. The result suggests that temporal scoring is necessary but insufficient unless the system also performs explicit state resolution.

6. Findings, misconceptions, and broader significance

MemEye’s principal empirical finding is that current architectures still struggle to preserve fine-grained visual details and reason about state changes over time. The paper condenses this into three bottlenecks: evidence routing, temporal tracking, and detail extraction (Guo et al., 14 May 2026).

The first common misconception addressed by MemEye is that multimodal memory can be adequately evaluated with captions. MemEye shows that caption-based storage is often sufficient at $X_4$3 and parts of $X_4$4, but it breaks down for instance-level discrimination and pixel-level details such as small labels, exact card faces, colors, and local object configurations. The second misconception is that longer context alone solves memory. Cross-topic scaling ablations show that naively concatenating more history degrades performance, especially for FC(V), while retrieval-based and structured systems are more robust but still not state-aware enough. The third misconception is that retrieval quality by itself is the dominant problem. Oracle evidence experiments show persistent degradation at $X_4$5, indicating that world-state revision and temporal authority remain open reasoning problems.

The benchmark also clarifies a trade-off between text-centric and image-centric memory systems. Text-centric structured methods are comparatively effective at organizing and summarizing evolving state but often lose fine visual details when images are compressed into text. Image-preserving methods retain decisive visual evidence but are frequently vulnerable to semantic retrieval errors, stale evidence, and missing “latest valid state” reasoning. A plausible implication is that high-performing multimodal memory will require both raw visual preservation and structured state records rather than choosing between them.

MemEye is best situated as an evaluation framework rather than an acquisition system. In adjacent work, MemX addresses attention-aware smart eyewear for personalized moment auto-capture, using gaze and scene fusion to decide which video snippets to record (Chang et al., 2021). MemEye addresses the downstream question that such systems raise: once multimodal experiences have been captured, can a memory architecture preserve the right evidence and later reason over it correctly? The two projects therefore occupy complementary positions in a broader pipeline for long-term multimodal memory.

The published limitations are explicit. MemEye covers only eight curated life-scenario tasks; some tasks use generated or rendered content; human evaluation is limited; and method comparisons are system-level rather than controlled component ablations. The paper also notes privacy and safety concerns, since stronger visual memory implies storage of personal visual histories. Those caveats do not weaken the framework’s core contribution. They indicate that MemEye is intended as a diagnostic foundation for future work on entity-centric memory, visual world models, explicit conflict resolution, and privacy-preserving multimodal storage (Guo et al., 14 May 2026).