DISBench: Context-Aware Image Retrieval Benchmark

• DISBench is the first large-scale benchmark designed to evaluate context-aware image retrieval, requiring multi-step, corpus-level reasoning across visual histories.
• The benchmark employs a human–model collaborative pipeline that synthesizes queries using graph-based memory and explicit metadata, ensuring high annotation accuracy.
• DISBench exposes limitations in current multimodal retrieval systems and drives research towards agentic, memory-augmented exploration for more robust visual search.

DISBench is the first large-scale benchmark specifically designed to evaluate context-aware image retrieval in visual histories, reframing retrieval as an agentic exploration and reasoning problem that inherently requires multi-step, context-driven navigation across photo corpora. Developed in conjunction with the DeepImageSearch paradigm, DISBench aims to surface the limitations of existing retrieval models when confronted with queries that demand spatiotemporal and cross-event reasoning, rather than independent per-image semantic matching (Deng et al., 11 Feb 2026).

1. Definition and Core Objectives

DISBench consists of 122 manually verified, context-dependent queries operating over 109,467 photos from 57 Flickr users, covering an average span of 3.4 years (up to 2,000 images per user). Each query requires not only retrieving relevant images but also (i) discovering latent anchor entities or events within an image history, (ii) interpreting and enforcing temporal or spatial constraints, and (iii) returning the complete set of matching images. The benchmark’s principal goal is to evaluate an agent’s ability for corpus-level contextual reasoning, contrasting sharply with independent image-query matching paradigms.

Key objectives:

• Surface the limitations of current multimodal retrieval and vision–LLMs under long-horizon, multi-step search regimes.
• Provide a reusable dataset and baseline agent framework featuring explicit memory and compositional tool use.
• Promote research on agentic, context-aware reasoning in image retrieval.

2. Motivation: From Independent Matching to Agentic Exploration

Contemporary multimodal retrieval systems rely on embedding-based similarity (e.g., sim(v_i, text(Q))) and treat each candidate in isolation. While some recent systems incorporate reasoning modules, their architecture typically collapses to single-shot final matching. However, image collections of personal visual histories naturally fragment relevant evidence across events or photosets, and target images may lack salient standalone features.

Example: Solving “find concert photos where only the lead singer appears onstage” often requires first using visual clues (e.g., blue–and–white logo) to localize the correct event, then filtering within it for images containing only the vocalist. DISBench queries are explicitly constructed to require such multi-stage reasoning, making corpus-wide traversal and context chaining essential.

3. Dataset Construction and Context-Dependent Scenario Design

The dataset construction pipeline builds on YFCC100M, leveraging its user→photoset→photo hierarchy while withholding photoset boundaries from models. Each selected user (≥2,000 photos, ≥90% metadata coverage, ≥50 photosets, ≥1 year) contributes an entire chronological photo archive. Queries are stratified into:

• Intra-event (≈47%): Locate anchor events/clues, filter within.
• Inter-event (≈53%): Scan for recurring elements under spatiotemporal constraints.

A four-stage human–model collaborative pipeline synthesizes the benchmark:

1. Visual Semantic Parsing: Vision–LLMs ingest pixels, timestamps, GPS, and local context, extracting scene summaries, lists of salient visual clues (objects, logos, landmarks), and person clusters.
2. Latent Association Mining: For each clue, top-k candidates are retrieved both within and outside its source context using multimodal embeddings, followed by VLM-based cross-event verification.
3. Memory Graph Construction: Photos, events, clues, and persons are vertices. Edges capture photoset structure, clue association, and human-verifiable rationales for entity linkage.
4. Subgraph Sampling & Query Synthesis: Local subgraphs are selected by random walk and serialized for LLM-guided query generation, ensuring the resulting query demands chaining structural and association-based evidence. Human annotators reject queries solvable by direct matching and require that distractors force the use of context. Comprehensive annotation ensures high target exhaustivity with IoU = 0.91 between independent annotators.

4. Task Specification and Agent Toolset

Given a user’s image corpus $C = \{I_1, I_2, ..., I_N\}$ and a natural-language query $Q$, agents must predict $R \subseteq C$ satisfying the context-imposed criteria. Agents decompose each query into:

• Episode: The latent spatiotemporal segment to be localized.
• Episode Breakdown: Stepwise plan linking images via relations (e.g., logo presence + date).
• Target: Final constraints (e.g., “only lead singer visible”).

Agents interact via a tool suite:

• ImageSearch: semantic/text/image-based top-k retrieval.
• GetMetadata, FilterMetadata: read/filter by time/geolocation.
• ViewPhotos: perform fine-grained visual confirmation.
• WebSearch: resolve named entities via external sources.

Memory mechanisms include persistent state variables for named subsets and compressed session/working memory for efficient context compression under token limits.

5. Evaluation Protocols and Baselines

Performance is measured along both strict and partial matching axes:

• Agentic metrics: Exact Match (EM), precision, recall, and F1 between predicted and gold targets across queries.

$$\mathrm{EM} = \frac{1}{N}\sum_{i=1}^N \mathbb{1}\left[\hat{R}_i = R_i\right]$$

$$\mathrm{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

• Retrieval baselines: Mean Average Precision (MAP@k), Recall@k, and NDCG@k based on semantic embeddings or cross-modal retrieval.

Baseline agent “ImageSeeker” operates by alternately leveraging semantic retrieval, metadata filtering, visual inspection, and external lookup, orchestrated by structured system prompts and dual-memory architecture.

Model	Intra-Event EM/F1	Inter-Event EM/F1	Overall EM/F1
GPT-4o	5.3 / 19.6	9.2 / 24.5	7.4 / 22.2
GPT-5.2	10.5 / 38.0	12.3 / 32.6	11.5 / 35.1
Claude-Opus-4.5	35.1 / 57.9	29.2 / 53.4	32.0 / 55.5

Ablations indicate large drops without explicit memory (–4.9 F1) or metadata tools (–5.7 F1). Test-time ensembling (“Best@k”) significantly increases F1, revealing latent model ability that is otherwise masked by suboptimal search path selection.

6. Empirical Findings and Analysis

Empirical evaluation reveals that even the strongest LLM agents achieve moderate overall F1 (≤ 55.5%), with frequent failure modes including:

• Multi-step reasoning failures (>40%): Agents lose track of complex search plans.
• Cross-event anchoring errors: Failure to localize anchor events or clues.
• Visual discrimination errors: Subtle visual cues missed, resulting in false dismissals.

Standard embedding-based methods yield low Recall@3 and NDCG@5, demonstrating the necessity of multi-step, tool-driven, memory-augmented agentic exploration.

7. Limitations and Future Research Directions

DISBench exposes a wide gap between current multimodal systems and robust, context-aware retrieval. Promising future directions include:

• Learned planning and reflection strategies for improved path selection.
• Architectures with enhanced memory for richer state tracking and backtracking.
• Retrieval–reasoning pipelines that incorporate association mining at inference.
• Expanded benchmarks with richer modalities (e.g., video) and more efficient annotation protocols.

By quantifying the gap in agentic context reasoning, DISBench provides a rigorous testbed that sets the stage for next-generation multimodal retrieval systems equipped for real-world, context-intensive scenarios (Deng et al., 11 Feb 2026).