PhotoBench: Personalized Photo Retrieval

• PhotoBench is a benchmark designed for personalized intent-driven photo retrieval that integrates visual semantics, spatial-temporal metadata, social identity, and event context.
• It leverages authentic, metadata-rich personal albums to drive intent-based queries, addressing retrieval challenges that go beyond mere visual matching.
• The benchmark evaluates multi-source fusion and zero-ground-truth queries, highlighting limitations in unified embeddings and naive tool orchestration.

PhotoBench is a benchmark for personalized intent-driven photo retrieval built from authentic, metadata-rich personal photo albums, introduced to shift retrieval evaluation from visual matching to personalized multi-source intent-driven reasoning. In this formulation, a personal album is treated not as a set of context-isolated web images but as a temporally continuous, socially entangled, metadata-rich life archive, so realistic queries may depend jointly on visible content, timestamps, locations, recurring identities, and event context. The benchmark formalizes this setting through multi-source profiling, intent-driven query synthesis, dense one-to-many ground truth, and zero-ground-truth queries for abstention testing (Xu et al., 2 Mar 2026).

1. Problem formulation and retrieval setting

PhotoBench targets a retrieval regime in which a natural-language query is posed against a personal album, and the system must return all relevant images. The central claim is that many realistic gallery-search queries cannot be resolved by appearance similarity alone. Queries such as those tied to a specific trip, social relation, or event order require combining visual semantics, spatial-temporal metadata, social identity, and temporal event context rather than relying on a single shared embedding space (Xu et al., 2 Mar 2026).

The benchmark represents each image $i$ by a structured profile

$$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$

where $\mathcal{V}_i$ denotes visual semantics, $\mathcal{M}_i$ spatial-temporal metadata, $\mathcal{F}_i$ social identity, and $\mathcal{E}_i$ temporal event context. A user intention descriptor is then inferred as

$$\mathcal{I}_i = \operatorname{MLLM}\big(\mathcal{P}_i,\ [\mathcal{E}_j]_{j \le i}\big),$$

and queries are synthesized from subsets of these sources so that successful retrieval requires satisfying their conjunction rather than matching only visible scene content (Xu et al., 2 Mar 2026).

This design is explicitly framed against web-image retrieval benchmarks such as MSCOCO and Flickr30k, which the paper characterizes as relying on context-isolated web snapshots and descriptive caption matching. PhotoBench instead emphasizes one-to-many retrieval, unmatched / zero-ground-truth queries, narrative and personalized queries, multi-source reasoning, realistic personal metadata, temporal continuity, high image quality variance, near-duplicates / burst shots, and dense candidate pools in a personal-gallery setting (Xu et al., 2 Mar 2026).

2. Corpus construction and multi-source profiling

PhotoBench is built from 3,582 images drawn from three authentic personal albums, paired with 1,188 bilingual queries (Chinese and English). The albums collectively span 2018 to 2025; 83.4% of images retain valid high-precision GPS and timestamp metadata; the corpus contains 20 distinct recurring individuals; and 25.1% of images are portraits. The three albums are characterized as event-centric, person-centric, and balanced (Xu et al., 2 Mar 2026).

Collection follows two stated principles: holistic metadata retention and privacy review with minimal curation. The albums were obtained from consenting participants, preserving timestamps, GPS coordinates, and device headers exactly as captured by source devices. Participants could flag sensitive material, and expert privacy review removed or masked highly private or identifying content such as confidential documents or personal IDs. Beyond that filtering, the benchmark intentionally avoided additional manual pruning in order to preserve the natural structure of real albums (Xu et al., 2 Mar 2026).

The benchmark’s multi-source profiling pipeline operationalizes each image along four channels. Visual semantics are extracted with GPT-4o and include salient objects, human poses, scene composition, and aesthetic attributes. Spatial-temporal metadata are converted into semantic descriptors through reverse geocoding and time normalization, producing tags such as “early morning,” “weekend,” and “Halloween.” Social identity is constructed by face detection, face clustering, ego-identification of the album owner, and expert annotation of recurring clusters with plausible social roles such as spouse or colleague based on co-occurrence patterns. Temporal events are derived by hierarchical temporal clustering using an adaptive 4-hour window and a minimum cluster size of 3 images, with each event assigned a concise textual summary (Xu et al., 2 Mar 2026).

The benchmark is image-level in output, but event structure is foundational in construction. That design allows queries to refer to event order, travel episodes, or recurring gatherings while still evaluating retrieval as a set of returned images rather than as event prediction. A plausible implication is that PhotoBench treats event modeling as latent retrieval infrastructure rather than as the end task itself.

3. Intent-driven query synthesis and source-aware taxonomy

PhotoBench synthesizes queries in three stages. First, for each anchor image, it infers a trajectory-conditioned intention descriptor from the image profile and the chronological sequence of earlier event summaries. Second, it samples a subset of sources and prompts an LLM/MLLM to generate concise, colloquial, search-style queries that are logically consistent with that intention and strictly require the chosen source combination. Third, it performs exhaustive ground-truth mining using Top-$K$ visual retrieval, semantic retrieval, and agentic multi-tool retrieval, with $K=50$ used empirically, followed by expert verification of all positives and rejection of ambiguous cases (Xu et al., 2 Mar 2026).

A distinctive feature is the inclusion of Zero-GT queries, introduced to simulate false memory. These are created by metadata perturbation, semantic variation, entity mismatch, scene mismatch, and detail enhancement, then human-verified to ensure that no relevant image exists. This makes abstention part of the benchmark rather than an auxiliary analysis (Xu et al., 2 Mar 2026).

The benchmark’s diagnostic structure is organized by a Source-Aware Query Taxonomy. Atomic categories isolate one source at a time, while compositional categories require joint reasoning over multiple sources.

Category	Required source(s)	Example
$S_V$	Vision-only	“photo of red flowers”
$$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$0	Metadata-only	“photos from Tokyo in 2025”
$$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$1	Face / identity-only	“photo of my sister”
$$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$2, $$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$3, $$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$4, $$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$5	Compositional	Multi-source conjunctions

These categories are defined as strict and non-overlapping. The paper emphasizes that many real personal queries fall into composite classes, especially those requiring time, place, and social-role disambiguation simultaneously. This taxonomy is central to the later diagnosis of model failures because it separates failures of visual understanding from failures of metadata access, face reasoning, and source fusion (Xu et al., 2 Mar 2026).

4. Evaluation protocol and system classes

PhotoBench evaluates three system families. The first consists of unified embedding models, including multimodal embedding systems such as CLIP, SigLIP2-base, SigLIP2-giant, VLM2Vec, Qwen3-VL-Embedding-2B, Qwen3-VL-Embedding-8B, Ops-MM-embedding-v1, RzenEmbed-v2-7B, and QQMM-embed-v2, as well as caption-based text pipelines that first caption images with GPT-4o and then retrieve with models such as multilingual-e5, bge-m3, and Qwen3-Embedding variants. The second family consists of tool-based agentic systems, instantiated with backbones such as ToolACE-2-Llama-3.1-8B, Qwen3-8B, Qwen3-32B, DeepSeek-V3, Qwen3-235B-A22B, GPT-4o, OpenAI-o3, Claude-Sonnet-4-5, and Claude-Opus-4-5. The third family comprises six anonymized flagship phones, evaluated through their native gallery search interfaces as Phone A through Phone F (Xu et al., 2 Mar 2026).

The agentic systems are equipped with tools aligned to the source taxonomy: $$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$6, a vector search engine over a FAISS index using RzenEmbed-v2-7B; $$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$7, a metadata filter over time, GPS, and POI; $$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$8, a face/identity search engine; and $$\mathcal{P}_i = \{\mathcal{V}_i, \mathcal{M}_i, \mathcal{F}_i, \mathcal{E}_i\},$$9, a set composition tool for logical set operations. These tools make the retrieval pipeline explicitly modular rather than collapsing all constraints into one embedding space (Xu et al., 2 Mar 2026).

Because outputs differ by paradigm, PhotoBench uses two metric families. For embedding models it reports Recall@K and NDCG@K with $\mathcal{V}_i$0. For agents and phones, which return variable-length result sets, it reports Precision, Recall, and F1. For Zero-GT queries it reports Reject-Precision, Reject-Recall, and Reject-F1, thereby evaluating whether a system can return an empty set when appropriate (Xu et al., 2 Mar 2026).

The mobile-gallery evaluation protocol is unusually concrete: each phone is factory-reset, album images are transferred, the device is allowed 24 hours for indexing, the native gallery query is issued, up to 100 returned results are recorded, and two annotators verify relevance. This makes commercial on-device systems directly comparable with research agents and embedding pipelines under the same query set (Xu et al., 2 Mar 2026).

5. Empirical findings: modality gap and source fusion paradox

The benchmark’s main empirical conclusions are summarized as two named failure modes: the modality gap and the source fusion paradox. The modality gap refers to the failure of unified embedding models on non-visual constraints such as metadata and private identity. In the source-type decomposition at Recall@10, embeddings are competitive on $\mathcal{V}_i$1 queries, but collapse on pure metadata and face queries: for $\mathcal{V}_i$2, the reported scores are 57.9 for agents, 7.2 for multimodal embeddings, and 7.9 for caption-based text pipelines; for $\mathcal{V}_i$3, they are 75.1, 11.7, and 8.4, respectively. On some composite categories such as $\mathcal{V}_i$4, $\mathcal{V}_i$5, and $\mathcal{V}_i$6, embeddings appear stronger, but the paper attributes much of that to a visual-anchor effect, where non-visual constraints correlate with distinctive visual cues and are not actually being satisfied as hard constraints (Xu et al., 2 Mar 2026).

The source fusion paradox refers to the observation that simply adding more specialized tools does not reliably improve multi-source retrieval. In the ablation using Qwen3-235B-A22B, $\mathcal{V}_i$7 alone yields 35.1 on $\mathcal{V}_i$8; adding all relevant tools raises performance sharply on $\mathcal{V}_i$9 and $\mathcal{M}_i$0 but reduces $\mathcal{M}_i$1 to 32.5. The interpretation offered is that execution planning, tool orchestration, aggressive set intersection, and error propagation can cause valid results to be filtered out when multiple noisy intermediate retrieval sets are combined (Xu et al., 2 Mar 2026).

The headline ranking reflects this broader pattern. Among multimodal embeddings on the Chinese benchmark, RzenEmbed-v2-7B achieves the best R@1 = 27.2, while QQMM-embed-v2 achieves the best R@5 = 50.0 and R@20 = 65.4. Agents set a higher ceiling: Claude-Opus-4-5 achieves the best R@1 = 32.1, Claude-Sonnet-4-5 the best R@20 = 73.1, and OpenAI-o3 the best R@10 = 68.7. On set-based metrics for normal queries, Qwen3-235B-A22B reaches Precision = 50.8 and F1 = 51.9, while Claude-Sonnet-4.5 achieves the best Recall = 78.3. Among mobile systems, Phone F records the best normal-query F1 = 40.2 and the best zero-ground-truth Rej-F1 = 46.3 (Xu et al., 2 Mar 2026).

The complexity trend is equally important. For the strongest agent, average single-source F1 is 63.5, average dual-source F1 is 52.4, and triple-source $\mathcal{M}_i$2 falls to 33.9, with $\mathcal{M}_i$3 and $\mathcal{M}_i$4. Commercial phones also degrade sharply from single-source to dual-source and triple-source settings. The benchmark therefore argues that future progress in personal multimodal retrieval lies beyond unified embeddings and beyond naive tool addition, toward robust agentic reasoning systems capable of precise constraint satisfaction and multi-source fusion (Xu et al., 2 Mar 2026).

6. Position in the benchmark landscape and title ambiguity

Within the broader benchmark landscape, PhotoBench is narrowly defined by personalized retrieval over authentic personal albums. It is not a generic image benchmark, not a standard caption-retrieval benchmark, and not an album-understanding benchmark in the profile-reconstruction sense. That distinction matters because nearby benchmarks can look similar at the level of input modality while targeting different inference contracts. For example, PAL-Bench also operates on longitudinal personal albums, but its task is evidence-grounded profile reconstruction of owner facts, identities, and relations rather than image retrieval (Yan et al., 15 Jun 2026).

The title “PhotoBench” is also not unique in the recent literature. A separate paper uses PhotoBench to denote a multiple-choice benchmark for aesthetic visual understanding, derived from Reddit’s PhotoCritique community and comprising 1,500 questions across 284 photography sub-topics; that benchmark evaluates whether multimodal models can critique images like photographers, rather than retrieve personal photos from album archives (Qi et al., 23 Sep 2025). The two projects therefore share a name but differ in task, data source, and evaluation target.

In that broader context, the retrieval-oriented PhotoBench is best understood as a benchmark for personalized multi-source intent-driven reasoning over real-life albums. Its specific contribution is diagnostic rather than merely leaderboard-oriented: it makes visible the mismatch between current retrieval paradigms and realistic personal-gallery search, showing that visual similarity, metadata filtering, and tool access are each insufficient when the task requires exact conjunctions over time, place, identity, and life-event structure (Xu et al., 2 Mar 2026).