MagicLens: Self-Supervised Image Retrieval

• MagicLens is a self-supervised image retrieval framework that leverages free-form text instructions to specify and retrieve multifaceted visual and semantic relations.
• The framework employs a dual-encoder architecture with shared vision and text encoders, enhanced by self-attention layers to produce unified image-text embeddings.
• MagicLens demonstrates state-of-the-art performance on multiple benchmarks while enabling real-time, open-ended retrieval in practical applications.

MagicLens is a self-supervised image retrieval framework that leverages open-ended text instructions to specify rich, multi-faceted visual and semantic relations between images. Unlike traditional retrieval systems that are restricted to either purely visual similarities or a limited set of pre-defined relationships, MagicLens enables retrieval conditioned on free-form instructions, demonstrating strong parameter efficiency and state-of-the-art performance on diverse retrieval benchmarks. Models and code are publicly available at https://open-vision-language.github.io/MagicLens/ (Zhang et al., 2024).

1. Model Architecture

MagicLens employs a dual-encoder design, where both images and accompanying instructions are projected into a shared embedding space. The architecture comprises vision and text encoders initialized from either CLIP (B/16 or L/14) or CoCa (B or L) checkpoints. After independent tokenization and encoding of the image and text, their token sequences are concatenated and passed through four additional self-attention layers. The resulting combined sequence is aggregated into a fixed-length embedding $r$ via a multi-head attention pooler.

Both the query stream (image plus instruction) and the target stream (image plus empty string) share all model weights, differing only in the textual input. This parameter sharing is a primary contributor to MagicLens's efficiency.

Embedding Diagram

• (Query image + instruction): Encoded with the shared image and text encoders, followed by 4 self-attention layers and attention pooling to produce $r_q$.
• (Target image + empty string): Same pathway and parameters, producing $r_t$.

2. Training Data and Instruction Synthesis

The MagicLens dataset consists of 36.7 million triplets: $$(\text{image}_q, \text{instruction}, \text{image}_t)$$. Images are mined from Common Crawl by aggregating all image URLs within individual web pages, followed by aggressive cleaning to remove near-duplicates (CLIP score > 0.98), low-resolution images, ads, and highly redundant groups.

Relation Mining

Pairs are selected to capture both visual and semantic relevance:

• Visual similarity: CLIP image-to-image similarity $\geq 0.82$
• Non-visual connection: Caption-to-caption similarity based on LLM-generated captions $\geq 0.90$
• No more than three pairs per page are retained to ensure dataset diversity.

Extensive metadata is associated with each image, including HTML alt-text, Google Vision ICA labels (~25 per image), and PaLI(-X) captions.

Instruction Generation

Instructions expressing the relation are synthesized using PaLM 2. With as input the alt-text, ICA labels, and captions for both images, a prompt with few-shot demonstrations and chain-of-thought is used to generate succinct, template-free instructions emphasizing both general similarity and distinguishing features specific to the target. This produces high-quality, open-ended instructions that extend beyond attribute or object-level edits, encompassing function, domain, and other non-visual relations.

Example: (Nikon camera, Nikon charger) yields the instruction “Find a charger for it.”

3. Self-Supervised Contrastive Training Objective

MagicLens is trained via a batch-based image-text contrastive loss. For a batch of $N$ triplets, loss for the $i$th sample is:

$$L_i = -\log \frac{\exp(\operatorname{sim}(r_q^i, r_t^i)/\tau)} {\sum_{j=1}^N \left[ \exp(\operatorname{sim}(r_q^i, r_t^j)/\tau) + \exp(\operatorname{sim}(r_q^i, r_t^{j'})/\tau) \right]}$$

where

• $r_q^i$: embedding for (query image + instruction)
• $r_q$0: embedding for (target image + “”)
• $r_q$1: embedding for (query image + “”), treated as a hard negative
• $r_q$2
• $r_q$3: learned temperature parameter

This loss encourages the embedding of the (query image + instruction) pair to be close to its corresponding target image, while pushing it away from all other images, including the unaltered query image.

4. Parameter Efficiency and Scalability

MagicLens achieves high retrieval performance with significantly reduced parameter count compared to prior art:

Model	Backbone	Parameters	Prior SOTA (params)
MagicLens-B	CLIP-B	166 M	CIReVL (12–14 B)
MagicLens-B	CoCa-B	267 M	LinCIR/CompoDiff (0.5–2.9 B)
MagicLens-L	CLIP-L	465 M
MagicLens-L	CoCa-L	613 M

CIReVL, using a BLIP2+FLAN-T5-XXL+CLIP stack, requires roughly 14 billion parameters, while LinCIR and CompoDiff utilize large CLIP variants up to 2.9 billion parameters. Even MagicLens-B (267M) surpasses CIReVL’s performance on several benchmarks, enabled by direct dual-encoder training and extensive self-supervised data.

The dual-encoder architecture enables sub-100M parameter models to be indexed via standard approximate nearest neighbor algorithms, supporting real-time Retrieval over million-scale databases.

5. Benchmark Results

MagicLens demonstrates state-of-the-art or competitive zero-shot performance across eight major retrieval benchmarks:

• Composed Image Retrieval, Domain Transfer, Conditional Similarity (CoCa-L, 613M params)
  • FIQ (R@10 average over dress/shirt/toptee): 38.0 (prior best 36.0)
  • CIRR (R@1 full/subset): 33.3 / 70.9 (prior best 27.2/59.5)
  • CIRCO (mAP@5): 34.1 (prior best 12.6)
  • DTIN (R@10): 48.2 (prior best 23.8)
  • GeneCIS (R@1): 16.7 (prior best 15.9)
• Sketch-Based Image Retrieval (MagicLens-L CoCa-L, 613M)
  • TU-Berlin (mAP): 70.2 (prior 56.9)
  • Sketchy (mAP@200): 75.7 (prior 52.5)
  • QuickDraw (mAP): 19.7 (prior 14.5)
• Text-to-Image Retrieval
  • Improvements of ~1–4 points on Flickr30K and MS COCO benchmarks

Single model checkpoints are used for all experiments, underscoring the generalizability of the MagicLens approach.

6. Human Evaluation and Diversity of Search Intents

MagicLens supports a wide spectrum of user-specified search intents, including those extending well beyond visual similarity. In human evaluations over an unseen 1.4M-image index using 150 queries:

• “Simple visual” (single-difference): MagicLens-L preferred in 50.7% vs. LinCIR’s 41.3%
• “Complex visual” (multi-difference): 61.3% vs. 24.0%
• “Beyond visual” (no visual similarity, e.g., “find other attractions in this country”): 80.0% vs. 4.7%

This demonstrates the coverage of real-world, open-ended retrieval scenarios using concise, flexible instructions synthesised from large-scale web data.

7. Applications and Implications

MagicLens can be integrated into retrieval pipelines in settings such as visual search UIs or e-commerce product modification tools, supporting free-form, template-free natural language queries for image retrieval at scale. Embeddings for millions of images with optional instruction context can be retrieved in under 100 ms using standard nearest neighbor infrastructure. The instruction interface enables specification of implicit relations—including attribute modification, function, domain, and temporal queries—without needing complex rule-based template engineering.

Potential extensions include interactive refinement loops (“refine my search”) and retrieval-augmented visual question answering, highlighting the adaptability of the MagicLens dual-encoder paradigm to a range of open-ended, instruction-driven vision–language tasks (Zhang et al., 2024).