---
title: 'subFlickr: Compositional Text-Image Retrieval Benchmark'
url: https://www.emergentmind.com/topics/subflickr
type: topic
---

# subFlickr: Compositional Text-Image Retrieval Benchmark

subFlickr is a text–image retrieval benchmark derived from the Flickr30k test split and introduced to evaluate compositional generalization under short, realistic search queries. It is built from the original Flickr30k test images, but replaces full-sentence captions with curated sub-sentential queries, or “subcaptions,” so that retrieval is driven by short compositional phrases such as “a person with a white shirt” rather than broad scene descriptions [2509.19203]. A broader, inferred reading from earlier Flickr research is that “subFlickr” can also denote a task-specific subset of Flickr images or metadata created for a particular problem, but the explicit benchmark name belongs to the retrieval setting introduced in 2025 [2509.19203].

## 1. Definition and conceptual scope

In its explicit sense, subFlickr is a benchmark for text–image retrieval in which the images are taken from the Flickr30k test set and the queries are short, compositional captions. The benchmark contains \(935\) images, \(280\) queries, an average query length of \(4.5\) tokens, and \(6.0\) captions per image on average; its query type is “Compositional Caption” [2509.19203].

Its stated purpose is to address two limitations of established retrieval benchmarks. First, widely used datasets such as Flickr30k and MS COCO are dominated by long, full-sentence captions, which “do not reflect how people actually search for images.” Second, keyword-style annotations lack “semantic richness, compositionality, and alignment with real user queries.” subFlickr is therefore positioned between full-sentence caption benchmarks and flat tag-style datasets, using short but semantically structured phrases rather than either complete sentences or isolated tags [2509.19203].

Relative to Flickr30k itself, the benchmark changes three properties at once. The granularity shifts from whole-scene description to specific compositional aspects such as objects, attributes, actions, and relations. The linguistic form shifts from complete captions to short subphrases. The evaluation target shifts from broad semantic matching to fine-grained compositional retrieval, with the stated aim of creating “a new text-image retrieval benchmark that cannot be easily solved by VLMs exhibiting bag-of-words behaviour” [2509.19203].

A plausible implication is that subFlickr should be understood not merely as a reduced version of Flickr30k, but as a change in the query distribution and the relevance relation. The same underlying images are retained, yet the retrieval problem becomes materially different because relevance is defined at the level of short compositional predicates rather than caption-level scene summaries.

## 2. Construction from Flickr30k

The construction pipeline begins from the Flickr30k test images and their five captions per image. Each caption is parsed with a pretrained constituency parser to decompose it into constituent nodes, including complete captions, sentences, sub-phrases, and lexical items. From this structure, the benchmark creators extract recurring subphrases that are likely to be visually grounded, then manually curate a set of short compositional expressions to serve as retrieval queries [2509.19203].

The image–query relevance relation is assigned in two stages. First, text-to-text similarity is computed between the extracted ground-truth subcaptions and the curated queries using the text encoder Bge-large-en-v1.5. Second, two vision–language models, Qwen2-VL-7B and InternVL2.5-8B, verify whether each query is visually present in the image. A query is assigned to an image only if both models agree. As a final step, \(20\%\) of the dataset is visually inspected [2509.19203].

This methodology is notable because it avoids treating subcaptions as automatically correct labels solely by syntactic extraction. The assignment procedure explicitly combines semantic matching and visual verification, with agreement between two VLMs used as a reliability constraint. The resulting benchmark therefore encodes many-to-many query–image relevance, rather than one caption per image or one image per query.

The benchmark’s frequency structure is also reported as long-tailed: a few queries are frequent, many are rare. This suggests closer alignment with realistic search behavior than uniformly balanced query sets. The construction procedure consequently targets both compositionality and ecological validity in retrieval evaluation [2509.19203].

## 3. Query structure and evaluation design

subFlickr queries are short phrases with average length \(4.5\) tokens. They are explicitly described as compositional expressions, with examples centered on noun phrases and related sub-sentential forms. The benchmark is consistently characterized as targeting attributes, relations, actions, and object configurations, although the paper does not provide a formal taxonomy analogous to the Replace/Swap/Add categories used in some other compositionality benchmarks [2509.19203].

The retrieval task is defined as follows: for each short query, the system must retrieve all relevant images from the test set. Each query is associated with a binary relevance label over the test images. Evaluation uses mean Average Precision and F1-score, specifically F1@1, F1@5, and F1@10, where the latter metrics are computed from the top-\(K\) ranked images for each query [2509.19203].

No train/validation/test split is introduced within subFlickr itself. It is used purely as an evaluation benchmark layered on top of the Flickr30k test split. This design is consequential: performance on subFlickr is intended to measure transfer and generalization rather than adaptation to the benchmark. In the reported experiments, models are evaluated in zero-shot mode or after training on external synthetic or auxiliary data, but not on subFlickr labels themselves [2509.19203].

A useful way to frame the benchmark is as a retrieval-oriented complement to smaller controlled compositionality tests. Prior datasets such as SugarCrepe, SugarCrepe++, and Winoground emphasize paired or adversarial examples and often evaluate discrimination in tightly controlled settings. subFlickr instead uses large-scale ranking over hundreds of images with multiple relevant matches per query, thereby shifting the emphasis toward practical search behavior [2509.19203].

## 4. Empirical performance and what the benchmark reveals

The principal reported comparison is between standard dual-encoder VLMs, tag-oriented models, and a vision-free retriever based on textual scene descriptions. On subFlickr, the reported results are:

| Model | mAP | F1@5 |
|---|---:|---:|
| CLIP (ViT-B) | 29.2 | 20.0 |
| OpenCLIP (ViT-BigG/14) | 36.5 | 25.4 |
| RAM++ | 49.1 | 32.7 |
| LexiCLIP (0.3B) zero-shot | 45.6 | 30.6 |
| LexiCLIP (0.3B) fine-tuned | 55.1 | 37.6 |

The same trend appears in F1@1 and F1@10. LexiCLIP fine-tuned reaches mAP \(55.1\), F1@1 \(17.9\), F1@5 \(37.6\), and F1@10 \(40.0\). RAM++ reaches mAP \(49.1\), F1@1 \(15.3\), F1@5 \(32.7\), and F1@10 \(35.5\). Conventional CLIP-like baselines remain in the approximate 29–37 mAP range [2509.19203].

These results are interpreted in the source work as evidence that short compositional retrieval exposes weaknesses that are less visible on standard full-sentence benchmarks. The relatively low subFlickr scores of CLIP-like models, despite their strong standing on Flickr30k-style evaluation, are presented as support for the claim that standard VLMs often exhibit shallow language understanding and bag-of-words behavior. By contrast, a text-only retrieval formulation built on generated image descriptions is reported to reduce the modality gap and improve compositional grounding [2509.19203].

The benchmark also highlights the strength of explicit tag supervision. RAM and RAM++ perform strongly in zero-shot evaluation, which the authors attribute to training on \(14\) million image–tag pairs and to explicit object–attribute supervision. Yet the vision-free retriever is reported to surpass them after fine-tuning, despite using only text as the retrieval representation [2509.19203].

A plausible implication is that subFlickr functions as a stress test for retrieval systems whose apparent success depends on broad semantic overlap rather than compositional discrimination. Because many subFlickr queries differ by attribute, relation, or object configuration, models that are insensitive to these distinctions are penalized more directly than on full-caption retrieval tasks.

## 5. Relation to earlier Flickr subset traditions

Earlier arXiv work does not use the benchmark name “subFlickr,” but it repeatedly describes task-specific subsets of Flickr data created for narrowly defined research problems. This suggests an informal broader usage in which “subFlickr” denotes a curated or structured subset of Flickr constructed for a particular task.

A clear example is the Flickr-Fire dataset introduced for fire detection from social media images. That work constructs a Flickr-based subset through the Flickr API, applies a Creative Commons licensing filter, starts from 5,962 images retrieved with fire-related textual queries, and then produces a balanced experimental set of 2,000 manually annotated images, with binary labels \(\{\text{fire}, \text{not fire}\}\) and 10-fold cross-validation [1506.03844]. In that context, the term “subFlickr” is not explicit, but the paper’s own structured explanation states that the collection is “a subset of Flickr” organized for a specific crisis-monitoring task.

A related implicit usage appears in work on social recommendation and social image recommendation. One paper formulates personalized image tagging on a subset of Flickr images drawn from NUS-WIDE, with user-specific ranking models over ordered tags and semi-supervised augmentation of sparse user histories [1601.06440]. Another introduces the FlickrUserFavor dataset, a Flickr-based collection with 350,000 images, 20,298 users, binary “like” feedback, social relations, and 42,496-dimensional visual features for social image recommendation [1604.07044]. In both cases, the operative object is again a task-defined Flickr subset rather than Flickr as an undifferentiated corpus.

The same pattern recurs in other domains. Flickr-derived subsets support work on popularity prediction over 30-day engagement sequences [2108.01326], housing-quality prediction from geotagged Flickr imagery filtered into “Flickr Outdoors” and “Flickr Buildings” subsets [2403.08915], tourist-attraction recommendation from geo-tagged Flickr photos [2109.08275], and large-scale learning from user-supplied Flickr tags in the YFCC100M setting [1411.6909]. In these papers, “subFlickr” is best read as an inferred umbrella term for a recurrent methodological practice: constructing a specialized Flickr subcorpus with task-specific labels, filters, or relevance definitions.

## 6. Broader significance, release, and limitations

subFlickr is released under the MIT license, and the authors state that the derived benchmarks are released exclusively for research use, consistent with the original licensing and access conditions of COCO and Flickr30k. The benchmark is distributed alongside code in the LexiCLIP evaluation suite [2509.19203].

Its broader significance lies in the evaluation axis it adds. Standard caption retrieval benchmarks emphasize alignment with long descriptions; tag datasets emphasize coverage of lexical labels; paired compositionality tests emphasize adversarial discrimination. subFlickr instead focuses on realistic short-form compositional search, many-to-many relevance, and ranking-based evaluation. This suggests a distinct role for the benchmark in diagnosing whether retrieval systems can support practical search behavior rather than only caption matching [2509.19203].

The benchmark also intersects with the paper’s argument for privacy-friendly retrieval. In the associated vision-free pipeline, raw images are replaced by structured textual descriptions, which the authors describe as more privacy-friendly because identity-related information such as faces or private rooms is removed in the conversion to text. subFlickr serves as one of the testbeds for showing that such caption-only systems can still perform strongly on short compositional queries [2509.19203].

Its limitations are also explicit. It is built only from the Flickr30k test split; it contains \(935\) images and \(280\) query types; and its labels depend on a pipeline that combines text similarity, two VLM verifiers, and manual inspection of only \(20\%\) of the dataset. The long-tailed query distribution is realistic, but it also implies nonuniform coverage of compositions. Moreover, because the benchmark is evaluation-only, it does not by itself provide a training resource for learning compositional retrieval; rather, it functions as a diagnostic benchmark for models trained elsewhere [2509.19203].

Taken together, the available evidence supports a two-level understanding of the term. In its precise contemporary sense, subFlickr is a benchmark for short-query compositional retrieval derived from Flickr30k. In a broader inferred sense drawn from earlier Flickr research, it belongs to a longer tradition of constructing narrowly scoped Flickr subsets for tasks such as crisis analysis, recommendation, popularity prediction, geospatial inference, and personalized tagging.

Source: https://www.emergentmind.com/topics/subflickr