subFlickr: Compositional Text-Image Retrieval Benchmark
- subFlickr is a benchmark for text–image retrieval that uses short, compositional queries to capture fine-grained visual details.
- It transforms full-sentence Flickr30k captions into curated subcaptions, employing dual-stage semantic matching and visual verification for relevance.
- Empirical evaluations reveal that standard vision-language models exhibit bag-of-words biases, highlighting the need for improved compositional understanding.
subFlickr is a text–image retrieval benchmark derived from the Flickr30k test split and introduced to evaluate compositional generalization under short, realistic search queries. It is built from the original Flickr30k test images, but replaces full-sentence captions with curated sub-sentential queries, or “subcaptions,” so that retrieval is driven by short compositional phrases such as “a person with a white shirt” rather than broad scene descriptions (Ntinou et al., 23 Sep 2025). A broader, inferred reading from earlier Flickr research is that “subFlickr” can also denote a task-specific subset of Flickr images or metadata created for a particular problem, but the explicit benchmark name belongs to the retrieval setting introduced in 2025 (Ntinou et al., 23 Sep 2025).
1. Definition and conceptual scope
In its explicit sense, subFlickr is a benchmark for text–image retrieval in which the images are taken from the Flickr30k test set and the queries are short, compositional captions. The benchmark contains $935$ images, $280$ queries, an average query length of $4.5$ tokens, and $6.0$ captions per image on average; its query type is “Compositional Caption” (Ntinou et al., 23 Sep 2025).
Its stated purpose is to address two limitations of established retrieval benchmarks. First, widely used datasets such as Flickr30k and MS COCO are dominated by long, full-sentence captions, which “do not reflect how people actually search for images.” Second, keyword-style annotations lack “semantic richness, compositionality, and alignment with real user queries.” subFlickr is therefore positioned between full-sentence caption benchmarks and flat tag-style datasets, using short but semantically structured phrases rather than either complete sentences or isolated tags (Ntinou et al., 23 Sep 2025).
Relative to Flickr30k itself, the benchmark changes three properties at once. The granularity shifts from whole-scene description to specific compositional aspects such as objects, attributes, actions, and relations. The linguistic form shifts from complete captions to short subphrases. The evaluation target shifts from broad semantic matching to fine-grained compositional retrieval, with the stated aim of creating “a new text-image retrieval benchmark that cannot be easily solved by VLMs exhibiting bag-of-words behaviour” (Ntinou et al., 23 Sep 2025).
A plausible implication is that subFlickr should be understood not merely as a reduced version of Flickr30k, but as a change in the query distribution and the relevance relation. The same underlying images are retained, yet the retrieval problem becomes materially different because relevance is defined at the level of short compositional predicates rather than caption-level scene summaries.
2. Construction from Flickr30k
The construction pipeline begins from the Flickr30k test images and their five captions per image. Each caption is parsed with a pretrained constituency parser to decompose it into constituent nodes, including complete captions, sentences, sub-phrases, and lexical items. From this structure, the benchmark creators extract recurring subphrases that are likely to be visually grounded, then manually curate a set of short compositional expressions to serve as retrieval queries (Ntinou et al., 23 Sep 2025).
The image–query relevance relation is assigned in two stages. First, text-to-text similarity is computed between the extracted ground-truth subcaptions and the curated queries using the text encoder Bge-large-en-v1.5. Second, two vision–LLMs, Qwen2-VL-7B and InternVL2.5-8B, verify whether each query is visually present in the image. A query is assigned to an image only if both models agree. As a final step, of the dataset is visually inspected (Ntinou et al., 23 Sep 2025).
This methodology is notable because it avoids treating subcaptions as automatically correct labels solely by syntactic extraction. The assignment procedure explicitly combines semantic matching and visual verification, with agreement between two VLMs used as a reliability constraint. The resulting benchmark therefore encodes many-to-many query–image relevance, rather than one caption per image or one image per query.
The benchmark’s frequency structure is also reported as long-tailed: a few queries are frequent, many are rare. This suggests closer alignment with realistic search behavior than uniformly balanced query sets. The construction procedure consequently targets both compositionality and ecological validity in retrieval evaluation (Ntinou et al., 23 Sep 2025).
3. Query structure and evaluation design
subFlickr queries are short phrases with average length $4.5$ tokens. They are explicitly described as compositional expressions, with examples centered on noun phrases and related sub-sentential forms. The benchmark is consistently characterized as targeting attributes, relations, actions, and object configurations, although the paper does not provide a formal taxonomy analogous to the Replace/Swap/Add categories used in some other compositionality benchmarks (Ntinou et al., 23 Sep 2025).
The retrieval task is defined as follows: for each short query, the system must retrieve all relevant images from the test set. Each query is associated with a binary relevance label over the test images. Evaluation uses mean Average Precision and F1-score, specifically F1@1, F1@5, and F1@10, where the latter metrics are computed from the top- ranked images for each query (Ntinou et al., 23 Sep 2025).
No train/validation/test split is introduced within subFlickr itself. It is used purely as an evaluation benchmark layered on top of the Flickr30k test split. This design is consequential: performance on subFlickr is intended to measure transfer and generalization rather than adaptation to the benchmark. In the reported experiments, models are evaluated in zero-shot mode or after training on external synthetic or auxiliary data, but not on subFlickr labels themselves (Ntinou et al., 23 Sep 2025).
A useful way to frame the benchmark is as a retrieval-oriented complement to smaller controlled compositionality tests. Prior datasets such as SugarCrepe, SugarCrepe++, and Winoground emphasize paired or adversarial examples and often evaluate discrimination in tightly controlled settings. subFlickr instead uses large-scale ranking over hundreds of images with multiple relevant matches per query, thereby shifting the emphasis toward practical search behavior (Ntinou et al., 23 Sep 2025).
4. Empirical performance and what the benchmark reveals
The principal reported comparison is between standard dual-encoder VLMs, tag-oriented models, and a vision-free retriever based on textual scene descriptions. On subFlickr, the reported results are:
| Model | mAP | F1@5 |
|---|---|---|
| CLIP (ViT-B) | 29.2 | 20.0 |
| OpenCLIP (ViT-BigG/14) | 36.5 | 25.4 |
| RAM++ | 49.1 | 32.7 |
| LexiCLIP (0.3B) zero-shot | 45.6 | 30.6 |
| LexiCLIP (0.3B) fine-tuned | 55.1 | 37.6 |
The same trend appears in F1@1 and F1@10. LexiCLIP fine-tuned reaches mAP $55.1$, F1@1 $17.9$, F1@5 $37.6$, and F1@10 $280$0. RAM++ reaches mAP $280$1, F1@1 $280$2, F1@5 $280$3, and F1@10 $280$4. Conventional CLIP-like baselines remain in the approximate 29–37 mAP range (Ntinou et al., 23 Sep 2025).
These results are interpreted in the source work as evidence that short compositional retrieval exposes weaknesses that are less visible on standard full-sentence benchmarks. The relatively low subFlickr scores of CLIP-like models, despite their strong standing on Flickr30k-style evaluation, are presented as support for the claim that standard VLMs often exhibit shallow language understanding and bag-of-words behavior. By contrast, a text-only retrieval formulation built on generated image descriptions is reported to reduce the modality gap and improve compositional grounding (Ntinou et al., 23 Sep 2025).
The benchmark also highlights the strength of explicit tag supervision. RAM and RAM++ perform strongly in zero-shot evaluation, which the authors attribute to training on $280$5 million image–tag pairs and to explicit object–attribute supervision. Yet the vision-free retriever is reported to surpass them after fine-tuning, despite using only text as the retrieval representation (Ntinou et al., 23 Sep 2025).
A plausible implication is that subFlickr functions as a stress test for retrieval systems whose apparent success depends on broad semantic overlap rather than compositional discrimination. Because many subFlickr queries differ by attribute, relation, or object configuration, models that are insensitive to these distinctions are penalized more directly than on full-caption retrieval tasks.
5. Relation to earlier Flickr subset traditions
Earlier arXiv work does not use the benchmark name “subFlickr,” but it repeatedly describes task-specific subsets of Flickr data created for narrowly defined research problems. This suggests an informal broader usage in which “subFlickr” denotes a curated or structured subset of Flickr constructed for a particular task.
A clear example is the Flickr-Fire dataset introduced for fire detection from social media images. That work constructs a Flickr-based subset through the Flickr API, applies a Creative Commons licensing filter, starts from 5,962 images retrieved with fire-related textual queries, and then produces a balanced experimental set of 2,000 manually annotated images, with binary labels $280$6 and 10-fold cross-validation (Bedo et al., 2015). In that context, the term “subFlickr” is not explicit, but the paper’s own structured explanation states that the collection is “a subset of Flickr” organized for a specific crisis-monitoring task.
A related implicit usage appears in work on social recommendation and social image recommendation. One paper formulates personalized image tagging on a subset of Flickr images drawn from NUS-WIDE, with user-specific ranking models over ordered tags and semi-supervised augmentation of sparse user histories (Nwana et al., 2016). Another introduces the FlickrUserFavor dataset, a Flickr-based collection with 350,000 images, 20,298 users, binary “like” feedback, social relations, and 42,496-dimensional visual features for social image recommendation (Liu et al., 2016). In both cases, the operative object is again a task-defined Flickr subset rather than Flickr as an undifferentiated corpus.
The same pattern recurs in other domains. Flickr-derived subsets support work on popularity prediction over 30-day engagement sequences (Dutta et al., 2021), housing-quality prediction from geotagged Flickr imagery filtered into “Flickr Outdoors” and “Flickr Buildings” subsets (Levering et al., 2024), tourist-attraction recommendation from geo-tagged Flickr photos (Chen et al., 2021), and large-scale learning from user-supplied Flickr tags in the YFCC100M setting (Izadinia et al., 2014). In these papers, “subFlickr” is best read as an inferred umbrella term for a recurrent methodological practice: constructing a specialized Flickr subcorpus with task-specific labels, filters, or relevance definitions.
6. Broader significance, release, and limitations
subFlickr is released under the MIT license, and the authors state that the derived benchmarks are released exclusively for research use, consistent with the original licensing and access conditions of COCO and Flickr30k. The benchmark is distributed alongside code in the LexiCLIP evaluation suite (Ntinou et al., 23 Sep 2025).
Its broader significance lies in the evaluation axis it adds. Standard caption retrieval benchmarks emphasize alignment with long descriptions; tag datasets emphasize coverage of lexical labels; paired compositionality tests emphasize adversarial discrimination. subFlickr instead focuses on realistic short-form compositional search, many-to-many relevance, and ranking-based evaluation. This suggests a distinct role for the benchmark in diagnosing whether retrieval systems can support practical search behavior rather than only caption matching (Ntinou et al., 23 Sep 2025).
The benchmark also intersects with the paper’s argument for privacy-friendly retrieval. In the associated vision-free pipeline, raw images are replaced by structured textual descriptions, which the authors describe as more privacy-friendly because identity-related information such as faces or private rooms is removed in the conversion to text. subFlickr serves as one of the testbeds for showing that such caption-only systems can still perform strongly on short compositional queries (Ntinou et al., 23 Sep 2025).
Its limitations are also explicit. It is built only from the Flickr30k test split; it contains $280$7 images and $280$8 query types; and its labels depend on a pipeline that combines text similarity, two VLM verifiers, and manual inspection of only $280$9 of the dataset. The long-tailed query distribution is realistic, but it also implies nonuniform coverage of compositions. Moreover, because the benchmark is evaluation-only, it does not by itself provide a training resource for learning compositional retrieval; rather, it functions as a diagnostic benchmark for models trained elsewhere (Ntinou et al., 23 Sep 2025).
Taken together, the available evidence supports a two-level understanding of the term. In its precise contemporary sense, subFlickr is a benchmark for short-query compositional retrieval derived from Flickr30k. In a broader inferred sense drawn from earlier Flickr research, it belongs to a longer tradition of constructing narrowly scoped Flickr subsets for tasks such as crisis analysis, recommendation, popularity prediction, geospatial inference, and personalized tagging.
