subCOCO: Short-Caption Retrieval Benchmark

• subCOCO is a COCO-derived benchmark for short, compositional retrieval queries that uses binary relevance labels over a fixed 4,030-image test set.
• It extracts visually grounded sub-phrases through constituency parsing and manual curation, capturing attribute, action, and relation semantics.
• The benchmark is integrated into the LexiCLIP framework to evaluate vision-language models on challenges like modality gaps and bag-of-words behavior.

Searching arXiv for the specified papers to ground the article in the current record. [Tool invocation] arxiv_search query="(Ntinou et al., 23 Sep 2025)" max_results=5 subCOCO is a test-time retrieval benchmark derived from the MS COCO test split and introduced for short-caption text-to-image retrieval in the context of compositional evaluation. It consists of a subset of COCO test images, a curated set of short compositional text queries termed sub-captions, and binary relevance labels for each query–image pair. In the reported release, subCOCO contains 4,030 images and 256 queries, with average query length 3.47 tokens and an average of 4.1 relevant images per query. Its stated purpose is to move away from long scene-level captions toward short, sub-sentential phrases that better reflect search behavior while stressing compositional understanding (Ntinou et al., 23 Sep 2025).

1. Definition, scope, and benchmark statistics

subCOCO is defined as a COCO-derived benchmark for short, compositional retrieval queries rather than for standard caption retrieval. The query type is explicitly identified as “Compositional Caption,” and the benchmark is paired with binary relevance labels over a fixed image set. This makes the task multi-label by construction: for a given query, multiple images may be relevant, and relevance is represented as a binary label over the 4,030-image test collection (Ntinou et al., 23 Sep 2025).

The benchmark is numerically characterized by 4,030 images, 256 queries, average query length of 3.47 tokens, and 4.1 average relevant images per query. These values position subCOCO between two regimes that the underlying paper contrasts. On one side are standard COCO and Flickr30k retrieval settings with long sentence captions; on the other are tag datasets with short but weakly structured keyword lists. subCOCO is intended to occupy the intermediate regime of short but still structured and compositionally meaningful phrases.

A central design claim is that subCOCO is meant to stress-test compositional retrieval. The benchmark is therefore not a simple truncation of COCO captions. Rather, it is organized around short expressions that preserve attribute, action, and relation structure. This suggests that subCOCO is best understood not as a variant of caption retrieval in the narrow sense, but as a retrieval benchmark for structured phrase semantics under realistic query brevity.

2. Construction from MS COCO captions

subCOCO is built from the test split of MS COCO, starting from COCO images and the five human-written captions per image. The construction procedure first applies a pretrained constituency parser, specifically spaCy’s implementation, to decompose each caption into constituent nodes including complete captions, sentences, sub-phrases, and individual lexical items. From this structure, the authors extract recurring subphrases that are likely to be visually grounded, including noun phrases, verb–object fragments, and prepositional phrases (Ntinou et al., 23 Sep 2025).

The extracted candidates are then manually curated into the final query inventory. The stated selection criterion is compositionality, with the paper explicitly giving “a person with a white shirt” as an example of the kind of expression chosen. The resulting queries are short, roughly 2–5 tokens on average, visually grounded, and intended to encode combinations such as object–attribute, object–action, and relation phrases. The paper notes that it does not print exact subCOCO examples in the main text; it instead gives analogous patterns such as “man in a green t-shirt,” “white dog on grass,” “red bus on street,” “woman holding umbrella,” and “child on skateboard.”

Query assignment to images proceeds in two stages. First, candidate matches are generated by computing text-to-text similarity between curated queries and parse-derived subcaptions using the text encoder BGE-large-en-v1.5. If $f(\cdot)$ denotes the encoder, the similarity is described in the standard embedding-based form as something like

$s(q, s) = \langle f(q), f(s) \rangle .$

Second, candidate query–image pairs are verified for visual presence using two VLMs, Qwen2-VL-7B and InternVL2.5-8B. A positive label is assigned only when both models agree that the query is visually present in the image. As a final quality-control step, 20% of the dataset is visually inspected manually.

The benchmark’s query-frequency distribution is described as long-tail, with subCOCO exhibiting “a slightly sharper drop in frequency” than subFlickr. This means that some generic queries have many relevant images while more specific queries have very few, a property that materially affects retrieval-as-ranking evaluation.

3. Task formulation and evaluation protocol

The task on subCOCO is short-caption text-to-image retrieval. The query space consists of curated short compositional phrases, and the target space consists of the COCO test images. The benchmark definition allows the target space to be instantiated either as images, for standard vision–LLMs, or as dense textual image descriptions, for the vision-free pipeline evaluated in the same work. The goal is stated explicitly: to recognize all relevant images given a short-caption query, where each query is associated with a binary relevance label over the test set (Ntinou et al., 23 Sep 2025).

For the vision-free LexiCLIP pipeline, each image is converted into a textual description $\mathcal{T}$, and a query $\mathcal{Q}$ may be prefixed by a prompt $p_Q$. Both are embedded with a single text encoder:

$\mathbf{f}_T = \Phi(\Theta, \mathcal{T}), \quad \mathbf{f}_Q = \Phi(\Theta, [p_Q; \mathcal{Q}]).$

Retrieval is then performed by similarity scoring, implicitly of the form

$s(\mathcal{Q}, \mathcal{T}) = \langle \mathbf{f}_Q, \mathbf{f}_T \rangle .$

For conventional baselines such as CLIP or RAM, the paper states that their original image encoders are used for COCO images and their text encoders are used for the short queries.

subCOCO is evaluated with retrieval-as-ranking metrics rather than with standard COCO Recall@$K$. The reported metrics are mean Average Precision and F1-score, specifically mAP together with F1@1, F1@5, and F1@10. mAP is defined as

$\text{mAP} = \frac{1}{|Q|} \sum_{q \in Q} \text{AP}_q,$

where $\text{AP}_q$ is Average Precision for the binary relevance vector of query $s(q, s) = \langle f(q), f(s) \rangle .$0. F1@K is computed from the top-$s(q, s) = \langle f(q), f(s) \rangle .$1 retrieved images treated as predicted positives, combining Precision@K and Recall@K in the standard way. This evaluation protocol differs from standard COCO retrieval, where Recall@$s(q, s) = \langle f(q), f(s) \rangle .$2 is used with caption–image matching and a single ground-truth image per query formulation.

The metric choice is significant because it aligns the benchmark with a multi-label ranking interpretation rather than single-match retrieval. A plausible implication is that subCOCO measures rank quality under ambiguous and overlapping phrase semantics more directly than standard caption retrieval benchmarks do.

4. Compositional semantics and sources of difficulty

subCOCO is explicitly presented as a compositional caption benchmark. The construction process and the examples described in the paper indicate that it probes several forms of composition: attribute–noun combinations such as “white shirt” or “red bus,” noun–noun and object–location combinations such as “tennis racket” or “dog on couch,” verb–object fragments such as “man riding bicycle,” multi-entity relations such as “man with a dog,” and prepositional phrases such as “on a sandy beach” or “in front of a building” (Ntinou et al., 23 Sep 2025).

The benchmark is described as difficult for several reasons. First, short queries amplify ambiguity: a phrase such as “man with a hat” may correspond to many images, so ranking quality depends on subtle distinctions rather than on broad scene overlap. Second, the paper argues that bag-of-words behavior is insufficient, because many query pairs share lexical items but differ in relation or attachment, as in “man behind dog” versus “dog behind man,” or “dog on skateboard” versus “dog near skateboard.” Third, compositional correctness matters even when constituent objects are present: an image containing both a dog and a skateboard is not sufficient evidence for the query “dog under skateboard.” Fourth, the long-tail query distribution means that many queries have few positives, increasing the sensitivity of mAP to errors.

The benchmark is therefore framed as a diagnostic instrument against shallow language understanding in retrieval models. The underlying paper explicitly states an intention to introduce “a new text-image retrieval benchmark that cannot be easily solved by VLMs exhibiting bag-of-words behaviour.” This places subCOCO within a line of work that distinguishes lexical matching from structured semantic matching, but it does so in a retrieval setting with thousands of candidate images rather than in small forced-choice tests.

A frequent misconception would be to treat subCOCO as merely “short COCO captions.” The benchmark definition does not support that simplification. Its queries are manually curated compositional expressions, its relevance labels are verified by two VLMs, and its evaluation protocol is multi-label ranking rather than caption-to-image Recall@$s(q, s) = \langle f(q), f(s) \rangle .$3.

5. Function within the LexiCLIP evaluation framework

subCOCO serves as one of the primary evaluation benchmarks for the LexiCLIP vision-free retrieval framework in the short-caption retrieval setting. In that pipeline, each COCO image is first converted into a textual description by a VLM, specified as InternVL-2.5-8B-MPO. The description comprises “Prompt A,” a detailed scene description, and “Prompt B,” a JSON-like object list with attributes. Retrieval then compares short textual queries directly against these textualized image representations using a single text encoder (Ntinou et al., 23 Sep 2025).

The paper uses subCOCO to evaluate both zero-shot and fine-tuned configurations. For subCOCO, the reported results are as follows: CLIP ViT-B/16 reaches mAP 33.6; CLIP ViT-L/14, 36.1; OpenCLIP ViT-G/14, 41.2; OpenCLIP ViT-BigG/14, 42.1; SigLIP ViT-B/16, 43.6; EVA-02-CLIP ViT-L-336, 41.6; RAM, 50.8; RAM++, 52.5; LexiCLIP (0.3B, zero-shot), 48.3; and LexiCLIP (0.3B, fine-tuned), 54.3. The corresponding best reported F1 values on subCOCO are also attained by LexiCLIP (0.3B, fine-tuned): F1@1 of 6.7, F1@5 of 18.5, and F1@10 of 27.5.

These results are interpreted in the paper as evidence that subCOCO is particularly sensitive to the modality-gap and compositionality issues that the authors attribute to dual-encoder VLMs. The paper’s broader argument is that CLIP- and SigLIP-style systems exhibit a modality gap and bag-of-words tendencies, whereas LexiCLIP removes the modality gap by converting images to text and using a single encoder for both sides of retrieval. subCOCO is treated as an important empirical test of that argument because it uses short, structurally constrained queries rather than long captions.

The paper also situates subCOCO within an ablation narrative. Object-based descriptions are reported as helpful for capturing the fine-grained object–attribute combinations that the benchmark targets. Sequence length beyond 256 tokens is said not to help much, and captioner variations are reported to have minimal effect. Although subCOCO-specific ablation tables are not separately provided, the benchmark is emphasized as one of the two central short-caption datasets in the study.

6. Release status, recommended usage, and terminological context

The released artifacts, including code, subFlickr, and subCOCO, are stated to be distributed under the MIT license, with the qualification that the derived benchmarks are released exclusively for research use and remain consistent with the original licensing and access conditions of COCO and Flickr30k. The benchmark is described as consisting of image IDs pointing to COCO images, a query list of 256 strings for subCOCO, and a binary relevance matrix or equivalent per-query lists of relevant image IDs. The associated repository is given as https://github.com/IoannaNti/LexiCLIP (Ntinou et al., 23 Sep 2025).

The recommended use is evaluation rather than training. The underlying work states that subCOCO is used only as a test set, not for training, and that LexiCLIP is trained instead on a separate synthetic corpus derived from OpenImages. Given the benchmark’s 256-query scale and 4,030-image test collection, it is characterized as suitable for benchmarking and ablation rather than for large-scale pretraining. This suggests that subCOCO’s main role is methodological diagnosis: it is a compact but targeted instrument for measuring short-caption retrieval and compositional generalization.

Within related work, subCOCO is positioned against standard retrieval benchmarks such as COCO and Flickr30k, long-text retrieval settings such as Urban1k, controlled compositional tests such as Winoground, SugarCrepe, and SugarCrepe++, and tag-based datasets such as Tag2Text, RAM, ADE20K, and OpenImages tag-based queries. Its distinctiveness lies in combining COCO-scale retrieval with short, phrase-level, internally structured queries and multi-image relevance.

The term “subCOCO” also admits a distinct usage within the COCO benchmarking ecosystem for black-box optimization. In the COCO framework, a suite is a collection of problems described by triples $s(q, s) = \langle f(q), f(s) \rangle .$4, and bbob-largescale is characterized as a “sub-suite / subCOCO” in the sense of being a sub-component parallel to bbob, bbob-noisy, and bbob-biobj, while remaining integrated into the same performance-assessment pipeline (Elhara et al., 2019). That usage refers to a large-scale extension of the bbob test suite rather than to the MS COCO-derived retrieval benchmark. The two senses are therefore homonymous rather than conceptually related.

Taken in its retrieval sense, subCOCO fills a specific gap: it is a COCO-derived, short, compositional retrieval benchmark whose evaluation protocol is designed for multi-label ranking and whose central research function is to probe whether retrieval systems can model attributes, actions, and relations under realistic query brevity.