ECCV Caption: Enhanced ITM Benchmark

• The paper introduces ECCV Caption, which expands valid image–caption associations up to 8.5× compared to MS-COCO by combining machine and human verification.
• It presents mAP@R, a ranking-based metric that offers a more fine-grained evaluation and better alignment with human judgment than traditional recall-based metrics.
• Multiple ITM models are used in the annotation pipeline to mitigate bias, ensuring a comprehensive, fair assessment of vision-and-language models.

ECCV Caption refers to the Extended COCO Validation Caption dataset and accompanying evaluation protocol designed to address the abundance of false negatives in the standard MS-COCO Image–Text Matching (ITM) benchmark. By leveraging both machine and human verification to identify valid image–caption associations that were absent in the original dataset, ECCV Caption provides a substantially enriched resource for the training and evaluation of Vision-and-Language (VL) models. A new ranking-based metric, mAP@R, is also introduced, replacing the conventional recall-based metrics and yielding more reliable model rankings and greater alignment with human judgment (Chun et al., 2022).

1. Problem Formulation and Motivation

Standard ITM benchmarks such as MS-COCO Caption pair each image with a fixed, small set of captions (five in the case of COCO) and associate each caption with only a single image. All other image–caption pairs are treated as negatives. This disregards the fact that visually similar images may be validly described by the same caption and that many captions generalize across multiple images. For instance, the caption “A man that is standing up and has a tennis racquet” can accurately describe several distinct images of tennis players. The original benchmarks therefore manifest massive false negatives: unannotated, yet semantically valid image–caption pairs treated incorrectly as non-matches (Chun et al., 2022).

Quantitative analysis on COCO reveals that, in the original val split, the average number of ground-truth captions per image ($N_{i \rightarrow c}$) is 5 and the average number of valid images per caption ($N_{c \rightarrow i}$) is 1. Through extended verification, these rise to approximately 17.9 (+3.58×) and 8.5 (+8.47×), respectively.

2. ECCV Caption Dataset Construction

2.1 Machine-in-the-Loop Verification

ECCV Caption employs a machine-in-the-loop annotation process to efficiently harvest plausible matches:

• For each of 1,261 image and 1,332 caption queries (sampled from COCO test), the top-5 cross-modal retrievals are collected from each of five diverse ITM models:
  1. CLIP (ViT-B/32, contrastive pre-trained on 400M pairs)
  2. ViLT (Vision-Language Transformer)
  3. VSRN (region features, semantic reasoning)
  4. PVSE (probabilistic many-to-many)
  5. PCME (probabilistic caption–image embedding)

• Redundant retrievals are deduplicated, yielding approximately 46,424 candidate image–caption pairs (Chun et al., 2022).

2.2 Human Verification and CxC Merge

Candidate pairs are then evaluated through crowd-sourced human verification (2,160 HITs, 970 workers) with four graded response categories. After filtering and qualification checks, 21,995 pairs are accepted as positives. These are further merged with positives from the CrissCrossed Caption (CxC) extension, which targets intra-text associations, resulting in the ECCV Caption dataset.

Table 1. Positive Matches in COCO-val Subset

Dataset	# Positive Images	# Positive Captions
Original COCO	1,332	6,305
CxC	1,895 (×1.42)	8,906 (×1.41)
Human-verified	10,814 (×8.12)	16,990 (×2.69)
ECCV Caption (final)	11,279 (×8.47)	22,550 (×3.58)

Extensive annotation results in 3.6× more caption→image and 8.5× more image→caption associations compared to the original COCO.

3. Evaluation Metrics: mAP@R Versus Recall@K

Traditional evaluation for ITM relies on Recall@K (R@K), which evaluates whether ground-truth matches appear in the top K model predictions. However, R@K is insensitive to the ranking among true positives and does not consider the actual number of valid associations per query.

ECCV Caption introduces mAP@R, a ranking-based metric defined as follows. For queries $q \in Q$ (images or captions) and $R_q$ verified positives:

• For ranked gallery items $k$:
  • $\mathrm{rel}_q(k) = 1$ if the $k$-th item is a verified positive, 0 otherwise
  • $$P_q(k) = \frac{1}{k} \sum_{i=1}^k \mathrm{rel}_q(i)$$
• The metric:

$$\mathrm{mAP@R} = \frac{1}{|Q|} \sum_{q \in Q} \left[ \frac{1}{R_q} \sum_{k=1}^{R_q} P_q(k) \cdot \mathrm{rel}_q(k) \right]$$

mAP@R is sensitive to the full ranking among all positives and better correlates with human assessments, as confirmed by a user study of 3,200 pairwise comparisons (Chun et al., 2022).

4. Re-Evaluation of Vision-and-LLMs

Using ECCV Caption and mAP@R, 25 VL models were reassessed, including:

• Classical VSE models: VSE0, VSE++, PVSE, PCME
• Region-based: VSRN, CVSE, SGR, SAF, VSE∞
• Pre-trained transformers: CLIP (ViT-B/32, B/16, L/14), ViLT, VinVL, BLIP
• PVSE variants with different negative mining strategies

Conventional recall-based metrics (COCO 1K/5K R@1/5/10, CxC R@1) exhibit high mutual rank correlation (Kendall's $\tau > 0.87$). In contrast, rankings under ECCV mAP@R shift notably; for example, CLIP ViT-L/14's COCO 1K R@1 = 72.17 but mAP@R = 27.98, while PCME (CutMix) yields 62.71 and 41.74, respectively. "Hardest" negative mining improves recall but not mAP@R, whereas semi-hard negative mining is superior under mAP@R. The plausible-match R-Precision (PMRP) does not correlate strongly with mAP@R (τ ≈ 0.20–0.45) (Chun et al., 2022).

5. Annotation Bias and Mitigation

The MITL (Machine-in-the-Loop) annotation protocol introduces bias if candidate associations are drawn from a single ITM model, favoring that model in subsequent evaluation (self-bias). Defining

$N_{c \rightarrow i}$0

where $N_{c \rightarrow i}$1 denotes the candidate model set, $N_{c \rightarrow i}$2 declines as more proposal models are included. Using all five retrieval models yields minimal bias, as evidenced by experiments: single-model proposals accentuate performance for the source model (PVSE best at 76.5% Recall@1 on its own proposals, but others score 42–68%), while using all proposals gives a fairer evaluation spread. Bias diminishes by half when increasing from one to three proposal models (Chun et al., 2022).

6. Contributions, Implications, and Recommendations

ECCV Caption demonstrates the prevalence and detrimental impact of false negatives in MS-COCO Caption, establishes a corrected benchmark by machine/human verification, and advocates a more discriminative evaluation metric (mAP@R) for ITM. These developments reveal shifts in model rankings and clarify the influence of negative mining strategies. Employing multiple proposal models during annotation is essential to minimize dataset-induced bias.

Implications for VL research include:

• Adoption of ECCV Caption and mAP@R for rigorous, human-aligned model evaluation.
• Encouraging the use of diverse model pools in machine-in-the-loop annotation to reduce bias.
• Reconsideration of conventional ranking metrics and benchmark protocols in favor of richer, more accurate association datasets (Chun et al., 2022).