AnyCapEval: Benchmark for Omni-modal Captioning

• AnyCapEval is a benchmark that decouples content accuracy from stylistic fidelity to provide diagnostic evaluations for captioning systems across image, video, and audio modalities.
• It introduces a novel evaluation protocol based on Keypoint Density for content and a discrete rubric for style, enhancing reliability over traditional n-gram metrics.
• Empirical results show significant improvements in content and style scores using the AnyCapModel, validating its impact on controlled captioning performance.

AnyCapEval is a benchmark proposed as part of the AnyCap Project to address challenges in the evaluation of controllable omni-modal captioning systems. It introduces an evaluation protocol and metrics that decouple content accuracy from stylistic fidelity, enabling more reliable and diagnostic assessment of models that generate captions conditioned on both user instructions and multi-modal data, such as images, videos, and audio (Ren et al., 17 Jul 2025).

1. Motivation and Conceptual Framework

Controllable captioning requires satisfying two orthogonal objectives: content accuracy—ensuring the generated caption reflects the explicit facts or instructions provided by the user—and stylistic fidelity—ensuring the caption conforms to a specified linguistic style (e.g., brevity, narrative form, or specific genres). Previous evaluation protocols, often based on n-gram overlap metrics (such as BLEU or CIDEr), confound these two aspects. A single score may reward stylistic repetition at the expense of factual adherence, or vice versa. AnyCapEval explicitly decouples these axes, providing a content score ("did you mention the right facts?") and a style score ("did you express them fluently and in the desired manner?"). This orthogonal structure increases diagnostic power, reduces variance, and avoids the trade-off where optimizing for one metric harms the other (Ren et al., 17 Jul 2025).

2. Evaluation Metrics and Formalism

AnyCapEval reports two independent metrics:

2.1 Keypoint Density (KPD) for Content Accuracy

For a given reference caption $r$ and instruction $q$, a set $\mathcal{K}_{r,q} = \{k_1, k_2, \ldots, k_n\}$ of "key points" is extracted. Given a generated caption $\hat y$, an automatic matcher (GPT-4o-based) counts the number of correctly mentioned key points, $N_{\mathrm{match}}$. Let $\ell(\hat y)$ denote the word count of $\hat y$. The KPD metric is defined as:

$$\mathrm{KPD}(\hat y; q) = \frac{N_{\mathrm{match}}}{\ell(\hat y)} \times 100$$

The mean across $N$ examples yields the content accuracy:

$$\mathrm{Acc}_{\mathrm{content}} = \frac{1}{N} \sum_{i=1}^{N} \mathrm{KPD}(\hat y_i; q_i)$$

2.2 Discrete Rubric for Stylistic Fidelity

For style, GPT-4o is prompted with the instruction–reference–candidate triplet $q$0 and assigns a score $q$1, where 0 indicates severe deviation/hallucination, 4 indicates exceeding the reference without hallucination, and intermediate levels measure gradations of stylistic error. The average over all examples yields the style fidelity:

$q$2

This two-dimensional scoring yields low-variance, interpretable metrics and facilitates the identification of specific failure modes relating to content or style.

3. Dataset Composition and Protocol

AnyCapEval covers three modalities: image, video, and audio. It samples a balanced evaluation set for each modality with 500–1,000 instruction–reference pairs per instruction type, spanning approximately 8,000 examples across all modalities. There are 28 user instruction types across content (e.g., Background, Instance, Perspective, Event, Movement) and style (e.g., Brief, Detail, Genre/Poem, Narrative, Length, Theme) dimensions. For each test instance, a caption is generated, content keypoints are manually annotated, and GPT-4o is used for both automatic keypoint matching (content) and rubric-based style scoring. Final metrics are computed and reported for both axes (Ren et al., 17 Jul 2025).

4. Empirical Findings and Comparative Results

Integration of AnyCapModel (ACM) with base models demonstrates marked gains on AnyCapEval. For example, with image captioning, ACM-8B increases GPT-4o's content score from 2.89 to 4.54 and style from 2.26 to 2.65; similar gains are observed across InternVL2.5-8B and Qwen2.5-VL-7B. Video evaluations show +62% improvement in content (GPT-4o: 3.55→5.74) and +8% in style; audio evaluations reveal +18% in content and +10% in style.

Performance on external benchmarks further corroborates the diagnosis rooted in AnyCapEval: On MIA-Bench, ACM-8B improves GPT-4o's instruction-following accuracy from 89.1% to 90.3%; for InternVL2.5-8B, the increase is from 80.9% to 85.4%. On VidCapBench, there are measurable improvements in precision and conciseness, with content and style gains statistically significant ($q$3 for content, $q$4 for style), and confidence intervals of $q$5 (content) and $q$6 (style) (Ren et al., 17 Jul 2025).

Model	Content	Δ	Style	Δ
GPT-4o	2.89	—	2.26	—
+ ACM-2B	4.11	+1.22	2.46	+0.20
+ ACM-8B	4.54	+1.65	2.65	+0.39

A plausible implication is that the adoption of orthogonal content and style axes enables benefits beyond catch-all n-gram metrics, with direct impact on robustness and interpretability of controllable captioning evaluation.

5. Statistical and Diagnostic Properties

KPD exhibits a strong correlation with human relevance (Pearson $q$7), outperforming the raw keypoint count ($q$8). In ablation studies with the InternVL2.5-8B model on MIA-Bench, AnyCapModel (ACM) outperforms baseline fine-tuning objectives: SFT (+11.2%), DPO (+13.5%), Self-Critic (+14.1%), versus ACM (+21.1%).

All reported content gains are statistically significant at $q$9 (paired bootstrap, 1,000 resamples), and style at $\mathcal{K}_{r,q} = \{k_1, k_2, \ldots, k_n\}$0. The two-axis protocol improves error diagnosis by revealing the sub-axis (content or style) of failure, and each metric uses tailored procedures to reduce cross-axis trade-offs (Ren et al., 17 Jul 2025).

6. Implementation and Reproducibility

Evaluation code and prompts for both KPD extraction and style rubric are publicly available. ACM training uses AdamW optimizer, learning rate $\mathcal{K}_{r,q} = \{k_1, k_2, \ldots, k_n\}$1, with cosine annealing and 3% warmup, a batch size of 256, and three epochs distributed over 32 × A100 GPUs. Mixed precision (bfloat16) and PyTorch Distributed are used for scalable model adaptation. All backbone vision/audio encoders are frozen, and only MLPs and LLM adapter layers are fine-tuned, facilitating plug-and-play integration with a range of LLM/VLM architectures (Ren et al., 17 Jul 2025).

7. Impact and Advancement Over Prior Evaluation Protocols

AnyCapEval advances the evaluation of controlled captioning by resolving critical ambiguities inherent in previous metrics. Its separation of content and style produces metrics that are robust, low-variance, and diagnostic, thereby supporting more trustworthy development of omni-modal generative systems. The consistent empirical improvements seen with AnyCapModel across public datasets and base models demonstrate AnyCapEval's utility in both benchmarking and ablation studies, enabling direct quantification of progress in both factual alignment and stylistic adherence (Ren et al., 17 Jul 2025).