---
title: 'AnyCap Project: Omni-Modal Captioning'
url: https://www.emergentmind.com/topics/anycap-project
type: topic
---

# AnyCap Project: Omni-Modal Captioning

Searching arXiv for the AnyCap Project paper and nearby controllable captioning work to ground the article.
arXiv search query: "AnyCap Project controllable omni-modal captioning"
The AnyCap Project is a unified framework for controllable omni-modal captioning across images, videos, and audio. It integrates a model, a dataset, and an evaluation benchmark to address two stated deficiencies of existing captioning systems: limited fine-grained controllability and unreliable evaluation protocols. The project introduces AnyCapModel (ACM), a plug-and-play framework that works on top of frozen foundation models; AnyCapDataset (ACD), a controllable captioning dataset spanning three modalities and 28 instruction types; and AnyCapEval, an evaluation protocol that decouples content accuracy from stylistic fidelity [2507.12841].

## 1. Integrated formulation

Controllable captioning is presented as essential for precise multimodal alignment and instruction following. Within this formulation, the central problem is not only to generate a caption for an input modality, but also to ensure that the caption conforms to an explicit user instruction about what to describe and how to describe it. The AnyCap Project treats this as an integrated systems problem: model behavior, supervision data, and evaluation criteria are designed jointly rather than as isolated components [2507.12841].

| Component | Role | Key details |
|---|---|---|
| AnyCapModel (ACM) | Controllable caption refinement | Plug-and-play; works above frozen base models |
| AnyCapDataset (ACD) | Training data for controllable captioning | 3 modalities; 28 instruction types; 300,000 entries |
| AnyCapEval | Benchmark and scoring protocol | Decouples content accuracy and stylistic fidelity |

This organization is significant because it operationalizes controllability as a property that must be learned, evaluated, and stress-tested under matched conditions. A plausible implication is that the project is aimed not only at improving raw caption quality, but also at making captioning systems more diagnostically transparent under user-specified constraints.

## 2. AnyCapModel (ACM)

ACM is described as a lightweight plug-and-play framework that enhances the controllability of existing foundation models for omni-modal captioning without retraining the base model. The workflow begins with a base, frozen captioning model $\mathcal{M}_g$, which produces an initial caption $y_0$ from a modality input $m$. ACM, denoted $\mathcal{M}_a$, then refines this output using the modality input, the user instruction $q$, and the initial caption:
$$
y_c = \mathcal{M}_a(m, q, y_0).
$$
The model therefore reuses the original captions from base models while incorporating user instructions and modality features to generate improved captions [2507.12841].

Its architecture consists of modality-specific encoders, MLP projectors, a textual embedding layer, and a transformer-based language model. The modality encoders extract features from images, video frames, or audio; examples given are InternViT for images and videos and EAT for audio. The MLPs map each modality’s feature vector into a shared semantic space. The textual embedding layer tokenizes and embeds both the user instruction and the initial caption. The transformer-based language model concatenates these embeddings and autoregressively generates the refined, instruction-following caption.

Training is framed as residual correction rather than de novo caption generation. ACM is trained to edit the base caption by correcting misalignment with the user instruction or the input content. About 40% of training samples use already-correct captions, so ACM learns when no correction is needed. Fine-tuning uses the AdamW optimizer, 3 epochs, learning rate $10^{-6}$, batch size 256, and frozen modality backbones. The result is an architecture that can be combined with a wide variety of base models, including open-source and closed-source systems.

## 3. AnyCapDataset (ACD)

ACD is a controllable captioning dataset covering three modalities, 28 user-instruction types, and 300,000 annotated triplets derived from 75,000 unique media objects. The modality distribution is 125k image samples, 100k video samples, and 75k audio samples. Each entry is organized as a triplet $(q_i, c_i, a_i)$, where $q_i$ is an explicit user instruction, $c_i$ is a high-quality instruction-compliant caption, and $a_i$ is a suboptimal caption with minor deficiencies, inaccuracies, or instruction violations [2507.12841].

The dataset uses a preference-based design: annotators only need to indicate which of $(c, a)$ is better under the instruction, which reduces labeling overhead. This is paired with a data-generation pipeline that uses strong MLLMs, including InternVL2.5-8B and GPT-4o, to synthesize qualified instructions and compliant or noncompliant captions. Prompt engineering and manual validation are emphasized; all prompts pass 100% validation, and random sampling for quality control reports more than 95% agreement with annotators. Additional data augmentation generates suboptimal captions by dropping guidance or introducing controlled errors.

The 28 instruction types are divided into content controls and style controls. Content controls include Background, Event, Instance, Instance Action, Instance Appearance, Instance Position, Movement, Perspective, and Region. Style controls include Brief, Detail, Genre, Length, and Theme.

| Instruction | Applicable to | Description |
|---|---|---|
| Background | Video | Provide/suppress scene/background details |
| Instance | Image, Video | Emphasize/ignore specific entities |
| Movement | Video | Specify type of camera movement |
| Brief | All | Generate concise description |
| Narrative | All | Render as a narrative/poem |

Because the dataset spans image, video, and audio under a common instruction schema, it functions as a cross-modal supervision resource rather than a collection of modality-specific caption corpora. This suggests that the project treats controllability as a modality-agnostic property of caption generation, even though the encoders themselves remain modality-specific.

## 4. AnyCapEval

AnyCapEval is introduced to address the stated inadequacy of standard captioning metrics and undifferentiated LLM-based scoring. The motivating claims are explicit: BLEU and CIDEr fail to detect instruction compliance or semantic error, and LLM-based automatic scoring is inconsistent and style-biased. The benchmark therefore separates content evaluation from style evaluation and reports both dimensions per instruction type, modality, and model [2507.12841].

For content evaluation, AnyCapEval uses Keypoint Density (KPD). Given a tested caption $\hat{y}$ under instruction $q$, the protocol first annotates a set of required content key points $\mathcal{K}_{r,q} = \{k_1, k_2, ..., k_n\}$ for a reference caption $r$ and instruction $q$. A GPT-4o-based matcher then checks which keypoints are covered in $\hat{y}$. KPD is defined as matched keypoints per 100 words, thereby normalizing for verbosity. The stated rationale is that KPD tracks effective information rate and empirically correlates much better with human relevance judgments.

For style evaluation, GPT-4o compares $\hat{y}$ to the reference $r$ under instruction $q$ using a 0–4 discrete score. The rubric is: 0 for highly deviant, hallucinated, or incorrect; 1 for many hallucinations or off-instruction output; 2 for slight deviations or some hallucination; 3 for highly similar output with no hallucination; and 4 for output superior to the reference with no hallucination. The paper emphasizes that explicit, structured rubrics reduce subjective variability, and that fluency, structure, and style aspects are all considered.

The benchmark also includes MIA-Bench and VidCapBench for external validation. The resulting evaluation protocol rejects the idea that controllable captioning can be captured by a single aggregate quality score; instead, it operationalizes content accuracy and stylistic fidelity as orthogonal dimensions.

## 5. Experimental findings

On AnyCapEval, ACM markedly improves caption quality across a diverse set of base models. The headline claim is that ACM-8B raises GPT-4o’s content scores by 45% and style scores by 12%, and also achieves substantial gains on widely used benchmarks such as MIA-Bench and VidCapBench. The paper further reports modality-specific gains for GPT-4o: on images, content rises from 2.89 to 4.54 and style from 2.26 to 2.65; on videos, content rises from 3.55 to 5.74 and style from 2.15 to 2.32; on audio, content rises from 1.59 to 1.88 and style from 1.18 to 1.30 [2507.12841].

| Modality | Content: GPT-4o $\rightarrow$ +ACM-8B | Style: GPT-4o $\rightarrow$ +ACM-8B |
|---|---|---|
| Image | 2.89 $\rightarrow$ 4.54 | 2.26 $\rightarrow$ 2.65 |
| Video | 3.55 $\rightarrow$ 5.74 | 2.15 $\rightarrow$ 2.32 |
| Audio | 1.59 $\rightarrow$ 1.88 | 1.18 $\rightarrow$ 1.30 |

The reported trend is consistent: content gains outpace style gains. The interpretation supplied in the paper is that accurate control signals are easier to enforce than stylistic nuance. It is also stated that open-source base models, including InternVL2.5-8B and Qwen2.5-VL-7B, show similar or greater relative improvements, and that several open-source models with ACM-8B achieve controllability scores on AnyCapEval comparable to, or even superior to, strong unassisted proprietary baselines.

Public-benchmark results reinforce the same pattern. On MIA-Bench, GPT-4o improves from an average of 89.1 to 90.3 with ACM-8B, while InternVL2.5-8B improves from 80.9 to 85.4. On VidCapBench, InternVL2.5-8B improves in Accuracy from 12.8 to 14.8, Precision from 51.6 to 57.1, and Conciseness from 7.1 to 10.2; Qwen2-VL-7B improves in Accuracy from 13.1 to 15.4. For audio captioning, the paper reports modest gains in SPICE and SBERT metrics, including GPT-4o SBERT improving from 0.41 to 0.45 with ACM.

## 6. Ablations, downstream effects, and interpretive issues

The ablation results are organized around controllability, computational cost, and training-data composition. ACM is reported to perform comparably or better than SFT, DPO, and self-critic training on controllability, but with far less computational cost, since no retraining of the base model occurs. Varying data ratios and triplet construction, including controlled:correct:hallucinated configurations, shows robustness; the best gains occur when 40% of training examples are fully correct. Qualitative samples and human evaluation confirm increased instruction compliance, less hallucination, and better control granularity [2507.12841].

The paper also reports downstream effects. ACM-refined captions improve faithfulness and semantic alignment in downstream image and video generation tasks. This suggests that the project’s utility is not limited to caption generation itself; the refined captions can function as better-conditioned textual interfaces for subsequent multimodal systems.

Two recurrent misconceptions are directly challenged by the project’s design. The first is that standard captioning metrics are sufficient for controllable captioning. AnyCapEval is built on the opposite premise: instruction compliance, semantic coverage, and stylistic fidelity must be scored separately. The second is that controllability necessarily requires retraining the underlying foundation model. ACM instead operates as a frozen-base refinement layer. Within the terms of the paper, the project’s contribution is therefore less a single model than a matched triad of architecture, supervision, and evaluation for controllable omni-modal captioning.

Source: https://www.emergentmind.com/topics/anycap-project