UCM-Caption: Controllable Omni-Modal Captioning

• UCM-Caption is a unified framework that refines frozen base captions by integrating user instructions with image, video, and audio inputs.
• The system leverages a lightweight alignment module and a preference-style dataset with suboptimal captions to enhance controllability in caption generation.
• Its evaluation protocol decouples content fidelity from stylistic adherence, resulting in significant performance gains over baseline models.

Searching arXiv for UCM-Caption / AnyCap and closely related controllable omni-modal captioning work. arxiv_search(query="AnyCap Project unified controllable omni-modal captioning", max_results=10, sort_by="relevance") arxiv_search(query="UCM-Caption controllable omni-modal captioning AnyCap", max_results=10, sort_by="relevance") UCM-Caption denotes a unified Controllable Omni-Modal Captioning framework that is also presented as the AnyCap Project. In this formulation, captioning is treated not only as multimodal description generation but as instruction-conditioned alignment: a modality sample, a natural-language control instruction, and an initial caption produced by a frozen base caption model are combined to produce a refined caption that strictly follows the instruction. The framework is organized around three coupled elements—AnyCapModel (ACM), AnyCapDataset (ACD), and AnyCapEval—and is intended to improve controllability across image, video, and audio captioning without retraining the base model (Ren et al., 17 Jul 2025).

1. Conceptual scope and system decomposition

UCM-Caption is defined around the joint problem of controllability, modality coverage, and evaluation reliability. Its formal input is a modality sample $m\in\{m^{\rm img},m^{\rm vid},m^{\rm aud}\}$, a natural-language control instruction $q$, and an initial caption $y_0$ produced by a frozen base caption model $\mathcal{M}_g$; its output is a refined caption $y_c$ that strictly follows $q$ (Ren et al., 17 Jul 2025).

The framework is decomposed into a lightweight alignment module, a controllable omni-modal dataset, and an evaluation protocol that explicitly separates content fidelity from stylistic adherence. This division is central to the project’s design: model training, data construction, and metric design are treated as a single research object rather than as independent components. A plausible implication is that UCM-Caption addresses a failure mode common in captioning research, namely that improvements in instruction following are difficult to verify when training data and evaluation criteria do not encode the same control variables.

Component	Formal role	Key contents
AnyCapModel (ACM)	Alignment module	Refines frozen base captions under user instructions
AnyCapDataset (ACD)	Training data	Three modalities, 28 instruction types, $\approx 300\,k$ triplets
AnyCapEval	Evaluation protocol	Decouples content accuracy and stylistic fidelity

A common misconception is to treat UCM-Caption as a conventional end-to-end captioner. The primary description instead presents it as a plug-and-play alignment module that reuses the original captions from base models while incorporating user instructions and modality features to generate improved captions (Ren et al., 17 Jul 2025).

2. Alignment architecture and residual correction

The core ACM equation is

$y_c \;=\;\mathcal{M}_a\bigl(m,\;q,\;y_0\bigr),$

where $\mathcal{M}_a$ is the lightweight alignment module being trained (Ren et al., 17 Jul 2025).

Feature extraction is modality-specific. For image and video, the system uses an InternViT encoder to produce a feature tensor $F^{\rm img/vid}$; for audio, it uses an EAT transformer to produce a feature tensor $q$0. These features are projected into a shared space by a small MLP $q$1:

$q$2

The instruction and the initial caption are embedded as

$q$3

They are then concatenated with the modality representation,

$q$4

and fed into an autoregressive LLM head to produce $q$5 (Ren et al., 17 Jul 2025).

Training follows a residual-correction strategy. The stated objective is to teach $q$6 to “fix” errors or omissions in $q$7 with respect to the gold caption $q$8, using autoregressive cross-entropy on target tokens of $q$9. The training mixture includes $y_0$0 of “already correct” $y_0$1 pairs so the model learns when no correction is needed (Ren et al., 17 Jul 2025). This suggests that ACM is optimized not merely for aggressive rewriting, but for selective intervention conditioned on whether the base caption already satisfies the instruction.

3. Dataset design and controllability schema

AnyCapDataset is specified as a preference-style dataset

$y_0$2

where $y_0$3 strictly follows $y_0$4, and $y_0$5 is a deliberately suboptimal caption, for example by omitting details or adding hallucinations (Ren et al., 17 Jul 2025).

The modality allocation is approximately 300 k triplets distributed as 125 k image samples, 100 k video samples, and 75 k audio samples. Instruction types total 28. Nine are content controls: Background (Bkg), Event (Evt), Instance (Ins), Instance Action (IAct), Instance Appearance (IApp), Instance Position (IPos), Movement (Mov), Perspective (Per), and Region (Reg). Nineteen are style controls, with examples including Brief (Brf), Detail (Det), Genre (Gen), Length (Len), Theme (Thm), Poem (Poe), and Narrative (Nar), alongside other stylistic forms such as humorous and formal (Ren et al., 17 Jul 2025).

The curation process is explicitly staged. Each instruction type has a human-written prompt with examples; prototype validation samples 20 generations per prompt and requires 100% manual pass; bulk generation uses InternVL2.5, GPT-4o mini, and small GPT-4o subsets; suboptimal captions are either generated with degradation prompts or copied from the compliant caption with injected noise; and a 5% random post-review reports more than 95% compliance with annotation standards (Ren et al., 17 Jul 2025).

This design matters because controllable captioning requires more than paired inputs and outputs. By storing both compliant and deliberately suboptimal captions relative to the same instruction, ACD operationalizes controllability as a preference relation over captions rather than as a single target string.

4. Evaluation formalism: decoupling content and style

AnyCapEval is built around the claim that content accuracy and stylistic fidelity should be scored separately. For content, the benchmark defines Key-Point Density (KPD):

$y_0$6

where $y_0$7 is the set of human-annotated key points required by $y_0$8, $y_0$9 is how many appear in $\mathcal{M}_g$0, and $\mathcal{M}_g$1 normalizes for length (Ren et al., 17 Jul 2025).

The reported analysis states that KPD has Pearson $\mathcal{M}_g$2 versus human relevance, compared with $\mathcal{M}_g$3 for raw keypoint count. Stylistic fidelity is scored by GPT-4o with a discrete rubric $\mathcal{M}_g$4, where 0 denotes severe hallucination, 1 major flaws, 2 slight hallucinations, 3 matches reference, and 4 exceeds reference. The prompts used for this score explicitly specify length constraints of $\mathcal{M}_g$5, tone match, and hallucination checks (Ren et al., 17 Jul 2025).

The benchmark is contrasted with earlier metric families. BLEU and CIDEr are characterized as n-gram-overlap measures with poor semantic behavior and no instruction check; unconstrained LLM scoring is characterized by high variance and prompt sensitivity; and keypoint extraction F1 is described as unstable because point extraction is unstable. The stated conclusion is that AnyCapEval yields lower variance and higher content-style diagnostic power (Ren et al., 17 Jul 2025).

A common misunderstanding is that controllable captioning can be evaluated adequately by a single scalar score. The AnyCapEval construction rejects that assumption and treats instruction compliance as a two-axis problem.

5. Quantitative results and ablation structure

On image controllability under AnyCapEval, GPT-4o reports Content (KPD) 2.89 and Style 2.26, while $\mathcal{M}_g$6 ACM-2B reports 4.11 $\mathcal{M}_g$7 and 2.46 $\mathcal{M}_g$8, and $\mathcal{M}_g$9 ACM-8B reports 4.54 $y_c$0 and 2.65 $y_c$1. The abstract separately reports that ACM-8B raises GPT-4o content scores by 45% and style scores by 12% (Ren et al., 17 Jul 2025).

On video results under AnyCapEval, GPT-4o reports Content 3.55 and Style 2.15, whereas $y_c$2 ACM-8B reports 5.74 $y_c$3 and 2.32 $y_c$4. On MIA-Bench, GPT-4o improves from 89.1 average accuracy to 90.3 $y_c$5 with ACM-8B, and InternVL2.5-8B improves from 80.9 to 85.4 $y_c$6. On VidCapBench, InternVL2.5-8B improves from 12.8 to 14.8 $y_c$7 in Accuracy and from 7.1 to 10.2 $y_c$8 in Conciseness (Ren et al., 17 Jul 2025).

The ablation program is unusually explicit about training composition. For image and video, a balanced mix

$y_c$9

yields the best content $q$0 and style $q$1 for ACM-8B, whereas over-reliance on perfect captions $q$2 leads to overfitting and style degradation. In objective comparisons on InternVL2.5-2B over MIA-Bench, SFT yields $q$3, DPO yields $q$4, Self-Critic yields $q$5, and ACM yields $q$6 average gain versus base. ACM-8B is also reported to outperform ACM-2B by an additional 5–10% in content and 2–5% in style (Ren et al., 17 Jul 2025).

These results establish UCM-Caption primarily as an alignment layer for controllability gains over strong frozen captioning backbones, rather than as a standalone modality encoder whose gains derive from larger pretraining alone.

6. Position within omni-modal captioning research

Within recent captioning literature, UCM-Caption occupies a distinct position. OmniCaptioner is presented as a unified captioning framework built around “pixel-to-word” mapping, using a Vision Transformer encoder, a linear merger layer, prompt-steered fusion, and a frozen or lightly tuned LLM to caption natural images, posters, UIs, documents, tables, charts, and geometric-math diagrams (Lu et al., 9 Apr 2025). By contrast, UCM-Caption takes as input a modality sample, a control instruction, and an initial caption from a frozen base caption model, then learns a lightweight correction module (Ren et al., 17 Jul 2025). This suggests that OmniCaptioner emphasizes direct visual-to-text generation across heterogeneous image domains, whereas UCM-Caption emphasizes controllable post-alignment across image, video, and audio.

A second comparison arises with UGC-VideoCaptioner, which is centered on omnimodal user-generated video, a benchmark of 1 000 TikTok clips and 3 975 QA pairs, and a 3B model distilled from Gemini 2.5-Flash with supervised fine-tuning followed by Group Relative Policy Optimization (Wu et al., 15 Jul 2025). Its architecture uses explicit audio and video feature extractors, cross-modal fusion, and a language decoder for detailed audiovisual captions. UCM-Caption shares the broader omni-modal ambition, but its modular interface is different: it refines captions generated by another model rather than replacing the captioning model with a new audiovisual encoder-decoder (Ren et al., 17 Jul 2025).

The nomenclature around UCM-Caption is not entirely stable across contemporary texts. One comparative description characterizes “the hypothetical UCM-Caption approach” as a multi-decoder alternative that is not present in open literature (Lu et al., 9 Apr 2025), whereas (Ren et al., 17 Jul 2025) explicitly describes “the ‘UCM-Caption’ system” as the unified Controllable Omni-Modal Captioning framework, also called the AnyCap Project. The most defensible reading is that “UCM-Caption” functions as a name for the AnyCap framework in (Ren et al., 17 Jul 2025), while other usage reflects comparative shorthand rather than a separate canonical model.

In the current literature, the distinctive contribution of UCM-Caption is therefore not merely omni-modal coverage, but the combination of instruction-conditioned residual correction, a preference-style controllable dataset, and an evaluation protocol that separates content fidelity from style adherence (Ren et al., 17 Jul 2025).