AnyCapModel (ACM)

• AnyCapModel (ACM) is a modular, lightweight framework that refines multimodal captions using a residual correction mechanism and joint instruction–modality–caption encoding.
• It operates in a plug-and-play manner with frozen base models like GPT-4o, requiring minimal fine-tuning to improve fidelity and controllability across images, videos, and audio.
• ACM’s integration with a specialized dataset and decoupled evaluation protocol demonstrates up to 60% gains in content accuracy and notable style improvements on diverse benchmarks.

AnyCapModel (ACM) is a modular, lightweight framework designed to enhance the fine-grained controllability of existing foundation models for omni-modal captioning. ACM introduces a residual correction mechanism and a joint instruction–modality–caption encoding strategy, enabling control over caption content and style for images, videos, and audio without retraining the base model. Integrated with a comprehensive dataset (AnyCapDataset, ACD) and a decoupled evaluation protocol (AnyCapEval), ACM demonstrates substantial improvements in caption fidelity and controllability on both bespoke and public benchmarks (Ren et al., 17 Jul 2025).

1. System Architecture and Integration

AnyCapModel operates in a plug-and-play manner by augmenting a frozen base captioning model $\mathcal{M}_g$, such as GPT-4o, InternVL2.5, or Qwen2.5-VL. For each input $m$ (image, video, or audio), the base model produces an initial caption $y_0 = \mathcal{M}_g(m)$. ACM then refines this output, given $m$ and a user instruction $q$, via a separate refinement module $\mathcal{M}_a$:

$y_c = \mathcal{M}_a(m, q, y_0)$

The design decouples modality understanding, instruction following, and language generation, allowing ACM to reuse the base model’s strong language prior and generation capabilities. This residual correction approach requires only light fine-tuning of ACM-specific layers, with the base model parameters remaining unchanged.

The pipeline components are:

• Modality Encoder: Extracts fixed representations (e.g., InternViT/MViTv2 for image/video, EAT for audio) and projects to $e_m$ via a learned MLP.
• Instruction Encoder: Tokenizes and embeds $q$ into $e_q$ using the base LLM’s tokenizer.
• Caption Encoder: Embeds the initial caption $m$0, yielding $m$1.
• Fusion and LLM Decoder: Concatenates $m$2 and feeds into a new transformer-based LLM head to generate the refined caption $m$3.

2. Controllability Formalism

Control in ACM is achieved through the explicit integration of instruction, modality, and base caption embeddings:

• $m$4
• $m$5 (projection via MLP)
• $m$6
• $m$7

Fusion is performed by sequence-wise concatenation:

$m$8

Caption generation is then autoregressive:

$m$9

Inference applies standard greedy or beam decoding.

3. Module Specifications

3.1 Instruction Encoder

• Input: Natural-language instruction $y_0 = \mathcal{M}_g(m)$0 (e.g., "Describe only the background elements").
• Output: Embedded sequence $y_0 = \mathcal{M}_g(m)$1 aligned with the LLM trunk via shared tokenizer and embeddings.

3.2 Modality Feature Extractor

• Visual Inputs: InternViT or MViTv2 extract global representations, typically frame- or patch- pooled features.
• Audio Inputs: EAT transformer computes a fixed audio embedding.
• Only the MLP projection on top of frozen backbones is trained.

3.3 Fusion and Decoding

Fusion concatenates modality, instruction, and caption embeddings along the sequence dimension, which are then processed by the ACM decoder (a transformer LLM head). No additional special tokens are used; conditioning arises exclusively from joint state fusion. The full refinement pseudocode is:

$q$5

4. Dataset and Training Procedures

4.1 AnyCapDataset (ACD)

• $y_0 = \mathcal{M}_g(m)$2300,000 triplets $y_0 = \mathcal{M}_g(m)$3 for three modalities: 125k images, 100k videos, 75k audio.
• Each entry contains a user instruction $y_0 = \mathcal{M}_g(m)$4 (1 of 28 types), a high-quality caption $y_0 = \mathcal{M}_g(m)$5 (satisfying $y_0 = \mathcal{M}_g(m)$6), and a suboptimal caption $y_0 = \mathcal{M}_g(m)$7 (typically uncontrolled or mildly hallucinated).
• 28 instruction types:
  • Content Controls (9): e.g., Background, Event, Instance, Action, Appearance, Position, Movement, Perspective, Region.
  • Style Controls (5): e.g., Brief, Detail, Genre, Length, Theme.
  • Subtypes and combinations span all modalities.
• Data composition: 40% $y_0 = \mathcal{M}_g(m)$8 (teaching do-nothing decision), 30% $y_0 = \mathcal{M}_g(m)$9 (minor instruction violations), 30% $m$0 (hallucinations).

4.2 Training Protocol

• Loss: Standard cross-entropy over the target caption $m$1

$m$2

• Residual Correction: Including examples where the base caption is already correct ensures ACM learns to keep outputs unchanged when no refinement is needed.
• Hyperparameters: AdamW optimizer (LR $m$3, weight decay 0.01), cosine schedule with 3% warmup, 3 epochs, batch size 256 (bfloat16), all major backbones frozen, only the MLP projector and new LLM layers are trained.

5. Evaluation Metrics and Protocol

AnyCapEval decouples content accuracy from stylistic fidelity:

• Content Accuracy (Keypoint Density, KPD):
  • For test pairs $m$4, annotators exhaustively list required keypoints $m$5 for the reference $m$6 given instruction $m$7.
  • An automatic system (GPT-4o) matches how many target keypoints $m$8 appear in candidate $m$9.
  • KPD score: $q$0
• Style Fidelity (Discrete Score):
  • Human-in-the-loop (GPT-4o) rubric: 0 (severe deviation) to 4 (superior to reference).
  • Average style score across evaluation set.

These metrics allow substantive assessment of both controllability (instruction following) and generation quality.

6. Empirical Results and Comparisons

6.1 Internal Benchmarks (AnyCapEval)

ACM demonstrates marked improvements over base models in both content and style metrics.

Controllable Image Captioning (averages)

Model	Content	Style
GPT-4o	2.89	2.26
+ ACM-2B	4.11	2.46
+ ACM-8B	4.54	2.65

Controllable Video Captioning (averages)

Model	Content	Style
GPT-4o	3.55	2.15
+ ACM-2B	5.30	2.30
+ ACM-8B	5.74	2.32

Controllable Audio Captioning

Model	Content	Style
GPT-4o	1.59	1.18
+ ACM-2B	1.79	1.28
+ ACM-8B	1.88	1.30

Improvements reflect relative gains of up to 60% in content and 20% in style for ACM-8B over base GPT-4o.

6.2 Public Benchmarks

ACM enhances performance on major multi-instruction and multimodal captioning tasks:

• MIA-Bench (Image): GPT-4o: 89.1 → 90.3 (+1.0); InternVL2.5-8B: 80.9 → 85.4 (+4.5); Yi-VL-34B: 59.9 → 70.4 (+10.5).
• VidCapBench (Video): InternVL2.5-8B: accuracy 12.8 → 14.8, precision 51.6 → 57.1, conciseness 7.1 → 10.2.

Downstream text-to-video generation also improves, e.g., Wan2.1-T2V-14B: 2.74 → 3.10; HunyuanVideo: 2.91 → 3.50 (average metric; precise metric context provided in (Ren et al., 17 Jul 2025)).

6.3 Ablation and Comparisons

• ACM achieves +16.9% controllability gain on InternVL2.5-2B compared to SFT, DPO, and Self-Critic baselines, which yield only low single-digit improvements.
• Ablation on data composition reveals that including ~40% “already correct” $q$1 samples is critical for stability.
• The tuning of $q$2 vs. $q$3 vs. $q$4 ratios directly modulates model sensitivity to hallucination and minor correction.

7. Significance and Implications

AnyCapModel introduces a modular and broadly compatible methodology for controllable caption refinement, leveraging a “drop-in” design for existing large multimodal models. The decoupling of content and style evaluation with AnyCapEval, combined with explicit instruction encoding and residual refinement, establishes a new benchmark for fine-grained, preference-driven multimodal captioning. A plausible implication is that ACM’s architecture enables rapid adaptation and compositional control across diverse use cases and instruction types, while its minimal impact on base model architecture facilitates practical deployment at scale (Ren et al., 17 Jul 2025).