---
title: AnyCapModel (ACM)
url: https://www.emergentmind.com/topics/anycapmodel-acm
type: topic
---

# AnyCapModel (ACM)

AnyCapModel (ACM) is a modular, lightweight framework designed to enhance the fine-grained controllability of existing foundation models for omni-modal captioning. ACM introduces a residual correction mechanism and a joint instruction–modality–caption encoding strategy, enabling control over caption content and style for images, videos, and audio without retraining the base model. Integrated with a comprehensive dataset (AnyCapDataset, ACD) and a decoupled evaluation protocol (AnyCapEval), ACM demonstrates substantial improvements in caption fidelity and controllability on both bespoke and public benchmarks [2507.12841].

## 1. System Architecture and Integration

AnyCapModel operates in a plug-and-play manner by augmenting a frozen base captioning model $\mathcal{M}_g$, such as GPT-4o, InternVL2.5, or Qwen2.5-VL. For each input $m$ (image, video, or audio), the base model produces an initial caption $y_0 = \mathcal{M}_g(m)$. ACM then refines this output, given $m$ and a user instruction $q$, via a separate refinement module $\mathcal{M}_a$:

$$
y_c = \mathcal{M}_a(m, q, y_0)
$$

The design decouples modality understanding, instruction following, and language generation, allowing ACM to reuse the base model’s strong language prior and generation capabilities. This residual correction approach requires only light fine-tuning of ACM-specific layers, with the base model parameters remaining unchanged.

The pipeline components are:  
- **Modality Encoder:** Extracts fixed representations (e.g., InternViT/MViTv2 for image/video, EAT for audio) and projects to $e_m$ via a learned MLP.
- **Instruction Encoder:** Tokenizes and embeds $q$ into $e_q$ using the base LLM’s tokenizer.
- **Caption Encoder:** Embeds the initial caption $y_0$, yielding $e_{y_0}$.
- **Fusion and LLM Decoder:** Concatenates $e_m, e_q, e_{y_0}$ and feeds into a new transformer-based LLM head to generate the refined caption $y_c$.

## 2. Controllability Formalism

Control in ACM is achieved through the explicit integration of instruction, modality, and base caption embeddings:

- $f_m = \mathrm{ModEnc}(m) \in \mathbb{R}^{d_m}$  
- $e_m = W_m f_m + b_m \in \mathbb{R}^{d_t}$ (projection via MLP)  
- $e_q = \mathrm{TextEmb}(q) \in \mathbb{R}^{d_t \times T_q}$  
- $e_{y_0} = \mathrm{TextEmb}(y_0) \in \mathbb{R}^{d_t \times T_0}$

Fusion is performed by sequence-wise concatenation:
$$
E_{\rm fuse} = [e_m\,; e_q\,; e_{y_0}] \in \mathbb{R}^{d_t \times (1 + T_q + T_0)}
$$

Caption generation is then autoregressive:
$$
P(y_c|m, q, y_0) = \prod_{t=1}^{L_c} P(y_c^{(t)} | y_c^{(<t)},\, E_{\rm fuse})
$$

Inference applies standard greedy or beam decoding.

## 3. Module Specifications

### 3.1 Instruction Encoder

- Input: Natural-language instruction $q$ (e.g., "Describe only the background elements").
- Output: Embedded sequence $e_q$ aligned with the LLM trunk via shared tokenizer and embeddings.

### 3.2 Modality Feature Extractor

- Visual Inputs: InternViT or MViTv2 extract global representations, typically frame- or patch- pooled features.
- Audio Inputs: EAT transformer computes a fixed audio embedding.
- Only the MLP projection on top of frozen backbones is trained.

### 3.3 Fusion and Decoding

Fusion concatenates modality, instruction, and caption embeddings along the sequence dimension, which are then processed by the ACM decoder (a transformer LLM head). No additional special tokens are used; conditioning arises exclusively from joint state fusion. The full refinement pseudocode is:

```python
def refineCaption(m, q, y0):
    f_m = ModalityEncoder(m)
    e_m = LinearProj(f_m)
    e_q = LLM.TokenEmbed(q)
    e_y0 = LLM.TokenEmbed(y0)
    E_fuse = concat([e_m], e_q, e_y0)
    y_c = LLM.decode(E_fuse)
    return y_c
```

## 4. Dataset and Training Procedures

### 4.1 AnyCapDataset (ACD)

- $\sim$300,000 triplets $(q_i, c_i, a_i)$ for three modalities: 125k images, 100k videos, 75k audio.
- Each entry contains a user instruction $q_i$ (1 of 28 types), a high-quality caption $c_i$ (satisfying $q_i$), and a suboptimal caption $a_i$ (typically uncontrolled or mildly hallucinated).
- 28 instruction types:  
  - **Content Controls (9):** e.g., Background, Event, Instance, Action, Appearance, Position, Movement, Perspective, Region.
  - **Style Controls (5):** e.g., Brief, Detail, Genre, Length, Theme.
  - Subtypes and combinations span all modalities.
- Data composition: 40% $(q, c, c)$ (teaching do-nothing decision), 30% $(q, a, c)$ (minor instruction violations), 30% $(q, \hat{a}, c)$ (hallucinations).

### 4.2 Training Protocol

- Loss: Standard cross-entropy over the target caption $c_i$
  $$
  \mathcal{L} = -\sum_{t=1}^{|c_i|} \log P(c_i^{(t)} | c_i^{(<t)},\, m_i, q_i, a_i)
  $$
- Residual Correction: Including examples where the base caption is already correct ensures ACM learns to keep outputs unchanged when no refinement is needed.
- Hyperparameters: AdamW optimizer (LR $1\times 10^{-6}$, weight decay 0.01), cosine schedule with 3% warmup, 3 epochs, batch size 256 (bfloat16), all major backbones frozen, only the MLP projector and new LLM layers are trained.

## 5. Evaluation Metrics and Protocol

AnyCapEval decouples content accuracy from stylistic fidelity:

- **Content Accuracy (Keypoint Density, KPD):**
  - For test pairs $(q,r)$, annotators exhaustively list required keypoints $\mathcal{K}_{r,q}$ for the reference $r$ given instruction $q$.
  - An automatic system (GPT-4o) matches how many target keypoints $N_{\rm match}$ appear in candidate $\hat{y}$.
  - KPD score: $\mathrm{KPD}(\hat y;q) = \frac{N_{\rm match}}{L_{\rm words}(\hat y)} \times 100$
- **Style Fidelity (Discrete Score):**
  - Human-in-the-loop (GPT-4o) rubric: 0 (severe deviation) to 4 (superior to reference).
  - Average style score across evaluation set.

These metrics allow substantive assessment of both controllability (instruction following) and generation quality.

## 6. Empirical Results and Comparisons

### 6.1 Internal Benchmarks (AnyCapEval)

ACM demonstrates marked improvements over base models in both content and style metrics.

**Controllable Image Captioning (averages)**

| Model              | Content | Style  |
|--------------------|---------|--------|
| GPT-4o             | 2.89    | 2.26   |
| + ACM-2B           | 4.11    | 2.46   |
| + ACM-8B           | 4.54    | 2.65   |

**Controllable Video Captioning (averages)**

| Model              | Content | Style  |
|--------------------|---------|--------|
| GPT-4o             | 3.55    | 2.15   |
| + ACM-2B           | 5.30    | 2.30   |
| + ACM-8B           | 5.74    | 2.32   |

**Controllable Audio Captioning**

| Model              | Content | Style  |
|--------------------|---------|--------|
| GPT-4o             | 1.59    | 1.18   |
| + ACM-2B           | 1.79    | 1.28   |
| + ACM-8B           | 1.88    | 1.30   |

Improvements reflect relative gains of up to 60% in content and 20% in style for ACM-8B over base GPT-4o.

### 6.2 Public Benchmarks

ACM enhances performance on major multi-instruction and multimodal captioning tasks:

- **MIA-Bench (Image):** GPT-4o: 89.1 → 90.3 (+1.0); InternVL2.5-8B: 80.9 → 85.4 (+4.5); Yi-VL-34B: 59.9 → 70.4 (+10.5).
- **VidCapBench (Video):** InternVL2.5-8B: accuracy 12.8 → 14.8, precision 51.6 → 57.1, conciseness 7.1 → 10.2.

Downstream text-to-video generation also improves, e.g., Wan2.1-T2V-14B: 2.74 → 3.10; HunyuanVideo: 2.91 → 3.50 (average metric; precise metric context provided in [2507.12841]).

### 6.3 Ablation and Comparisons

- ACM achieves +16.9% controllability gain on InternVL2.5-2B compared to SFT, DPO, and Self-Critic baselines, which yield only low single-digit improvements.
- Ablation on data composition reveals that including ~40% “already correct” $(q,c,c)$ samples is critical for stability.
- The tuning of $(q,a,c)$ vs. $(q,c,c)$ vs. $(q,\hat a,c)$ ratios directly modulates model sensitivity to hallucination and minor correction.

## 7. Significance and Implications

AnyCapModel introduces a modular and broadly compatible methodology for controllable caption refinement, leveraging a “drop-in” design for existing large multimodal models. The decoupling of content and style evaluation with AnyCapEval, combined with explicit instruction encoding and residual refinement, establishes a new benchmark for fine-grained, preference-driven multimodal captioning. A plausible implication is that ACM’s architecture enables rapid adaptation and compositional control across diverse use cases and instruction types, while its minimal impact on base model architecture facilitates practical deployment at scale [2507.12841].

Source: https://www.emergentmind.com/topics/anycapmodel-acm