AnyCapDataset (ACD): Multimodal Captioning

• ACD is a unified, multimodal dataset that enables fine-grained, instruction-following caption generation across images, videos, and audio.
• It provides paired 'good' and 'bad' captions to support contrastive and preference-based learning through a rigorously validated annotation pipeline.
• ACD facilitates both training and benchmarking of controllable caption models with evaluation metrics such as Keypoint Density (KPD) and GPT-4o style scores.

AnyCapDataset (ACD) is a large-scale, meticulously validated corpus constructed to advance fine-grained, controllable, omni-modal captioning. Designed to address the limitations of earlier caption datasets—which are typically constrained to images or videos and lack the capacity for rich, natural-language control—the ACD enables precise model alignment with explicit user instructions across images, videos, and audio. It is a foundational component of the AnyCap Project’s unified approach to controllable captioning, serving both as a training resource and a benchmark for model evaluation and preference-based learning (Ren et al., 17 Jul 2025).

1. Purpose, Scope, and Motivation

The principal objective of ACD is to enable fine-grained, instruction-following caption generation for multimodal data. Existing datasets provide limited controllability and typically lack comprehensive, natural-language instruction sets, restricting their utility for training or benchmarking models that require nuanced or user-specific caption outputs. ACD is constructed to:

• Cover three modalities—images, video, and audio—in a single integrated corpus
• Support 28 distinct instruction types, spanning content and style, facilitating advanced controllability and instruction following
• Supply paired preference data ("good" vs. "bad" captions for the same instruction and media), suitable for contrastive or preference-based methods, such as RLHF and DPO
• Enable reproducible and scalable data generation by combining high-capacity open-source MLLMs and thorough human validation

This approach positions ACD as a paradigm for both general-purpose instruction tuning and targeted preference learning in multimodal captioning systems.

2. Modalities, Data Sources, and Statistical Composition

ACD encompasses three modalities, each with substantial scale:

Modality	Number of Triplets	Proportion (%)
Image	125,000	41.7
Video	100,000	33.3
Audio	75,000	25.0

Images are sourced from COCO, DCI, DOCCI, ShareGPT-4o, and Urbanek-Picture, videos from MSR-VTT, MSVD, ShareGPT-4o video, InstanceCap-video, and MiraData, and audio from AudioCaps, Clotho, WavCaps, and MACS. Each entry in ACD is a triplet (q, c, a) comprising a user instruction, a high-quality ("good") caption, and a suboptimal ("bad") caption for the same media and instruction, facilitating contrastive and preference-aligned learning.

The data schema for each record is JSON, with fields such as id, modality, source_dataset, instruction, caption_good, caption_bad, and optional metadata. Caption lengths average 62 words for "good" captions and 57 words for "bad" captions. Train/validation/test splits are not standardized in the release; users are advised to apply conventional 80/10/10 or custom splits (Ren et al., 17 Jul 2025).

3. Instruction Set and Control Signals

ACD instruction space is explicitly partitioned into 28 control dimensions encompassing content (9 types) and style (5 primary types, with further stylistic subdivisions). Examples of instruction categories include:

Content Controls:

• Background: “Provide a brief description of the scene background...”
• Event: “Mention any discrete event or temporal change...”
• Instance: “Focus only on the red car in the frame; ignore other objects.”
• Instance Action, Instance Appearance, Instance Position
• Movement: “Identify the camera movement (pan, tilt, zoom).”
• Perspective: “Describe the scene from a bird’s-eye view.”
• Region: “Restrict your caption to the upper half of the image.”

Style Controls:

• Brief: “Write a caption in no more than 10 words.”
• Detail: “Provide a highly detailed description, including colors and textures.”
• Genre (Poem, Narrative): “Compose the caption as a four-line poem.”
• Length: “Limit the caption to two sentences.”
• Theme: “Use a formal academic tone.”

This extensive taxonomy supports both common and highly specialized caption modifications, providing a testbed for models to demonstrate and be evaluated on nuanced controllability (Ren et al., 17 Jul 2025).

4. Annotation Pipeline and Quality Assurance

The ACD annotation pipeline operates as follows:

1. Sample selection: Raw media sampled from designated corpora.
2. Instruction and caption generation: User instruction and compliant caption generated via prompt-based queries to open-source MLLMs (primarily InternVL2.5-78B; GPT-4o mini for audio; proprietary models as needed).
3. Prompt validation: For each (modality, instruction) pair, ~20 outputs are manually validated. Batch generation proceeds only after 100% validation for adherence to instruction and factual correctness.
4. Generation of suboptimal captions: Controlled degradations (omitted details, minor factual errors, style violations) produced by instructing the MLLM to deviate from the optimal instruction-compliance.
5. Quality control: 5% of total generated triplets are randomly sampled for post-generation human review; >95% of these must meet the designated preference (i.e., "good" should be preferred over "bad").

The pipeline is frozen post-validation to guarantee scalability. Inter-annotator agreement is not explicitly quantified but quality assurance proceeds via consensus validation and spot checks (Ren et al., 17 Jul 2025).

5. Training Objectives, Benchmarking, and Evaluation Protocols

Preference-based frameworks are intrinsic to ACD's design: each triplet (q, c, a) supports loss formulations such as

$$L = – \log \sigma( s(c_i; m_i, q_i) – s(a_i; m_i, q_i) )$$

where $s(\cdot)$ is a model-derived compliance score. ACD underpins controllable caption generation (via fine-tuning plug-in modules or instruction tuning), preference-based training (RLHF, DPO, CPO, contrastive learning), and multi-modal instruction-tuned LMM development.

Evaluation Benchmarks:

• AnyCapEval: Separates content and style dimensions for evaluation.
  • Content: Measured by Keypoint Density (KPD), defined as

  $$\text{KPD}(\hat{y}; q) = \frac{N_{\text{match}}}{L_{\text{words}}(\hat{y})} \times 100$$

  where $N_{\text{match}}$ is the number of required keypoints found by a GPT-4o matcher, and $L_{\text{words}}(\hat{y})$ is the word length of the candidate caption. - Style: Graded on a 5-point scale (0–4) by GPT-4o, where 0 indicates severe deviation and 4 corresponds to outperforming the gold-standard style with no hallucinations.

Evaluation tools, data loaders, and scripts are available via GitHub, and the resource is distributed under a permissive license.

6. Access, Use Cases, and Ongoing Updates

ACD and its corresponding benchmark code are publicly accessible (https://github.com/qishisuren123/AnyCap). The primary use cases are:

• Training and evaluation of controllable caption models: Especially plug-in frameworks that do not require foundation model retraining.
• Preference-based alignment: Enabling learning protocols that exploit paired (c, a) data.
• Unified multi-modal instruction tuning: Supporting fine-tuning and benchmarking of large multimodal models (LMMs).
• Reliable benchmarking: AnyCapEval provides separate axes of evaluation, advancing methodical progress in controllable captioning.

The dataset is maintained with ongoing releases, including expanded instruction types and new data splits.

7. Context, Limitations, and Comparative Landscape

By offering richer coverage and granularity of control than prior caption datasets, ACD fills a critical gap for both model development and systematic benchmarking. Unlike region-labeled or multi-reference corpora, ACD's preference-pair design, broad control taxonomy, and generalized modal coverage enable more direct progress on explicit, user-driven captioning tasks.

Limitations include the absence of formal inter-annotator agreement metrics and the frozen nature of post-validation annotations (which, while supporting scale, might limit correction of residual errors). The pipeline's emphasis on manual prompt validation over per-instance correction reflects a pragmatic trade-off between fidelity and scalability (Ren et al., 17 Jul 2025).

This construct situates ACD as a foundational infrastructure for research in controllable omni-modal captioning, serving both as a rigorous training set and as a methodological benchmark for evaluating multidimensional controllability in caption generation models.