---
title: 'ReCap-DataComp-1B v2: Enhanced Vision-Language Dataset'
url: https://www.emergentmind.com/topics/recap-datacomp-1b-v2
type: topic
---

# ReCap-DataComp-1B v2: Enhanced Vision-Language Dataset

ReCap-DataComp-1B v2 is a large-scale, recaptioned image–text dataset designed to advance training and evaluation of vision-language models, emphasizing high semantic alignment and textual richness in image descriptions. Derived from the DataComp-1B dataset—which itself is a filtered 1.3 billion image–text pair subset of a 12.8B web crawl—ReCap-DataComp-1B v2 replaces noisy, terse web-sourced captions with detailed, LLaMA-3-powered language model captions, intended for broad distribution and open-source research in multimodal learning.

## 1. Recaptioning Methodology and Model Architecture

The central component of ReCap-DataComp-1B v2 is its recaptioning pipeline, which enhances each image–text pair by generating new, semantically richer captions. The foundation is a vision-language architecture comprising:

- **Frozen Vision Backbone**: CLIP ViT-L/14 for robust visual feature extraction.
- **Projection MLP**: Trained to map vision encoder outputs to the embedding space of the language model.
- **Language Model**: LLaMA-3–8B, selected for its strong open-source language modeling and reasoning capabilities, replacing the default LLM in LLaVA-1.5.
- **Two-Stage Fine-Tuning**:
  1. **Stage 1**: The projection MLP is trained using approximately 558K filtered image–text pairs from public datasets (e.g., LAION, CC3M, SBU).
  2. **Stage 2**: Both the projection MLP and LLaMA-3–8B language decoder are fine-tuned using an additional 665K instruction-following (i.e., conversational or reasoning) pairs plus high-quality samples from datasets like HQ-Edit.

Caption generation relies on autoregressive decoding (greedy search, maximum 128 tokens) following prompts such as:  
“Please generate a detailed caption of this image. Please be as descriptive as possible.”

The process uniformly recaptions all 1.3B image–text pairs, yielding the Recap-DataComp-1B dataset with substantially longer and more informative descriptions compared to the originals [2406.08478].

## 2. Dataset Characterization and Enhancements

Several distinct enhancements in ReCap-DataComp-1B v2 contrast with previous web-crawled datasets:

- **Caption Length and Detail**: Average caption length increases from 10.22 words (web-crawled) to 49.43 words (recaptioned), integrating a broader spectrum of attributes, object descriptors, and contextual features.
- **Lexical Richness**: Recaptioned captions account for 82.86% of the collective token vocabulary compared to the original corpus, indicating both expansion and diversification of semantic content.
- **Semantic Alignment**: Assessed with a LongCLIP-Large model, the image–caption similarity score escalates to 89.91 for recaptioned text versus 10.09 for original captions.
- **Human Evaluation**: A GPT-4V audit over 10,000 samples reports average fluency and alignment ratings rising from 3.71 (original) to 4.14 (recaptioned).

Figure examples illustrate the qualitative difference: recaptioned samples accurately describe foreground/background relations, object colors, contextual cues, and actions, surpassing the original captions’ brevity or misalignment [2406.08478].

## 3. Effects on Vision-Language Model Training

### Discriminative Models (e.g., CLIP)
- Models trained on Recap-DataComp-1B v2 (“Recap-CLIP”) exhibit consistent improvement on zero-shot cross-modal retrieval.  
- A mixed training regime, stochastically blending original and recaptioned captions with proportion $p$, is formalized as:
  $$
  \text{Caption} = 
  \begin{cases}
    \text{Original} & \text{w.p. } p \\
    \text{Recaption} & \text{w.p. } 1-p
  \end{cases}
  $$
  Empirically, $p=0.8$ (80% recaptioned) yields a 5% lift in standard retrieval benchmarks compared to models trained exclusively on originals.

- Increased text encoder size further augments long-context retrieval (Urban1K) and detailed attribute recognition (VG-Attribution), with score improvements sometimes in the 19–36% range [2406.08478].

### Generative Models (Text-to-Image Diffusion Transformers)
- Training DiT models on the recaptioned data yields lower FID (Fréchet Inception Distance), with reductions such as FID by 8.4 when comparing pure recaption use to pure original captions.
- CLIP and Recap-CLIP-based image-text scores both increase ($+3.1$–$+8.4\%$), and GPT-4V ratings on prompt-image alignment improve by 1.1 points, facilitating superior compliance with intricate or compositional text queries.

## 4. Optimization Strategies and Technical Specifications

The recaptioning and model training process involves several technical decisions tailored to the demands of large-scale, high-fidelity data creation:

- **Training Parameters**: Recap-CLIP models are trained using AdamW with large batch sizes (tens of thousands) and a two-phase procedure: initial low-resolution image training followed by higher-resolution fine-tuning.
- **Tokenization**: The tokenizer and text encoder accommodate up to 128 tokens per caption to handle the increased caption length from recaptioning.
- **Instruction-Following Supervision**: The fine-tuning corpus for the captioner leverages a substantial share of instruction-following examples, assisting the model to produce contextually appropriate, detailed, and semantically grounded descriptions.

## 5. Comparisons with Prior Data Filtering and Caption Modification Approaches

Earlier DataComp work focused on boosting data quality through model-based filtering and synthetic captioning methods:

| Approach                                       | Core Model(s) Used         | Principal Data Enhancement                  | Downstream Impact              |
|------------------------------------------------|----------------------------|---------------------------------------------|-------------------------------|
| CLIP/BLIP-2 Filtering [2310.14581]             | CLIP ViT-L/14, BLIP-2-COCO | Filter by image–text similarity; modify captions using BLIP-2 | 6.6%–48.5% gain vs. baselines |
| ReCap-DataComp-1B v2 [2406.08478]              | LLaVA-1.5, LLaMA-3–8B      | Re-caption 1.3B pairs with detailed, context-rich text         | +5% retrieval, +1.1 GPT-4V    |

* BLIP-2–COCO (fine-tuned on MSCOCO) emerged as optimal for similarity-based filtering in earlier DataComp challenges [2310.14581].
* Caption modification in prior work leveraged BLIP-2/Flan-T5-XL, selecting between generated and original captions via CLIP similarity.
* The current LLaMA-3–powered recaptioning surpasses prior synthetic captions in length, fluency, and alignment, yielding greater improvements in downstream metrics.

## 6. Implications for Future Foundation Model Development

The open-source release of Recap-DataComp-1B v2 is poised to substantially affect the development and evaluation of large-scale vision-language models:

- **Data Quality**: High-quality textual annotations are expected to become a key competitive differentiator for both discriminative and generative multimodal systems, with direct influence on retrieval, classification, and image generation tasks.
- **Open-Source Ecosystem**: Availability of a GPT-4–level, richly annotated dataset provides a public alternative to proprietary corpora maintained by industry actors.
- **Research Directions**: Promising future research includes further refinement of recaptioning pipelines, new ratios or blending strategies for original and generated captions, improved long-text fusion schemes, and the examination of even larger or more specialized vision-language models.
- **Application Scope**: Enhanced training data supports improved performance not only in conventional retrieval or generation settings but also enables new directions in visually grounded reasoning and interactive multimodal agents [2406.08478].

## 7. Relationship to DataComp-LM and Broader Data Curation Research

The principle of dataset-centric performance improvement is echoed in parallel work on language modeling datasets such as DataComp-LM [2406.11794]:

- Both in vision-language and language-only domains, aggressive data deduplication, quality filtering (e.g., model-based fastText classifiers), and careful data mixing are crucial to maximizing downstream accuracy per unit of compute.
- For Recap-DataComp-1B v2, the blending of high-quality generated captions parallels findings that introducing or curating annotation quality leads to measurable performance gains even with fixed model architectures and training recipes.

A plausible implication is that continuous advances in high-capacity, open-source captioners and large-scale synthetic annotation will increasingly underpin future progress in multimodal learning.

Source: https://www.emergentmind.com/topics/recap-datacomp-1b-v2