BLIP3o-60k Instruction Tuning Dataset

• The BLIP3o-60k dataset is a high-quality, open-source collection of 60k prompt–image pairs designed to improve multimodal image generation and prompt adherence.
• It integrates synthetic and real data by generating ~10k pairs per category via GPT-4o along with an additional ~20k pairs from JourneyDB and DALL·E 3–style sources.
• Its application in supervised fine-tuning boosts GenEval and visual aesthetics scores by 5–10 points, addressing gaps in under-represented visual scenarios.

The BLIP3o-60k instruction-tuning dataset is a high-quality, open-source collection developed as part of the BLIP3-o family of unified multimodal models, targeting deficiencies observed during image generation pretraining. It is explicitly curated to enhance multimodal models' ability to generate images in under-represented scenarios, supporting fine-tuning for improved prompt alignment and visual fidelity within unified architectures leveraging CLIP feature spaces and diffusion-based generation heads (Chen et al., 14 May 2025).

1. Rationale and Purpose

BLIP3o-60k was constructed to address gaps in pretraining distribution after stage 2, particularly with respect to complex human gestures, representations of common objects, landmarks, and simple in-image text. These categories were consistently under-represented in the large-scale pretraining data (55M+ images), potentially limiting the generative and alignment capabilities of the BLIP3-o models. The dataset is applied as an instruction-tuning (IT) stage via supervised fine-tuning (SFT) to reliably improve downstream generation metrics and address coverage in challenging prompt classes.

2. Construction Methodology

The dataset's design integrates both synthetic and selected real prompts, employing the following steps:

• For each of the four targeted categories (complex human gestures, common objects, landmarks, in-image text), GPT-4o was programmatically prompted to generate approximately 10,000 high-quality prompt–image pairs.
• An additional ∼20,000 prompts were sourced from JourneyDB and DALL·E 3–style datasets, emphasizing diversity and stylistic coverage.
• The aggregated set produces approximately 60,000 prompt–image pairs, systematically spanning a breadth of scenes, objects, human-centric activities, and textual content embedded in images.

3. Automatic and Manual Quality Assurance

To ensure data integrity, diversity, and utility:

• Automated heuristics filter out prompts identified as low-diversity, malformed, or likely to produce template repetition.
• Approximately 5% of the data underwent manual spot-checking to verify the absence of hallucinated content or high-frequency templates from GPT-4o.
• This curation pipeline is designed to maximize prompt–image fidelity and to minimize artifacts that could bias the instruction-tuning phase.

4. Integration into BLIP3-o Training and Empirical Impact

The BLIP3o-60k set is employed for simple SFT on top of existing BLIP3-o models, following stage 2 of the sequential pretraining recipe. The direct application of instruction tuning with this dataset yields empirical performance boosts:

• Observed improvements in both GenEval and visual aesthetics scores by +5–10 points, as reported in Table 7 of the reference work.
• This suggests that instruction-tuning on BLIP3o-60k specifically augments prompt fidelity and sample plausibility in otherwise under-served prompt categories.

5. Relationship to Other Data Releases and Reproducibility

BLIP3o-60k complements two other foundational data assets in the BLIP3-o suite:

• The pretraining dataset: consisting of approximately 25 million open images and captions from CC12M, SA-1B, and JourneyDB, used for initial backbone and understanding/generation-stage training.
• All datasets, including BLIP3o-60k, are released openly under an Apache 2.0 license and available through Hugging Face at https://huggingface.co/datasets/BLIP3o/BLIP3o-60k. Full details and scripts can be accessed via the official GitHub repository (github.com/JiuhaiChen/BLIP3o).

6. Dataset Summary Table

Aspect	Details	Source
Size	≈60,000 prompt–image pairs	(Chen et al., 14 May 2025)
Main Categories	Complex human gestures, common objects, landmarks, in-image text	(Chen et al., 14 May 2025)
Prompt generation method	GPT-4o (∼10k/category) + ∼20k JourneyDB/DALL·E 3–style prompts	(Chen et al., 14 May 2025)
Filtering/QA	Automatic heuristics + manual spot checks (5% sample)	(Chen et al., 14 May 2025)
Licensing	Apache 2.0 (research/commercial use)	(Chen et al., 14 May 2025)
Access	huggingface.co/datasets/BLIP3o/BLIP3o-60k	(Chen et al., 14 May 2025)

7. Role in Unified Multimodal Modeling

BLIP3o-60k is instrumental in advancing prompt adherence and image diversity within the BLIP3-o architecture, which unifies image understanding and generation via a CLIP-embedding space and a lightweight DiT diffusion head. The dataset operationalizes instruction-tuning in an otherwise unified, modality-agnostic model pipeline, directly contributing to state-of-the-art performance across image generation and understanding benchmarks, as evidenced in GenEval, DPG-Bench, and MME evaluations (Chen et al., 14 May 2025). The openly-available BLIP3o-60k set thereby serves both as a benchmark for future multimodal research and an actionable resource for the broader research community.