HoneyPipe: Modular Data Curation Pipeline

• HoneyPipe is a transparent and modular data curation pipeline that integrates rigorous cleaning, deduplication, and rule-based filtering to produce high-quality multimodal datasets.
• HoneyPipe employs a dual-level Chain-of-Thought enrichment strategy, generating both short and long reasoning chains to boost performance in tasks such as VQA and document understanding.
• HoneyPipe supports reproducible, community-driven updates through the DataStudio framework and demonstrates state-of-the-art results on benchmarks like CountBench with its Honey-Data-15M dataset.

HoneyPipe is a transparent and modular data curation pipeline designed to address the persistent limitations of open-source multimodal LLMs (MLLMs), especially those arising from suboptimal data quality in supervised fine-tuning (SFT). Developed as the core of the Bee project, HoneyPipe orchestrates the generation of Honey-Data-15M, a large-scale, chain-of-thought-enriched dataset for MLLM training. Built atop the DataStudio framework, HoneyPipe integrates rigorous cleaning, filtering, and reasoning enrichment steps to yield a dataset that supports advanced reasoning and instruction-following performance on par with or exceeding other fully open and recent semi-open models (Zhang et al., 15 Oct 2025).

1. Architecture and Conceptual Foundations

HoneyPipe is instantiated within the DataStudio framework, which is designed for systematic, reproducible, and extensible dataset curation. Its modular pipeline consists of:

• Data aggregation from diverse public sources with image–text pairs.
• Multi-stage cleaning via deduplication, rule-based filters, and model-based content assessments.
• Dual-level Chain-of-Thought (CoT) enrichment to promote both concise and extensive reasoning, leveraging state-of-the-art multimodal open models as annotators.
• Output in the form of Honey-Data-15M, used to successfully train Bee-8B and related models.

Unlike previous dataset releases, HoneyPipe is structured for continuous, community-driven updates, emphasizing pipeline transparency and evaluation at each phase.

2. Data Aggregation and Cleaning Methodologies

The HoneyPipe pipeline employs a sequence of cleaning techniques to address noise, irrelevance, and lack of alignment in raw data:

• Deduplication: Perceptual and semantic hashes identify and remove duplicate image–text pairs, ensuring sample diversity.
• Rule-Based Filtering: Heuristics filter malformed records, such as missing instructions, incomplete responses, or low-quality/abnormally proportioned images.
• Model-Based Assessment: A LLM operator acts as a judge, verifying that instructions correspond to visual content and eliminating mismatched or ambiguous cases.

The net result is a substantially cleaner corpus with improved structural integrity and direct task alignment relative to most prior open MLLM datasets.

3. Dual-Level Chain-of-Thought Enrichment Strategy

A defining feature of HoneyPipe is its Chain-of-Thought (CoT) enrichment process, applied at two distinct levels:

• Short CoT: For standard instructions, template discouraging statements (such as “answer directly”) are removed, and intermediate reasoning steps are generated using robust open-source MLLMs (e.g., Qwen2.5-VL); this produces roughly 12.2M samples.
• Long CoT Loop: For complex, ambiguous, or fidelity-flagged samples, a second, more elaborate reasoning chain is introduced. Marked-up tags (e.g., “> …”) delineate intermediate steps, supporting multi-step solution paths (~2.7M samples).

The pipeline's enrichment logic is informed by validation loops that identify insufficient reasoning, thereby triggering long CoT generation. Ablation studies demonstrate that this two-tier enrichment is critical for improving not just output diversity but actual task performance, particularly in advanced VQA and document/chart understanding settings.

4. Training Stages and Optimization Protocols

The Bee-8B model is trained on Honey-Data-15M via a five-stage process:

1. MLP Warmup: Training only the MLP visual-linguistic projector on 1M curated examples to bootstrap alignment.
2. Vision-Language Alignment: Joint training of all components over ~12.6M mixed pairs (plus additional text-only data) for robust multimodal feature acquisition.
3. SFT on Honey-Data-15M: Supervised fine-tuning across the full, CoT-enriched dataset to enhance complex reasoning and instruction following.
4. Efficient Refinement SFT: Further training on a quota-based 1M high-quality subset, emphasizing both longest and randomly sampled responses for representational diversity.
5. Reinforcement Learning (GRPO): Policy optimization with a reward function (format:accuracy at 0.2:0.8) to penalize repetition and formatting errors while enforcing factual correctness:

$$\text{Reward} = 0.2\cdot R_{\text{format}} + 0.8\cdot R_{\text{accuracy}}$$

Comprehensive benchmarks show progressive gains at each stage, culminating in state-of-the-art or competitive scores on CountBench, MathVista-mini, and other reasoning-focused evaluations.

5. Evaluation and Empirical Performance

The HoneyPipe pipeline's effectiveness is substantiated through extensive ablation studies and benchmark evaluations. For instance, on CountBench, Bee-8B trained with full HoneyPipe curation achieves a top score of 93.0. Detailed tables show that successive pipeline and enrichment refinements directly correlate with improvements in VQA, mathematical problem solving, and instruction following. Notably, long CoT-enriched models outperform those trained on short CoT or baseline, with pronounced gains on tasks requiring multi-step logical deductions.

6. Transparency, Impact, and Community Contributions

HoneyPipe, together with DataStudio, provides the community not just with a static dataset (Honey-Data-15M) but with the full-stack suite for its transparent curation and evaluation, including:

• Source code for the pipeline and DataStudio framework
• Explicit training recipes and evaluation harnesses
• Model weights for direct benchmarking and fine-tuning

This principled, modular, and reproducible approach is explicitly framed as the key to closing the performance gap with proprietary and semi-open MLLMs. By foregrounding data quality, reasoning depth, and curation transparency, HoneyPipe establishes a new standard for open-source dataset construction and MLLM pretraining (Zhang et al., 15 Oct 2025).