---
title: 'ReasonPro-16M: Structured Reasoning Dataset'
url: https://www.emergentmind.com/topics/reasonpro-16m
type: topic
---

# ReasonPro-16M: Structured Reasoning Dataset

Searching arXiv for the cited paper to ground the article.
ReasonPro-16M is a large-scale training corpus introduced in "ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP" as the second, more structured supervision dataset in the paper’s two-stage continual pretraining framework for CLIP-style encoders. It consists of category-specific commonsense reasoning captions paired with images, and it is used in Stage 2 to impose explicit category-level reasoning supervision after a first stage based on more open-form reasoning captions. In the paper’s formulation, ReasonPro-16M is intended not merely as additional caption data, but as a controlled supervision mechanism that organizes visually grounded reasoning patterns inside the joint image-text representation space [2606.26794].

## 1. Position within the ReasonCLIP framework

ReasonPro-16M occupies the second stage of the ReasonCLIP-58M recipe. The framework introduces reasoning progressively rather than in a single undifferentiated training step. Stage 1 uses ReasonLite-42M, which provides open-form, visually verifiable reasoning captions and mixes those captions with regenerated descriptive captions in order to preserve descriptive alignment while increasing reasoning awareness. Stage 2 then switches to ReasonPro-16M, whose supervision is explicitly category-specific and no longer uses the descriptive captions \(T_b\).

The paper defines ReasonPro as a dataset of “category-specific commonsense reasoning captions” that provides “more structured and controllable supervision than the open-form generation in ReasonLite.” The motivation is that alignment alone does not explicitly encourage discriminability across different reasoning categories. ReasonPro therefore targets predefined visual reasoning types and supports fine-grained per-category analysis aligned for CLIP.

This design also links directly to the evaluation protocol. RCLIP-Bench uses the same five reasoning categories for its V2 and V3 levels, so ReasonPro supplies the training taxonomy while RCLIP-Bench supplies category-aware diagnostic evaluation. A plausible implication is that ReasonPro-16M is intended less as an open-ended reasoning corpus than as a structured supervision layer for category separability and analyzable reasoning behavior [2606.26794].

## 2. Dataset composition and reasoning taxonomy

ReasonPro-16M contains **5,521,563 valid images** after filtering, each paired with **3 category-specific reasoning captions**, for a total of **16,564,689 image-caption pairs**. The main text summarizes this as **“5.5M images and 16.6M image-caption pairs spanning up to five reasoning categories.”** The source images come from **CC12M-Refined**, specifically the **5.7M images reserved from CC12M-Refined** after separating the subset used for ReasonLite.

Its supervision is intrinsically one-to-many and multi-view. Each image is first assigned a multi-label set of supported reasoning categories, after which three categories are selected and three corresponding captions are generated. One image may therefore support several valid reasoning types, but each caption is tied to one specific category. The dataset’s core ontology is the fixed five-way taxonomy
\[
\{S, A, C, T, P\}.
\]

The five categories are defined as follows. **Spatial / Geometric (S)** covers spatial relations, direction, distance, occlusion, containment, accessibility, and geometry. **Attribute / State (A)** covers appearance, texture, wetness, openness, deformation, brightness, on/off state, and related visible states. **Creature / Action (C)** covers human or animal posture, gestures, interaction, and current or imminent behavior. **Temporal / Phase (T)** covers whether an event just happened, is ongoing, or is about to happen. **Intuitive Physics (P)** covers visually inferable stability, support, contact, force, and physical interaction; the appendix characterizes it as reasoning about “relatively obvious physical relationships that can be inferred directly from visual cues,” and “not relying on semantic common sense, but on visual physical intuition.”

The released category counts are imbalanced: **Spatial/Geometric 4,766,178**, **Attribute/State 4,883,348**, **Creature/Action 2,469,372**, **Temporal/Phase 947,656**, and **Physical Intuition 3,498,135**. Caption statistics are also reported: average length **29.4 words** and variance **18.2**. The comparatively small Temporal/Phase count is a direct indication that the dataset emphasizes high-volume automatic generation with quality control rather than strict class balancing [2606.26794].

## 3. Construction pipeline and quality control

ReasonPro-16M is built from **CC12M** through a multistage recaptioning and annotation process. The paper first collects **10,388,539 images** from CC12M and constructs **CC12M-Refined** by generating **3 regenerated descriptive captions \(T_b\)** per image using **Qwen2.5-VL-72B**, yielding **31,165,584 image-\(T_b\) pairs**. From this refined pool, **4.7M images** are assigned to ReasonLite and **5.7M images** are reserved for ReasonPro.

All ReasonPro annotations are then produced by **Qwen3-VL-32B**, which the paper says was chosen because it offered “stronger category discrimination and more structured outputs than the model used for ReasonLite.” The first step is multi-label category annotation. Given the predefined category definitions, the model predicts
\[
Y \subseteq \{S, A, C, T, P\}
\]
for each image, where \(Y\) is the set of visually supported reasoning types. Only images satisfying
\[
|Y| \ge 3
\]
are retained. The appendix reports a start from **5,720,000 samples**, removal of **198,437** samples, and retention of **5,521,563 valid samples**. The main text reports a filtering rate of **3.97% (0.20M)** due to “inconsistent or low-confidence labels,” while the appendix reports **3.47%**; both point to low-single-digit filtering.

The second step is category-conditioned caption generation. For each retained image, the pipeline randomly selects **three reasoning categories** \(Y' \subseteq Y\), and feeds the model the image, its base caption \(T_b\), the selected categories \(Y'\), and the category definitions in a single generation request. The model then generates one category-specific reasoning caption for each selected category. These captions are denoted \(T_{rp}\), so each image contributes three pairs:
\[
(x, T_{rp}^{(1)}), \quad (x, T_{rp}^{(2)}), \quad (x, T_{rp}^{(3)}).
\]

Quality control combines manual validation and automatic filtering. Before large-scale generation, **500 samples per stage** were reviewed by **five graduate students**, prompts were iteratively refined, and generation proceeded only after pass rate exceeded **99.5%**. For ReasonPro specifically, the initial prompt had ambiguous category boundaries; after reinforcing strict and mutually exclusive definitions, pass rate reached **99.8%**. After full generation, random inspection gave **99.2%** pass rate across stages. Automatic filtering includes non-English outputs, outputs shorter than **10 tokens**, outputs longer than **60 tokens**, and repetition or degeneracy loops; the pipeline also uses structural and grounding constraints and rule-based cleanup to remove malformed outputs. The paper does not explicitly document a separate deduplication pass for ReasonPro-16M beyond these screening steps [2606.26794].

## 4. Supervision format and Stage 2 objective

ReasonPro-16M uses caption-form supervision rather than QA, instructions, or class names. Each training instance includes an image \(x\), a category-specific reasoning caption \(T_{rp}\), an image-level multi-label category set \(Y\), and a caption-level single category label \(c\). The corresponding structured targets are an image-side multi-hot vector
\[
\mathbf{y} \in \{0,1\}^5
\]
and a text-side single label
\[
c \in \{1,\dots,5\}.
\]

The paper characterizes the resulting training regime as image-text alignment augmented with category classification. The Stage 2 objective is
\[
\mathcal{L}^{(2)}=\mathcal{L}_{\text{align}(x,T_{rp})}+\gamma\bigl(\mathcal{L}_{\text{img-cls}(x,Y)}+\mathcal{L}_{\text{txt-cls}(T_{rp},c)}\bigr),
\]
with auxiliary losses
\[
\mathcal{L}_{\text{img-cls}}=\mathrm{BCE}\bigl(g_v(x), \mathbf{y}\bigr), \qquad
\mathcal{L}_{\text{txt-cls}}=\mathrm{CE}\bigl(g_t(T_{rp}), c\bigr).
\]
For the alignment term, **CLIP** uses symmetric **InfoNCE**, while **SigLIP** uses sigmoid-based binary cross-entropy alignment.

ReasonPro-16M is used only in Stage 2, and the paper is explicit that descriptive captions \(T_b\) are no longer used at that point. Descriptive alignment is instead preserved in Stage 1 through a separate objective,
\[
\mathcal{L}^{(1)}
=
\lambda(t)\,\mathcal{L}_{\text{align}(x, T_b)}
+
(1-\lambda(t))\,\mathcal{L}_{\text{align}(x, T_{rl})}
+
\beta \|\theta - \theta_0\|_2^2,
\]
which mixes descriptive and reasoning captions while regularizing deviation from the initial model. Stage 2 therefore specializes a model that has already been made reasoning-aware without fully abandoning descriptive alignment.

The category-discrimination branch added for Stage 2 uses lightweight heads. The method section describes them as “lightweight MLP-based classification heads,” while the appendix gives the implementation as **Text classifier: \(\mathrm{Linear}(D, 5)\)** and **Image classifier: \(\mathrm{Linear}(D, 5)\)** with **L2-normalized text/image embeddings** as input, bias enabled, and **bfloat16** dtype. These heads are used only during training and are discarded at inference. Stage 2 is run for **1 epoch** on ReasonPro-16M, with **per-device batch size 512**, **gradient accumulation 2**, **effective batch size 32,768**, **AdamW**, **classifier learning rate \(1.5 \times 10^{-3}\)**, **weight decay 0.10**, **warm-up ratio 0.10**, and **32 A100 64GB** GPUs. The vision/text/logit learning rates are \(1\times10^{-5} / 2\times10^{-5} / 1\times10^{-4}\) for SigLIP and \(1\times10^{-5} / 3\times10^{-5} / 1\times10^{-4}\) for CLIP; \(\gamma\) is **0.05** for SigLIP and **0.10** for CLIP; and there is **no L2 regularization in Stage 2** [2606.26794].

## 5. Empirical effects attributable to ReasonPro-16M

The cleanest empirical view of ReasonPro-16M comes from Stage 2 comparisons. On **RCLIP-V3-5K** retrieval, Stage 2 often improves substantially over the base backbone. For **CLIP ViT-B/32**, base CLIP reports **I→T R@1 51.0** and **T→I R@1 26.7**, while Stage 2 reports **55.3** and **33.8**. For **CLIP ViT-L/14 @224**, the corresponding values move from **54.0 / 31.7** to **67.1 / 41.9**. For **CLIP ViT-L/14 @336**, they move from **55.9 / 33.2** to **68.0 / 42.8**. For **SigLIP So400m**, they move from **70.5 / 43.4** to **72.4 / 49.5**. These results indicate that ReasonPro-based Stage 2 generally strengthens reasoning-caption retrieval.

The appendix also reports RCLIP-Bench behavior. For **CLIP-L/14 @336**, base **V1/V2/V3 avg** is **22.9 / 17.7 / 25.0**, while Stage 2 gives **29.5 / 18.0 / 23.6**. For **SigLIP So400m @384**, base is **24.4 / 22.2 / 21.8**, while Stage 2 gives **30.7 / 24.1 / 22.8**. The paper’s own interpretation is that explicit category supervision is not uniformly beneficial on every reasoning dimension for every backbone: V1 improves strongly, V2 changes modestly, and V3 does not always exceed the base model for lower-capacity CLIP variants.

ReasonPro-16M is also tied to the paper’s compositional reasoning gains, because the main compositional table reports Stage 2 variants. For **ReasonCLIP B/32 (+58M, which includes ReasonPro)** versus base **CLIP B/32**, the reported scores are **WhatsUp 51.3 vs 41.2**, **VALSE 72.5 vs 67.5**, **CREPE 24.0 vs 23.9**, **SugarCREPE 75.6 vs 73.1**, **SugarCrepe++ TOT 61.5 vs 46.7**, and **Avg 57.7 vs 52.1**. The corresponding averages are **57.9 vs 51.4** for **ReasonCLIP L/14 @224**, **58.1 vs 51.8** for **ReasonCLIP L/14 @336**, and **62.7 vs 56.6** for **ReasonSigLIP So/14**. The paper also reports that **ReasonCLIP + READ** exceeds **READ** alone in the B/32 case, with average **65.7 vs 64.1**.

At the same time, the ablations make the trade-off explicit. For **ReasonSigLIP**, Stage 2 gives the most balanced overall performance, especially retrieval. For some compositional and RCLIP-Bench settings, Stage 1 can be better. For **ReasonCLIP**, Stage 1 improvements are more pronounced and Stage 2 gains are relatively limited; the paper states that excessive explicit supervision in lower-capacity CLIP backbones may weaken alignment. It also reports that the classification heads help somewhat on V2 and V3 but often only marginally, and that replacing the default image-side multi-label classification with single-label classification hurts both retrieval and reasoning metrics. This suggests that ReasonPro-16M’s main empirical contribution is strongest when the backbone can absorb explicit reasoning-category supervision without substantially disrupting the underlying alignment space [2606.26794].

## 6. Limitations, caveats, and reproducibility

ReasonPro-16M is intentionally narrow in reasoning scope. The appendix states that it excludes **false causality**, **over-extension**, **hallucination**, and **complex multi-step reasoning beyond what is visually supported**. It therefore does not target open-ended chain-of-thought or long-horizon reasoning; instead, it focuses on moderately difficult, directly image-grounded inference suitable for CLIP-style contrastive learning.

The dataset also has several explicit caveats. Category frequencies are imbalanced, with **Temporal/Phase** far smaller than **Spatial/Geometric** or **Attribute/State**. All annotations are model-generated with **Qwen3-VL-32B**, so residual noise may remain despite the manual pilot validation and filtering pipeline. The paper notes that early prompt iterations suffered from ambiguous boundaries between categories, especially between **Spatial/Geometric** and **Intuitive Physics**. No explicit category-balancing sampler is reported for Stage 2, and no separate deduplication pass is documented for ReasonPro-16M. The full **58M** corpus is also much smaller than industrial billion-scale CLIP pretraining, which limits the extent to which the reported gains can be generalized.

A further caveat is architectural and empirical. Stage 2 can weaken alignment, especially for lower-capacity CLIP backbones; some standard zero-shot classification results drop slightly after reasoning-oriented pretraining, and the appendix interprets this as a shift from object-centric to relation/context-centric representations. The paper therefore positions ReasonPro not as a universally dominant supervision source, but as a structured specialization stage whose benefits are clearest on reasoning retrieval and compositional benchmarks, particularly in higher-capacity or SigLIP-family models.

From a reproducibility standpoint, the dataset is specified precisely: **5,521,563 valid images**, **16,564,689 image-caption pairs**, **3 \(T_{rp}\) captions per image**, image-side multi-label \(Y\), and text-side single label \(c\). The paper does not describe train/validation/test splits because ReasonPro-16M is a training corpus used for **one epoch** of Stage 2 continual pretraining. Release links are provided through the project GitHub and Hugging Face collections. The appendix states that the work is based on **CC12M** under its permissive terms, that images are accessed only via URLs, and that the authors do **not redistribute images**. Reported annotation cost is **~280 GPU hours** for category annotation and **~960 GPU hours** for \(T_{rp}\) annotation, both with **Qwen3-VL-32B**. Taken together, these details characterize ReasonPro-16M as a controlled, category-aware supervision corpus whose primary importance lies in structured visually grounded reasoning supervision rather than in open-ended multimodal reasoning coverage [2606.26794].

Source: https://www.emergentmind.com/topics/reasonpro-16m