ReasonPro-16M: Structured Reasoning Dataset

• ReasonPro-16M is a large-scale dataset featuring 5.5M images paired with 16.6M category-specific reasoning captions for structured supervision in vision-language models.
• It provides explicit category-level reasoning across five taxonomy types, enhancing discriminability and enabling fine-grained analysis in CLIP-style encoders.
• The dataset is built using a multistage pipeline with Qwen3-VL-32B, rigorous quality control, and controlled supervision to optimize reasoning alignment.

Searching arXiv for the cited paper to ground the article. ReasonPro-16M is a large-scale training corpus introduced in "ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP" as the second, more structured supervision dataset in the paper’s two-stage continual pretraining framework for CLIP-style encoders. It consists of category-specific commonsense reasoning captions paired with images, and it is used in Stage 2 to impose explicit category-level reasoning supervision after a first stage based on more open-form reasoning captions. In the paper’s formulation, ReasonPro-16M is intended not merely as additional caption data, but as a controlled supervision mechanism that organizes visually grounded reasoning patterns inside the joint image-text representation space (Zhang et al., 25 Jun 2026).

1. Position within the ReasonCLIP framework

ReasonPro-16M occupies the second stage of the ReasonCLIP-58M recipe. The framework introduces reasoning progressively rather than in a single undifferentiated training step. Stage 1 uses ReasonLite-42M, which provides open-form, visually verifiable reasoning captions and mixes those captions with regenerated descriptive captions in order to preserve descriptive alignment while increasing reasoning awareness. Stage 2 then switches to ReasonPro-16M, whose supervision is explicitly category-specific and no longer uses the descriptive captions $T_b$.

The paper defines ReasonPro as a dataset of “category-specific commonsense reasoning captions” that provides “more structured and controllable supervision than the open-form generation in ReasonLite.” The motivation is that alignment alone does not explicitly encourage discriminability across different reasoning categories. ReasonPro therefore targets predefined visual reasoning types and supports fine-grained per-category analysis aligned for CLIP.

This design also links directly to the evaluation protocol. RCLIP-Bench uses the same five reasoning categories for its V2 and V3 levels, so ReasonPro supplies the training taxonomy while RCLIP-Bench supplies category-aware diagnostic evaluation. A plausible implication is that ReasonPro-16M is intended less as an open-ended reasoning corpus than as a structured supervision layer for category separability and analyzable reasoning behavior (Zhang et al., 25 Jun 2026).

2. Dataset composition and reasoning taxonomy

ReasonPro-16M contains 5,521,563 valid images after filtering, each paired with 3 category-specific reasoning captions, for a total of 16,564,689 image-caption pairs. The main text summarizes this as “5.5M images and 16.6M image-caption pairs spanning up to five reasoning categories.” The source images come from CC12M-Refined, specifically the 5.7M images reserved from CC12M-Refined after separating the subset used for ReasonLite.

Its supervision is intrinsically one-to-many and multi-view. Each image is first assigned a multi-label set of supported reasoning categories, after which three categories are selected and three corresponding captions are generated. One image may therefore support several valid reasoning types, but each caption is tied to one specific category. The dataset’s core ontology is the fixed five-way taxonomy

$\{S, A, C, T, P\}.$

The five categories are defined as follows. Spatial / Geometric (S) covers spatial relations, direction, distance, occlusion, containment, accessibility, and geometry. Attribute / State (A) covers appearance, texture, wetness, openness, deformation, brightness, on/off state, and related visible states. Creature / Action (C) covers human or animal posture, gestures, interaction, and current or imminent behavior. Temporal / Phase (T) covers whether an event just happened, is ongoing, or is about to happen. Intuitive Physics (P) covers visually inferable stability, support, contact, force, and physical interaction; the appendix characterizes it as reasoning about “relatively obvious physical relationships that can be inferred directly from visual cues,” and “not relying on semantic common sense, but on visual physical intuition.”

The released category counts are imbalanced: Spatial/Geometric 4,766,178, Attribute/State 4,883,348, Creature/Action 2,469,372, Temporal/Phase 947,656, and Physical Intuition 3,498,135. Caption statistics are also reported: average length 29.4 words and variance 18.2. The comparatively small Temporal/Phase count is a direct indication that the dataset emphasizes high-volume automatic generation with quality control rather than strict class balancing (Zhang et al., 25 Jun 2026).

3. Construction pipeline and quality control

ReasonPro-16M is built from CC12M through a multistage recaptioning and annotation process. The paper first collects 10,388,539 images from CC12M and constructs CC12M-Refined by generating 3 regenerated descriptive captions $T_b$ per image using Qwen2.5-VL-72B, yielding 31,165,584 image-$T_b$ pairs. From this refined pool, 4.7M images are assigned to ReasonLite and 5.7M images are reserved for ReasonPro.

All ReasonPro annotations are then produced by Qwen3-VL-32B, which the paper says was chosen because it offered “stronger category discrimination and more structured outputs than the model used for ReasonLite.” The first step is multi-label category annotation. Given the predefined category definitions, the model predicts

$Y \subseteq \{S, A, C, T, P\}$

for each image, where $Y$ is the set of visually supported reasoning types. Only images satisfying

$|Y| \ge 3$

are retained. The appendix reports a start from 5,720,000 samples, removal of 198,437 samples, and retention of 5,521,563 valid samples. The main text reports a filtering rate of 3.97% (0.20M) due to “inconsistent or low-confidence labels,” while the appendix reports 3.47%; both point to low-single-digit filtering.

The second step is category-conditioned caption generation. For each retained image, the pipeline randomly selects three reasoning categories $Y' \subseteq Y$, and feeds the model the image, its base caption $T_b$, the selected categories $Y'$, and the category definitions in a single generation request. The model then generates one category-specific reasoning caption for each selected category. These captions are denoted $\{S, A, C, T, P\}.$0, so each image contributes three pairs: $\{S, A, C, T, P\}.$1

Quality control combines manual validation and automatic filtering. Before large-scale generation, 500 samples per stage were reviewed by five graduate students, prompts were iteratively refined, and generation proceeded only after pass rate exceeded 99.5%. For ReasonPro specifically, the initial prompt had ambiguous category boundaries; after reinforcing strict and mutually exclusive definitions, pass rate reached 99.8%. After full generation, random inspection gave 99.2% pass rate across stages. Automatic filtering includes non-English outputs, outputs shorter than 10 tokens, outputs longer than 60 tokens, and repetition or degeneracy loops; the pipeline also uses structural and grounding constraints and rule-based cleanup to remove malformed outputs. The paper does not explicitly document a separate deduplication pass for ReasonPro-16M beyond these screening steps (Zhang et al., 25 Jun 2026).

4. Supervision format and Stage 2 objective

ReasonPro-16M uses caption-form supervision rather than QA, instructions, or class names. Each training instance includes an image $\{S, A, C, T, P\}.$2, a category-specific reasoning caption $\{S, A, C, T, P\}.$3, an image-level multi-label category set $\{S, A, C, T, P\}.$4, and a caption-level single category label $\{S, A, C, T, P\}.$5. The corresponding structured targets are an image-side multi-hot vector

$\{S, A, C, T, P\}.$6

and a text-side single label

$\{S, A, C, T, P\}.$7

The paper characterizes the resulting training regime as image-text alignment augmented with category classification. The Stage 2 objective is

$\{S, A, C, T, P\}.$8

with auxiliary losses

$\{S, A, C, T, P\}.$9

For the alignment term, CLIP uses symmetric InfoNCE, while SigLIP uses sigmoid-based binary cross-entropy alignment.

ReasonPro-16M is used only in Stage 2, and the paper is explicit that descriptive captions $T_b$0 are no longer used at that point. Descriptive alignment is instead preserved in Stage 1 through a separate objective,

$T_b$1

which mixes descriptive and reasoning captions while regularizing deviation from the initial model. Stage 2 therefore specializes a model that has already been made reasoning-aware without fully abandoning descriptive alignment.

The category-discrimination branch added for Stage 2 uses lightweight heads. The method section describes them as “lightweight MLP-based classification heads,” while the appendix gives the implementation as Text classifier: $T_b$2 and Image classifier: $T_b$3 with L2-normalized text/image embeddings as input, bias enabled, and bfloat16 dtype. These heads are used only during training and are discarded at inference. Stage 2 is run for 1 epoch on ReasonPro-16M, with per-device batch size 512, gradient accumulation 2, effective batch size 32,768, AdamW, classifier learning rate $T_b$4, weight decay 0.10, warm-up ratio 0.10, and 32 A100 64GB GPUs. The vision/text/logit learning rates are $T_b$5 for SigLIP and $T_b$6 for CLIP; $T_b$7 is 0.05 for SigLIP and 0.10 for CLIP; and there is no L2 regularization in Stage 2 (Zhang et al., 25 Jun 2026).

5. Empirical effects attributable to ReasonPro-16M

The cleanest empirical view of ReasonPro-16M comes from Stage 2 comparisons. On RCLIP-V3-5K retrieval, Stage 2 often improves substantially over the base backbone. For CLIP ViT-B/32, base CLIP reports I→T R@1 51.0 and T→I R@1 26.7, while Stage 2 reports 55.3 and 33.8. For CLIP ViT-L/14 @224, the corresponding values move from 54.0 / 31.7 to 67.1 / 41.9. For CLIP ViT-L/14 @336, they move from 55.9 / 33.2 to 68.0 / 42.8. For SigLIP So400m, they move from 70.5 / 43.4 to 72.4 / 49.5. These results indicate that ReasonPro-based Stage 2 generally strengthens reasoning-caption retrieval.

The appendix also reports RCLIP-Bench behavior. For CLIP-L/14 @336, base V1/V2/V3 avg is 22.9 / 17.7 / 25.0, while Stage 2 gives 29.5 / 18.0 / 23.6. For SigLIP So400m @384, base is 24.4 / 22.2 / 21.8, while Stage 2 gives 30.7 / 24.1 / 22.8. The paper’s own interpretation is that explicit category supervision is not uniformly beneficial on every reasoning dimension for every backbone: V1 improves strongly, V2 changes modestly, and V3 does not always exceed the base model for lower-capacity CLIP variants.

ReasonPro-16M is also tied to the paper’s compositional reasoning gains, because the main compositional table reports Stage 2 variants. For ReasonCLIP B/32 (+58M, which includes ReasonPro) versus base CLIP B/32, the reported scores are WhatsUp 51.3 vs 41.2, VALSE 72.5 vs 67.5, CREPE 24.0 vs 23.9, SugarCREPE 75.6 vs 73.1, SugarCrepe++ TOT 61.5 vs 46.7, and Avg 57.7 vs 52.1. The corresponding averages are 57.9 vs 51.4 for ReasonCLIP L/14 @224, 58.1 vs 51.8 for ReasonCLIP L/14 @336, and 62.7 vs 56.6 for ReasonSigLIP So/14. The paper also reports that ReasonCLIP + READ exceeds READ alone in the B/32 case, with average 65.7 vs 64.1.

At the same time, the ablations make the trade-off explicit. For ReasonSigLIP, Stage 2 gives the most balanced overall performance, especially retrieval. For some compositional and RCLIP-Bench settings, Stage 1 can be better. For ReasonCLIP, Stage 1 improvements are more pronounced and Stage 2 gains are relatively limited; the paper states that excessive explicit supervision in lower-capacity CLIP backbones may weaken alignment. It also reports that the classification heads help somewhat on V2 and V3 but often only marginally, and that replacing the default image-side multi-label classification with single-label classification hurts both retrieval and reasoning metrics. This suggests that ReasonPro-16M’s main empirical contribution is strongest when the backbone can absorb explicit reasoning-category supervision without substantially disrupting the underlying alignment space (Zhang et al., 25 Jun 2026).

6. Limitations, caveats, and reproducibility

ReasonPro-16M is intentionally narrow in reasoning scope. The appendix states that it excludes false causality, over-extension, hallucination, and complex multi-step reasoning beyond what is visually supported. It therefore does not target open-ended chain-of-thought or long-horizon reasoning; instead, it focuses on moderately difficult, directly image-grounded inference suitable for CLIP-style contrastive learning.

The dataset also has several explicit caveats. Category frequencies are imbalanced, with Temporal/Phase far smaller than Spatial/Geometric or Attribute/State. All annotations are model-generated with Qwen3-VL-32B, so residual noise may remain despite the manual pilot validation and filtering pipeline. The paper notes that early prompt iterations suffered from ambiguous boundaries between categories, especially between Spatial/Geometric and Intuitive Physics. No explicit category-balancing sampler is reported for Stage 2, and no separate deduplication pass is documented for ReasonPro-16M. The full 58M corpus is also much smaller than industrial billion-scale CLIP pretraining, which limits the extent to which the reported gains can be generalized.

A further caveat is architectural and empirical. Stage 2 can weaken alignment, especially for lower-capacity CLIP backbones; some standard zero-shot classification results drop slightly after reasoning-oriented pretraining, and the appendix interprets this as a shift from object-centric to relation/context-centric representations. The paper therefore positions ReasonPro not as a universally dominant supervision source, but as a structured specialization stage whose benefits are clearest on reasoning retrieval and compositional benchmarks, particularly in higher-capacity or SigLIP-family models.

From a reproducibility standpoint, the dataset is specified precisely: 5,521,563 valid images, 16,564,689 image-caption pairs, 3 $T_b$8 captions per image, image-side multi-label $T_b$9, and text-side single label $T_b$0. The paper does not describe train/validation/test splits because ReasonPro-16M is a training corpus used for one epoch of Stage 2 continual pretraining. Release links are provided through the project GitHub and Hugging Face collections. The appendix states that the work is based on CC12M under its permissive terms, that images are accessed only via URLs, and that the authors do not redistribute images. Reported annotation cost is ~280 GPU hours for category annotation and ~960 GPU hours for $T_b$1 annotation, both with Qwen3-VL-32B. Taken together, these details characterize ReasonPro-16M as a controlled, category-aware supervision corpus whose primary importance lies in structured visually grounded reasoning supervision rather than in open-ended multimodal reasoning coverage (Zhang et al., 25 Jun 2026).