ReasonCLIP-58M: Enhancing CLIP with Reasoning

• The paper introduces a framework that injects visually grounded commonsense reasoning into CLIP and SigLIP backbones without altering the architecture.
• It employs a stage-wise continual pretraining strategy using paired datasets, ReasonLite-42M and ReasonPro-16M, to balance descriptive alignment with reasoning supervision.
• Empirical results show significant improvements in retrieval, reasoning, and compositional benchmarks across various CLIP-scale models.

Searching arXiv for the requested paper and closely related context. {"query":"arXiv (Zhang et al., 25 Jun 2026) ReasonCLIP-58M", "max_results": 5} {"query":"ReasonCLIP-58M Visually Grounded Commonsense Reasoning Supervision for CLIP", "max_results": 10} {"query":"CLIP Learning Transferable Visual Models From Natural Language Supervision arXiv", "max_results": 5} ReasonCLIP-58M is a continual pretraining framework, dataset suite, and resulting family of CLIP-style encoders that injects visually grounded commonsense and compositional reasoning supervision into standard CLIP and SigLIP backbones without changing the backbone architecture. The designation “58M” refers to the total scale of reasoning supervision, with ReasonLite-42M and ReasonPro-16M together providing 58.9M samples. The central premise is that standard CLIP pretraining is dominated by descriptive image-text alignment, whereas downstream multimodal systems increasingly require visually grounded reasoning: inferring plausible consequences from visible evidence, rejecting coherent but visually unsupported conclusions, preserving correct binding among objects and relations, and distinguishing multiple reasoning types. ReasonCLIP addresses this mismatch through stage-wise continual pretraining, a paired dataset construction pipeline, and a dedicated diagnostic benchmark, RCLIP-Bench (Zhang et al., 25 Jun 2026).

1. Problem setting and conceptual motivation

ReasonCLIP is formulated around a specific limitation of standard CLIP-style pretraining. Descriptive image-text alignment is highly effective for retrieval, but it is not explicitly optimized for visually grounded commonsense inference or compositional reasoning. A descriptive caption can teach that an image contains entities or attributes, yet it does not necessarily teach a model to infer what follows from visible evidence, reject visually unsupported reasoning, distinguish reasoning categories, or maintain correct object-relation-state binding (Zhang et al., 25 Jun 2026).

The framework therefore treats visually grounded reasoning as a supervision problem that can be introduced during continual pretraining rather than through architectural modification. In this formulation, the key question is whether CLIP-style encoders can support reasoning-oriented downstream demands without changes to the inference-time model structure. ReasonCLIP answers this by separating reasoning injection from reasoning structuring. First, reasoning signals are introduced in a way that preserves the original alignment geometry; second, category-structured supervision is applied to organize reasoning patterns more explicitly in representation space.

This yields a specific contrast with naïve direct reasoning pretraining. The paper defines a baseline in which all reasoning captions are mixed as training targets without staging or category structure, and uses this to show that progressive integration is more effective than direct reasoning-only continual pretraining. A plausible implication is that the mismatch is not only about data content, but also about how reasoning supervision is scheduled during optimization.

2. Continual pretraining framework and objective design

ReasonCLIP uses a stage-wise continual pretraining scheme with four stages: Stage 0 for comparison baselines, Stage 1 for reasoning-aware alignment using open-form reasoning captions, Stage 2 for explicit category-level reasoning supervision, and Stage 3 for drop-in integration into multimodal systems such as LLaVA-NeXT (Zhang et al., 25 Jun 2026).

Stage 0 defines two comparison baselines. S0-Des. performs continual pretraining on descriptive captions $T_b$ from CC12M-Refined only. S0-Rea. naively mixes all reasoning captions, $T_{rl}$ and $T_{rp}$, as training targets without staging or category structure. Both use the same generic alignment objective:

$$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$

where $x$ is the image, $t$ is the paired text, and $\mathcal{L}_{\text{align}}$ is the backbone-specific image-text alignment loss: symmetric InfoNCE for CLIP-style models and sigmoid BCE alignment loss for SigLIP.

Stage 1 uses ReasonLite-42M and is designed to preserve descriptive alignment while gradually injecting reasoning awareness. Its loss is

$$\mathcal{L}^{(1)} = \lambda(t)\,\mathcal{L}_{\text{align}}(x, T_b) + \bigl(1-\lambda(t)\bigr)\,\mathcal{L}_{\text{align}}(x, T_{rl}) + \beta \|\theta - \theta_0\|_2^2$$

where $T_b$ is the descriptive caption, $T_{rl}$ is the ReasonLite reasoning caption, $T_{rl}$0 is the current parameter set, $T_{rl}$1 is the original pretrained parameter set, and $T_{rl}$2 is the L2 regularization coefficient that prevents drift from the base model. The scheduling function is

$T_{rl}$3

and the text states that $T_{rl}$4 decreases over time in practice, gradually increasing the influence of reasoning captions.

The practical Stage 1 schedule differs slightly by backbone. For CLIP, the weights on $T_{rl}$5 are $T_{rl}$6 over 0–20%, linearly transition to $T_{rl}$7 over 20–80%, and are fixed at $T_{rl}$8 over 80–100%. For SigLIP, they are $T_{rl}$9, then linearly to $T_{rp}$0, then fixed at $T_{rp}$1. The paper also compares descriptive-heavy, reasoning-heavy, and exponential schedules, and reports that the chosen piecewise strategy works best overall. This suggests a curriculum effect in which abrupt overemphasis on reasoning destabilizes alignment.

Stage 2 uses ReasonPro-16M and shifts from blended supervision to explicit category-structured reasoning supervision. Its loss is

$T_{rp}$2

where $T_{rp}$3 is a category-specific reasoning caption, $T_{rp}$4 is the multi-label category set associated with the image, $T_{rp}$5 is the single reasoning category of the specific caption, and $T_{rp}$6 is the balance weight for category supervision. The auxiliary losses are

$T_{rp}$7

with $T_{rp}$8, $T_{rp}$9, and lightweight classification heads $$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$0 and $$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$1 attached to image and text encoders during training only (Zhang et al., 25 Jun 2026).

The five reasoning categories are Spatial / Geometric (S), Attribute / State (A), Creature / Action (C), Temporal / Phase (T), and Intuitive Physics (P). Image-side labels are multi-label because one image may support several reasoning types, whereas text-side labels are single-label because each ReasonPro caption is intentionally written for one selected category. Appendix details specify $$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$2 classifiers for both text and image, using L2-normalized embeddings as input, with the heads discarded at inference. The deployment model therefore remains a standard CLIP or SigLIP encoder with no extra inference cost.

Stage 3 formalizes downstream replacement of the original vision tower:

$$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$3

where $$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$4 is the multimodal system, $$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$5 is the original visual encoder, $$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$6 is the reasoning-enhanced visual encoder, and $$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$7 is the rest of the multimodal pipeline.

3. Dataset suite: CC12M-Refined, ReasonLite-42M, and ReasonPro-16M

The framework is supported by a dataset pipeline built from CC12M and refined into three major assets: CC12M-Refined, ReasonLite-42M, and ReasonPro-16M (Zhang et al., 25 Jun 2026).

CC12M-Refined is the descriptive anchor. From CC12M, the pipeline retains 10,388,539 valid images. Using Qwen2.5-VL-72B with AWQ, it regenerates three descriptive captions per image, denoted $$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$8, yielding 31,165,584 image-$$\mathcal{L}^{(0)} = \mathcal{L}_{\text{align}}(x, t)$$9 pairs. The average caption length is 21.8 words, compared with 17.3 words for the original raw captions, whose variance is much larger. The reported annotation cost is approximately 1440 A100-64GB GPU hours.

ReasonLite-42M is the open-form reasoning dataset used in Stage 1. The 10,388,539 valid images are split so that 4,668,515 images are used for ReasonLite and 5.7M are reserved for ReasonPro. For each ReasonLite image and its descriptive caption $x$0, the annotation model extracts a visual evidence set $x$1, partitions $x$2 into three semantic clusters $x$3, and generates one visually grounded reasoning caption $x$4 for each cluster. The paper states that each image is paired with 9 $x$5-$x$6 annotations, producing 42,016,635 triplets. The supervision is dual: the descriptive caption $x$7 and the open-form reasoning caption $x$8. The captions are free-form rather than templated, are intended to remain consistent with the descriptive caption, and are constrained to avoid false causality, over-extension, hallucination, and complex multi-step reasoning beyond visible support. The average caption length is 25.0 words, with variance 8.1, and generation cost is about 1150 A100-64GB GPU hours.

ReasonPro-16M is the category-structured reasoning dataset used in Stage 2. Starting from the 5.7M images reserved from CC12M-Refined, the pipeline uses Qwen3-VL-32B for multi-label category annotation,

$x$9

retains only images with $t$0, and obtains 5,521,563 valid images after filtering. For each image, it randomly selects 3 reasoning categories $t$1, provides the image, $t$2, selected categories, and category definitions, and generates one category-specific reasoning caption per selected category. The final dataset contains 16,564,689 image-caption pairs.

The category distribution is explicitly reported: 4,766,178 Spatial/Geometric samples, 4,883,348 Attribute/State samples, 2,469,372 Creature/Action samples, 947,656 Temporal/Phase samples, and 3,498,135 Physical Intuition samples. The average caption length is 29.4 words with variance 18.2. Filtering removed 198,437 samples; the appendix reports 3.47% removed, whereas the main text reports 3.97% / 0.20M during category annotation. The paper notes this discrepancy and indicates it should be checked for exact reproduction.

Resource	Scale	Primary role
CC12M-Refined	31,165,584 image-$t$3 pairs	Descriptive anchors
ReasonLite-42M	42,016,635 triplets	Stage 1 dual supervision
ReasonPro-16M	16,564,689 image-caption pairs	Stage 2 category supervision

Quality control uses a two-stage process. A manual validation pilot reviews 500 samples per stage, each by 5 graduate students; prompts are iteratively refined; large-scale generation begins only once the pass rate exceeds 99.5%; and post-generation random inspection of 500 more samples yields a pass rate above 99.0%. Appendix prompt iteration reports 86.4% $t$4 95.2% $t$5 99.6% locked for ReasonLite, 89.8% $t$6 99.8% locked for ReasonPro, and a final post-annotation pass rate of 99.2%. Automatic filtering removes non-English outputs, captions shorter than 10 tokens, captions longer than 60 tokens, and repetition or degeneration. Reported filtering rates are below 0.01‰ for ReasonLite, around 0.20M for ReasonPro during category annotation, and approximately 13% for RCLIP-Bench negatives.

4. RCLIP-Bench and diagnostic evaluation design

RCLIP-Bench is introduced as a benchmark for diagnosing visually grounded reasoning failures in CLIP-style encoders. Its central design claim is that existing retrieval and compositional benchmarks do not disentangle whether a failure arises from bad perception, bad evidence selection, or bad reasoning despite correct facts. RCLIP-Bench is therefore constructed to isolate these failure sources (Zhang et al., 25 Jun 2026).

The benchmark is built from DOCCI image-caption pairs. For each image, the original DOCCI caption is used as the positive, and hard negatives are generated for different reasoning tiers. There are three levels. V1: Visual Grounding — Incorrect Facts contains factual inconsistencies such as subject/object form errors, noun substitution, relational misuse, attribute confusion, and sentence structure alteration. V2: Evidence Awareness — Incorrect Reasoning from Incorrect Facts starts from false premises and derives a coherent but unsupported conclusion. V3: Visual Reasoning — Incorrect Reasoning from Correct Facts preserves descriptive correctness but introduces invalid reasoning. V2 and V3 use the same five categories as ReasonPro: S, A, C, T, and P.

The appendix names the three benchmark splits rclip_5k_v1, rclip_5k_v2, and rclip_5k_v3. Each contains 5,000 unique images; each image has 5 categories or tags; each category has 1 ground-truth caption and 4 hard negatives; and each dataset version contains 125,000 annotations. Ground-truth mean token counts are 29.07 for V1, 33.41 for V2, and 33.39 for V3. Per-category mean token counts for V2/V3 are 40.41 / 39.18 for Scene/Spatial, 31.45 / 31.62 for Attribute, 29.18 / 29.33 for Creature, 32.93 / 33.35 for Temporal, and 33.08 / 33.47 for Physical.

Evaluation is conducted in a contrastive retrieval setting, namely whether the model ranks the correct caption above the hard negatives. The tables report accuracy-like percentages per subtype or category. This structure makes RCLIP-Bench less a general-purpose benchmark than a diagnostic instrument for grounded reasoning failure analysis.

5. Experimental setup and empirical results

The framework is applied to multiple CLIP-style backbones and scales: CLIP ViT-B/32 @224, CLIP ViT-L/14 @224, CLIP ViT-L/14 @336, SigLIP So400m/14 @384, SigLIP2 So400m/14 @384, and SigLIP2 Giant/16 @384. The paper trains six variants total across CLIP and SigLIP scales. Training uses NVIDIA A100 64GB hardware, approximately 3.8k GPU hours for data generation and 3.5k GPU hours for training, one epoch per stage, effective batch size 24,576 or 32,768 depending on scale, bfloat16 precision, AdamW, cosine schedulers, and warmup ratio 0.10 (Zhang et al., 25 Jun 2026).

Stage 1 uses ReasonLite-42M with weight decay 0.05, $t$7, logit learning rate $t$8, CLIP/SigLIP vision learning rate $t$9, and text learning rate typically $\mathcal{L}_{\text{align}}$0. It uses FlashAttention2 for SigLIP2 and PyTorch SDPA for CLIP. Stage 2 uses ReasonPro-16M, classifier learning rate 1.5e-3, $\mathcal{L}_{\text{align}}$1 for SigLIP and 0.10 for CLIP, no L2 regularization, and effective batch size 32,768.

The evaluation protocol re-evaluates models under a unified protocol using CLIP-bench where possible and measures zero-shot text-image retrieval on COCO-5K, Flickr30K, Urban-1K, and RCLIP-V3-5K; reasoning on WinoGAViL and RCLIP-Bench; compositional performance on WhatsUp, VALSE, CREPE, SugarCREPE, and SugarCREPE++; and downstream MLLM transfer through LLaVA-NeXT. Baselines include CLIP, OpenCLIP, SigLIP and SigLIP2, MetaCLIP, DataComp, EVA-CLIP-02, ViTamin, Long-CLIP, and READ-CLIP, while very large intrusive or LLM-augmented methods such as EVA-CLIP-18B, PE-Core, and LLM2CLIP are excluded from fair comparison.

Retrieval improvements are a major empirical result. For CLIP ViT-B/32 @224 on COCO-5K, baseline CLIP achieves I→T R@1 of 50.0 and T→I R@1 of 30.4, while Stage 1 reaches 56.2 and 37.9. On Flickr30K, I→T R@1 improves from 40.7 to 42.5; on Urban-1K, from 61.0 to 70.4; and on RCLIP-V3-5K, I→T R@1 improves from 51.0 to 56.0 and T→I R@1 from 26.7 to 30.4. For CLIP ViT-L/14 @336, baseline I→T R@1 values are 58.0 on COCO, 51.4 on Flickr, 73.0 on Urban, and 55.9 on RCLIP-V3; after Stage 1 they become 65.1, 61.0, 83.0, and 67.2. For SigLIP So400m/14 @384, baseline I→T R@1 values are 72.6 on COCO, 69.9 on Flickr, 74.5 on Urban, and 70.5 on RCLIP-V3; after Stage 2 they become 73.7, 73.3, 78.6, and 72.4, while RCLIP-V3 T→I R@1 improves from 43.4 to 49.5.

Reasoning-oriented evaluation shows a more differentiated pattern. For ViT-B/32 baseline CLIP, WinoGAViL average is 50.6, RCLIP V1 average is 22.3, V2 is 16.2, and V3 is 22.6; ReasonCLIP Stage 1 reaches 55.8, 23.6, 23.1, and 23.8, with the largest jump in V2 evidence awareness. For ViT-L/14 @336 baseline CLIP, WinoGAViL average is 50.2, V1 is 22.9, V2 is 17.7, and V3 is 25.0; ReasonCLIP Stage 1 reaches 60.2, 27.2, 19.3, and 25.4; and ReasonCLIP Stage 2 reaches 60.2, 29.5, 18.0, and 23.6. For SigLIP So400m/14 @384 baseline SigLIP, WinoGAViL average is 61.7, V1 is 24.4, V2 is 22.2, and V3 is 21.8; ReasonSigLIP Stage 1 yields 56.8, 30.8, 25.8, and 26.0; and Stage 2 yields 30.7, 24.1, and 22.8 on V1, V2, and V3. Human averages are 95.7 on V1, 91.1 on V2, and 91.7 on V3, while GPT-5.2 records 86.5, 86.5, and 88.4, indicating substantial remaining headroom.

Compositional benchmarks are reported for Stage 2 models. For ViT-B/32, baseline CLIP averages 52.1 across compositional benchmarks and ReasonCLIP reaches 57.7, a gain of +5.6; per benchmark, WhatsUp improves from 41.2 to 51.3, VALSE from 67.5 to 72.5, CREPE from 23.9 to 24.0, SugarCREPE from 73.1 to 75.6, and SugarCrepe++ TOT from 46.7 to 61.5. For ViT-L/14 @224, the average rises from 51.4 to 57.9; for ViT-L/14 @336, from 51.8 to 58.1; for SigLIP So400m/14, from 56.6 to 62.7; and for SigLIP2 Giant/16, from 55.7 to 60.2. The paper summarizes these as roughly +5 to +7 average gains.

MLLM transfer is evaluated by replacing CLIP-L/14-336 with ReasonCLIP-L/14-336 in LLaVA-NeXT using Qwen3-1.7B as the LLM, with the visual tower frozen, 558K pretraining data, and 779K finetuning data. Relative to baseline LLaVA-NeXT, the ReasonCLIP variant changes AI2D from 58.1 to 58.6, ChartQA from 26.8 to 28.0, SciQA from 73.1 to 73.0, RealWorldQA from 49.4 to 49.5, VisualLogic from 25.7 to 25.7, OKVQA from 34.0 to 35.8, GQA from 52.8 to 53.4, MME from 1296.1 to 1301.8, MMStar from 39.7 to 39.9, and MMVP from 58.7 to 62.3.

6. Ablations, limitations, and interpretive significance

Ablation results consistently indicate that stage-wise training is better balanced than naïve direct continual pretraining. The comparison among Description CP, Reasoning CP, and Stage 1 / Stage 2 shows that Stage 1 often preserves descriptive retrieval best while adding reasoning, that Stage 2 can improve structured reasoning more strongly for larger models, especially in the SigLIP family, and that direct reasoning pretraining is less stable or less effective than the progressive design (Zhang et al., 25 Jun 2026).

The Stage 1 schedule is itself important. In a CLIP-L/14-like setting, the default Stage 1 schedule yields COCO 55.6, Flickr 50.5, V1 26.6, V2 19.2, and V3 24.3. Descriptive-heavy scheduling weakens Flickr and reasoning, reasoning-heavy scheduling causes a large drop in retrieval and reasoning, and exponential scheduling underperforms the default. This supports the paper’s interpretation that too much reasoning too early harms alignment, whereas too much descriptive weight under-injects reasoning.

Stage 2 category-structured supervision produces a more mixed but still informative pattern. In one ablation, Stage 2 gives COCO 53.2, Flickr 47.3, V1 28.6, V2 18.3, and V3 23.1; a variant without classification heads has almost the same retrieval but slightly weaker V2 and V3; a single-label image-side variant is weaker; and adding L2 is not helpful. The appendix further notes that some SigLIP models show Stage 2 without classification heads can be competitive or stronger on some compositional metrics. This suggests that category supervision helps organize features, but is not uniformly dominant across all benchmarks.

The paper also distinguishes the roles of ReasonLite and ReasonPro. ReasonLite and Stage 1 are especially effective for preserving CLIP alignment while boosting reasoning. ReasonPro and Stage 2 are more beneficial when model capacity is larger and structured reasoning can be absorbed without degrading alignment. The authors state that for ReasonSigLIP, Stage 2 gives the most balanced overall performance, especially on retrieval, whereas for some compositional and RCLIP tasks Stage 1 still performs better. For ReasonCLIP, Stage 1 improvements are more pronounced and Stage 2 gains are relatively limited, suggesting that smaller-capacity or less scalable CLIP backbones are more sensitive to explicit supervision.

A key trade-off concerns preservation of descriptive alignment versus object-centric discrimination. The framework does not destroy retrieval and often improves it, but the appendix reports slight degradation on standard zero-shot classification tasks such as ImageNet. The authors explain this as a shift from object-centric discrimination toward relation- and context-centric understanding. A plausible implication is that the method reallocates representational capacity toward inter-object structure, evidence binding, and grounded inference, which is beneficial for retrieval, reasoning, and MLLM transfer but not uniformly beneficial for single-object classification.

Several limitations are stated explicitly. The 58.9M reasoning scale is still much smaller than industrial CLIP pretraining corpora; evaluation does not cover all CLIP applications; stronger retrieval does not necessarily imply stronger reasoning; Stage 2 can partially disrupt the balance between semantic alignment and reasoning for some backbones; and standard object-centric classification may degrade slightly. The large gap between CLIP-like models and human or GPT-5.2 performance on RCLIP-Bench further indicates that visually grounded reasoning remains far from solved (Zhang et al., 25 Jun 2026).

Within the broader CLIP literature, ReasonCLIP is characterized by large-scale visually grounded reasoning supervision during continual pretraining, non-intrusive backbone compatibility, explicit staged integration of reasoning, and a category-structured reasoning taxonomy. This differs from approaches centered on larger descriptive corpora, architectural changes, long-text adaptation, smaller-scale compositional finetuning, or external knowledge injection. Its main significance is therefore methodological: it argues that structured visually grounded reasoning supervision can extend the expressive capacity of CLIP-style representations through pretraining alone, while keeping the deployment-time encoder unchanged.