ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

Abstract: CLIP and its variants are widely adopted visual backbones in multimodal systems, but their pretraining remains dominated by descriptive image-text alignment. As downstream applications increasingly demand visually grounded commonsense inference and compositional reasoning, it remains unclear whether CLIP-style encoders can support such reasoning without architectural changes. To address this, we present ReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models through our two-stage strategy, which progressively integrates reasoning signals while preserving descriptive alignment, followed by category-structured reasoning supervision. To support this framework, we construct two complementary datasets and a benchmark: ReasonLite-42M, with open-form, visually verifiable reasoning captions; ReasonPro-16M, with category-specific reasoning supervision; and RCLIP-Bench for diagnostic evaluation of visually grounded reasoning. We train a family of ReasonCLIP that improves visually grounded commonsense and compositional reasoning while also enhancing zero-shot retrieval performance. As a drop-in visual encoder for multimodal LLMs such as LLaVA-NeXT, ReasonCLIP delivers consistent gains without additional inference cost, demonstrating that structured reasoning supervision enhances the expressive capacity of CLIP-style visual representations. All datasets, models, and training code are available at https://github.com/RISys-Lab/ReasonCLIP.

• The paper introduces a two-stage continual pretraining framework that integrates reasoning signals into CLIP without altering its backbone architecture.
• It leverages large-scale ReasonLite-42M and category-specific ReasonPro-16M datasets to enhance both descriptive retrieval and structured, evidence-aware reasoning.
• Empirical results show significant improvements in visual grounding, compositional reasoning, and multimodal integration across several benchmarks.

ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

Motivation and Problem Statement

Contrastive Language-Image Pretraining (CLIP) constitutes the canonical backbone for multimodal vision-LLMs, but prevailing pretraining paradigms emphasize descriptive image-text alignment at scale. However, downstream requirements—such as compositional reasoning, evidence-aware commonsense inference, and structured visual logic—are not explicitly targeted in these pretraining objectives. This disconnect raises the question: Can reasoning-oriented visual representations be injected into CLIP-style encoders, augmenting their expressive capacity, without modifying the backbone architecture?

Two-Stage Continual Pretraining Framework

ReasonCLIP-58M introduces a structured two-stage continual pretraining pipeline designed to enable visually grounded commonsense reasoning and compositional logic inference in CLIP-style visual encoders:

1. Stage 1 — Reasoning-Aware Alignment: The model is progressively exposed to large-scale visually verifiable reasoning statements in addition to standard descriptive captions, using the newly constructed ReasonLite-42M dataset. The dual-supervision schedule explicitly preserves the original semantic alignment while smoothly integrating reasoning signals via dynamic weighting and parameter regularization.
2. Stage 2 — Explicit Category-Level Reasoning Supervision: ReasonPro-16M enables fine-grained, category-specific supervision across five orthogonal reasoning categories (Spatial/Geometric, Attribute/State, Creature/Action, Temporal/Phase, Intuitive Physics). Multi-label and single-label classification heads facilitate discriminability and organize reasoning patterns.

Crucially, the pipeline is architecturally non-intrusive: no changes to the backbone structure, ensuring compatibility as a drop-in visual encoder for existing MLLMs.

Figure 1: ReasonCLIP-58M follows a two-stage continual pretraining strategy, injecting visually grounded reasoning signals into CLIP without architectural modification, resulting in improved retrieval, structured reasoning, and MLLM integration.

Dataset Construction and Benchmarking

Three new resources underpin the ReasonCLIP framework:

• ReasonLite-42M: Open-form visually grounded reasoning statements extracted and grouped from CC12M images.
• ReasonPro-16M: Category-specific reasoning captions rigorously assigned via high-fidelity annotation, supporting multi-label per image and enforcing mutually exclusive definitions for each reasoning category.
• RCLIP-Bench: A diagnostic benchmark decomposing visually grounded reasoning into three hierarchical levels—visual grounding (V1), evidence awareness (V2), and structured reasoning (V3)—with well-controlled negative distractors stratified for precise failure mode analysis.

Quality control combines dense human validation (>99.5% pass rates), automatic rule-based filtering, and tiered prompt iteration, guaranteeing factual consistency and minimizing generation bias.

Empirical Results

Retrieval and Reasoning

Across COCO, Flickr30K, Urban1K, and RCLIP-Bench, ReasonCLIP achieves strong gains in both descriptive retrieval and reasoning-oriented retrieval benchmarks, outperforming both scaled-data approaches (e.g., MetaCLIP, DataComp, EVA-CLIP) and compositional fine-tuning methods, despite using a comparatively modest corpus. ReasonCLIP and ReasonSigLIP deliver robust improvements regardless of backbone, scale, or architecture.

Summary of Key Numerical Findings

• Zero-shot text-image retrieval: ReasonCLIP-L/14-336 achieves +7 absolute improvement in R@1 (Image→Text) on COCO-5K relative to the vanilla backbone.
• RCLIP-Bench hierarchical reasoning: ReasonCLIP achieves stable improvements across all three V1/V2/V3 tiers, indicating enhanced evidence-awareness and visual logical inference beyond standard vision-language association metrics.

Compositional Reasoning

Evaluated on WhatsUp, VALSE, CREPE, SugarCREPE, and SugarCREPE++, ReasonCLIP consistently surpasses base models by average margins of +5 to +7 points across categories, demonstrating stable enhancement of structured compositional reasoning. When combined with state-of-the-art compositional finetuning (READ-CLIP), ReasonCLIP provides further additive gains.

MLLM Integration

Substitution of the original CLIP visual encoder in LLaVA-NeXT with ReasonCLIP yields reproducible improvements (e.g., +1.8 points in OKVQA, +3.6 in MMStar) on commonsense-driven benchmarks, without loss on standard non-reasoning tasks (e.g., AI2D, ScienceQA), confirming that reasoning-aware visual representations propagate benefits to downstream multimodal models.

Ablation and Training Strategy

Ablation studies show that the progressive two-stage alignment strategy is essential: excessive bias toward reasoning supervision (Rea.-heavy) disrupts semantic alignment; insufficient reasoning integration (Des.-heavy) yields suboptimal reasoning capacity; exponential scheduling is less stable. Category supervision and dynamic weights are necessary for optimal tradeoff between retrieval and reasoning.

Limitations and Theoretical Implications

ReasonCLIP-58M is trained at a moderate scale, serving as a proof-of-concept for reasoning-oriented continual pretraining. While exhaustive coverage of all vision-language downstream tasks is infeasible within this framework, enabling reasoning supervision at scale demonstrates extensibility and generalizability. This approach systematically extends the representational capacity of CLIP-style models beyond descriptive alignment and opens pathways toward unified visual reasoning within large multimodal paradigms.

Future Directions in AI

Given the efficacy of reasoning-enhanced supervision, anticipated future developments include:

• Scaling ReasonCLIP to industrial-scale corpora for further benchmark dominance;
• Integration into generalized MLLM frameworks as reasoning-aware vision towers to maximize downstream performance on evidence-aware and compositional benchmarks;
• Automated discovery of novel reasoning categories and cross-modal logical patterns leveraging large LLM-based annotation engines;
• Investigation of the interactions between reasoning supervision and dense vision-language tasks (e.g., object detection, segmentation).

Conclusion

ReasonCLIP-58M establishes that CLIP-style visual encoders can be systematically augmented with visually grounded commonsense and compositional reasoning through non-intrusive, structured continual pretraining. The improvements are consistent across retrieval, hierarchical reasoning, and compositional benchmarks, and propagate into multimodal LLMs. This approach provides a principled, scalable methodology for advancing reasoning-aware vision-language foundation models.