VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RL (2505.23977v1)

Abstract: Vision LLMs (VLMs) are expected to perform effective multimodal reasoning and make logically coherent decisions, which is critical to tasks such as diagram understanding and spatial problem solving. However, current VLM reasoning lacks large-scale and well-structured training datasets. To bridge this gap, we propose VisualSphinx, a first-of-its-kind large-scale synthetic visual logical reasoning training data. To tackle the challenge of image synthesis with grounding answers, we propose a rule-to-image synthesis pipeline, which extracts and expands puzzle rules from seed questions and generates the code of grounding synthesis image synthesis for puzzle sample assembly. Experiments demonstrate that VLM trained using GRPO on VisualSphinx benefit from logical coherence and readability of our dataset and exhibit improved performance on logical reasoning tasks. The enhanced reasoning capabilities developed from VisualSphinx also benefit other reasoning tasks such as algebraic reasoning, arithmetic reasoning and geometry reasoning.

• The paper presents VisualSphinx, which generates over 660,000 synthetic vision logic puzzles using a novel four-stage pipeline.
• It employs a genetic algorithm for rule expansion and LLM-generated Python scripts for diverse image synthesis to boost dataset quality.
• Experimental results indicate that training with VisualSphinx significantly improves VLM accuracy in both logical reasoning tasks and broader domains such as algebra and geometry.

VisualSphinx: Enhancing Logical Reasoning in Vision-LLMs

The paper presents VisualSphinx, a system designed to generate large-scale synthetic vision logic puzzles for training and evaluating Vision LLMs (VLMs) on logical reasoning tasks. The authors introduce a comprehensive pipeline that generates over 660,000 visual puzzles, enhancing the multimodal reasoning capabilities of VLMs through reinforcement learning (RL).

Introduction

VisualSphinx addresses the need for structured datasets in training VLMs for tasks that require visual logical inference. The lack of large, high-quality training datasets has been a significant limitation for VLMs, particularly in tasks demanding multimodal logical reasoning. VisualSphinx fills this gap by providing a scalable and cost-effective solution, generating diverse puzzles that cover a broad range of reasoning styles, such as inductive, deductive, spatial, and structural logic.

Methodology

VisualSphinx adopts a four-stage pipeline for generating visual logic puzzles:

1. Seed Question Collection and Rule Abstraction: Begins with collecting 4,000 seed puzzles, which are abstracted into structured rule descriptions using LLMs.
2. Rule Expansion via Genetic Algorithm: Expands these into 40,000 high-quality rules through a genetic algorithm that applies mutation and crossover operations to enhance rule diversity.
3. Rule-to-Image Synthesis: Uses LLM-generated Python scripts to create image sets that include correct sequences and distractors. This stage incorporates three rendering styles to diversify visual features in 110,000 image groups (Figure 1).
4. Puzzle Assembly: Constructs puzzles using multiple strategies to ensure variety and difficulty, resulting in over 660,000 puzzles suitable for training multimodal reasoning models.

   Figure 1: The four-stage pipeline for generating VisualSphinx datasets.

Dataset Analysis and Model Evaluation

The dataset was analyzed based on readability, logical coherence, and solvability metrics. A custom VLM model was trained on the VisualSphinx dataset using GRPO and evaluated on a specially designed VisualSphinx test set and the MathVista-testmini benchmark. The model exhibited notable improvements in accuracy across all levels of difficulty, outperforming existing VLMs on logical reasoning tasks (Figure 2).

Figure 2: Model performance on VisualSphinx-Test across varying difficulty levels.

Experimental Results

Experiments confirm that training with VisualSphinx significantly enhances VLM accuracy in challenging visual logic puzzles. The fine-tuned model not only excels in visual logic tasks but also shows improved performance across other reasoning domains, including algebra and geometry, as evidenced on the MathVista-testmini benchmark (Figure 3).

Figure 3: Enhanced logical consistency of synthetic rules visualized using t-SNE.

Implications and Future Work

VisualSphinx effectively enhances the logical reasoning skills of VLMs, paving the way for more versatile AI systems capable of sophisticated multimodal reasoning. The approach balances scalability and cost-effectiveness, with the entire dataset generated for under $1000. Future research may extend VisualSphinx to include more complex reasoning paradigms, exploring temporal and interactive tasks to advance the theoretical understanding of VLM enhancement through synthetic datasets.

Conclusion

VisualSphinx represents a significant step in addressing data scarcity for multimodal reasoning tasks in AI. By providing a large-scale, diverse puzzle dataset, it offers a robust foundation for training VLMs, supporting improved logical reasoning capabilities and expanding applicability in real-world scenarios.