Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models

Abstract: Multimodal LLMs (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at https://github.com/zwyang6/SaGe.

• The paper introduces SaGe, which converts image–text pairs into hierarchical scene graphs to enable fine-grained visual reasoning in multimodal LLMs.
• It employs a two-stage training pipeline combining supervised fine-tuning with node-centric reinforcement learning to improve spatial grounding and semantic tracing.
• Experimental results demonstrate significant gains on fine-grained perception tasks, outperforming larger models despite using fewer parameters.

Scene Graph Thinking: Structured Visual Reasoning for Multimodal LLMs

Introduction

The paper "Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal LLMs" (2607.05716) introduces SaGe, a new paradigm designed to enhance multimodal LLMs through explicit scene-graph–aligned reasoning. Current MLLMs, despite substantial progress in multimodal perception and understanding, are limited by their reliance on flat visual representations that neglect structured inter-object relationships. This hampers their ability to perform fine-grained and holistic visual reasoning, as they operate on isolated objects without leveraging the compositional structure intrinsic to complex scenes.

SaGe addresses these limitations via (1) automated conversion of image–text pairs into hierarchical, attribute-rich scene graphs; (2) graph-aligned chain-of-thought (CoT) data sampling with rigorous quality verification; and (3) a two-stage training pipeline, including supervised fine-tuning and node-centric RL with proxy node rewards. These innovations collectively induce MLLMs to internalize structured visual reasoning and facilitate efficient navigation and inference in high-resolution, visually intensive tasks.

Figure 1: SaGe enables structured scene-graph–driven reasoning in MLLMs, contrasting with prior methods that overlook relational structure.

Scene Graph Construction and Data Synthesis

SaGe's foundation is an automated data engine that transforms flat image–text corpora into richly annotated scene graphs. For each image, salient objects and their subcomponents are mined as nodes, each equipped with semantic categories, bounding boxes, multi-attribute annotation, and depth cues. The extraction process follows a hybrid pipeline, leveraging a high-capacity MLLM for coarse object detection, followed by cropping and zooming for compositional sub-entity mining and re-annotation at multiple spatial scales. Each node is thus precisely localized in 2D (bounding box) and 3D (depth range) space, enhancing spatial grounding.

Edges in the graph—covering spatial, interactive, and semantic relations—are constructed using node-level priors (bbox overlap, component structure, depth proximity) rather than unreliable direct relation inference from MLLMs. This hierarchical, densely connected structure provides an explicit compositional prior unavailable to standard MLLM pipelines.

Figure 2: Pipeline for hierarchical scene graph construction incorporating compositional node mining, depth cues, and semantic relationships.

To generate training data, SaGe samples both node-centric (attribute, state, region, etc.) and edge-centric (spatial relation, interaction) queries, each paired with node-articulated CoT traces. Reasoning chains are explicitly formulated with tags for entity identity, localization, and depth, offering traceable evidence for the chain-of-thought. Multi-hop navigation and local/global caption tasks are synthesized by traversing the scene graph, producing 120K high-quality, structured reasoning samples after a two-stage verification pipeline—vision–text similarity for visual–semantic alignment, and validity-checking via proprietary MLLMs. This results in a failure rate below 1% by human validation.

Figure 3: Node- and edge-centric query sampling and verification yields high-quality, CoT-augmented structured training data.

Two-stage Graph-aligned Model Training

SaGe adopts a two-stage post-training paradigm. First, supervised fine-tuning on the 120K graph-derived corpus internalizes graph-aligned reasoning patterns into the MLLM's parametric memory. Fine evidence in the form of entity, bbox, and depth tags in CoT traces directly conditions the model toward explicit structural reasoning and fine-grained spatial grounding.

Second, reinforcement learning with Group Relative Policy Optimization (GRPO) further consolidates structured reasoning and mitigates spurious or untraceable solutions. Importantly, reward design avoids ambiguous edge-level supervision and instead introduces node-as-proxy rewards:

• Node-grounded reward: enforces visual–textual coherence at the node level, granting positive reward only if region crops align with entity descriptions.
• Node-relevance reward: encourages navigation through semantically relevant or target-supporting nodes, penalizing ambiguous or spurious routes in reasoning traces.

This hierarchical reward aggregation is combined with answer and format compliance rewards, and optimized over sampled rollouts against a reference SFT model.

Figure 4: Two-stage pipeline: supervised graph-aligned fine-tuning followed by RL with node-as-proxy graph rewards.

Experimental Results

SaGe demonstrates strong empirical advantages across diverse multimodal benchmarks. On fine-grained and high-resolution perception tasks (e.g., VStarBench, HRBench-4K/8K), SaGe-3B outperforms Qwen2.5-VL-3B by 13.6 points overall, and outperforms proprietary systems such as GPT-4o and rivals much larger models (e.g., InternVL3-78B) despite using an order of magnitude fewer parameters. The improvements are consistent for both attribute-focused and spatial reasoning sub-tasks.

In spatial understanding (2D/3D relational reasoning; CVBench-2D/3D), node-articulated CoT with multi-modal grounding yields substantial accuracy gains. For example, SaGe-3B achieves 17.3% and 5.8% gains over Qwen2.5-VL-3B on 2D and 3D spatial relation reasoning, respectively.

On general vision–language tasks (MMStar, RefCOCO, ChartQA), SaGe offers robust generalization, with improved grounding and chart analysis due to its localization- and relation-aware evidence tracing.

Figure 5: Case studies demonstrate SaGe's superiority in diverse QA scenarios, yielding correct, node-articulated CoT explanations where baselines fail.

Ablation studies substantiate the criticality of both node-articulated CoT patterns and node-centric RL rewards. Removal of structured tags ($<$entity$>$, $<$bbox$>$, $<$depth$>$) degrades performance, particularly for spatial reasoning. Similarly, ablations isolating node-relevance or node-grounded rewards show the necessity of their combination—exclusive use of only one leads to either reward hacking or inefficient navigation, while their synergy regularizes semantic tracing and visual grounding.

SaGe's data scaling and composition analysis show linear performance improvements with corpus size and a steady benefit from diversified graph-aligned query types (attributes, spatial, multi-hop, caption).

Figure 6: SaGe demonstrates consistent performance improvement with increased structured data scale and diversified graph-aligned reasoning types.

Scene Graph Quality and Data Diversity

SaGe's automatically constructed scene graphs are validated to have 94.5% node and 92.9% edge accuracy, verified using Gemini-3-Pro. Most node errors (~52%) are attribute–region mismatches; most edge errors arise from incorrect relation types. Human cross-validation confirms a sub-1% error rate in the final curated dataset.

The corpus is compositionally diverse: 25K attribute-focused (color, state, shape, material), 30K spatial, 25K multi-hop, 20K captioning, and 20K auxiliary (grounding/counting) samples. Captioning samples are further filtered using a multi-criteria approach emphasizing node/edge coverage and reasoning fidelity.

Figure 7: Node error breakdown confirms high precision in scene graph node construction.

Figure 8: Edge error breakdown validates edge reliability and relation quality in constructed scene graphs.

Implications and Future Directions

Practically, SaGe demonstrates that explicit, hierarchical scene-graph representations directly impart fine-grained, context-sensitive, spatial/relational understanding to MLLMs, even in parameter-limited regimes. The explicit compositional structure obviates the need for laborious heuristic searching or model-internal attention resampling, thus reducing navigation redundancy and enabling superior generalization on visually-intensive tasks.

Theoretically, this work suggests that explicit, high-quality structured knowledge injection—when paired with tailored RL reward design—enables MLLMs to go beyond weakly induced compositionality, supporting more precise, interpretable, and verifiable reasoning. The node-as-proxy reward structure provides a robust alternative to edge-based or path-constrained RL regimes, offering both regularization and strong cross-modal alignment.

Further research directions include:

• Scaling scene-graph construction pipelines to more unconstrained, dynamic, or real-world image domains.
• Integrating temporal and causal relations for video and event understanding.
• Differentiable scene graph representations and their influence on intrinsic model interpretability.
• Extension towards agentic navigation and task-planning scenarios where structured world modeling is critical.

Conclusion

Scene Graph Thinking (SaGe) establishes a rigorously structured paradigm for endowing MLLMs with compositional, spatially and semantically grounded visual reasoning. Utilizing automated scene-graph construction, graph-aligned data synthesis, and node-centric RL, SaGe significantly advances the state-of-the-art on fine-grained visual benchmarks and general multimodal tasks. The findings provide compelling evidence that explicit, hierarchical structure and tailored reward schemes are essential for enabling robust and interpretable visual reasoning in MLLMs.