SaGe: Scene Graph Thinking for MLLMs

• Scene Graph Thinking (SaGe) is a paradigm that represents scenes as explicit graphs of entities and relations to organize visual perception and reasoning.
• The framework converts flat image-text data into structured scene graph representations and uses node-as-proxy rewards to fine-tune multimodal large language models.
• Empirical evaluations on benchmarks like VStarBench and CVBench demonstrate significant gains in spatial and 3D reasoning, affirming its effectiveness.

Scene Graph Thinking (SaGe) denotes a mode of visual and multimodal reasoning in which a scene is represented and processed as an explicit graph of entities and relations rather than as a flat collection of isolated objects. In its most explicit formulation, SaGe is a post-training paradigm for multimodal LLMs (MLLMs) that improves fine-grained visual perception, structured relational understanding, and efficient visual reasoning through scene-graph-conditioned supervision and graph-aligned reinforcement fine-tuning (Yang et al., 7 Jul 2026). In a broader research interpretation, the term also refers to methods that use scene graphs as intermediate structures for memory, retrieval, planning, grounding, or control rather than treating them only as final prediction targets.

1. Conceptual basis

A scene graph, in the standard scene-understanding literature, is a visually grounded graph whose nodes correspond to objects and attributes and whose edges represent pairwise relations; in practice, much of the field operationalizes this structure through triplets such as $\langle subject, relation, object \rangle$ and $\langle object, is, attribute \rangle$ (Zhu et al., 2022). SaGe inherits this representational substrate but changes its epistemic role. Rather than using a graph only to summarize perception after the fact, it treats the graph as the structure through which perception and reasoning are organized.

The original SaGe formulation targets a specific weakness of contemporary MLLMs: they often reason over isolated objects or heuristic crop sequences rather than over the structured organization of a scene. SaGe therefore reframes the training signal itself. A flat image-text corpus is converted into scene-graph-conditioned data, and the model is trained to reason as if it is traversing a graph: identifying relevant nodes, grounding them visually, following relations, and producing answers supported by explicit node-level evidence. In this design, entities are not left implicit inside latent activations; they are surfaced as named nodes with attributes, bounding boxes, and depth cues, while relations supply the connective tissue for multi-hop inference (Yang et al., 7 Jul 2026).

A common misconception is that SaGe necessarily implies a new graph neural architecture or an inference-time graph parser. The original method does neither. It is a graph-aligned post-training framework built on Qwen2.5-VL-3B and Qwen2.5-VL-7B, and its central claim is that scene-graph reasoning can be internalized by an existing MLLM through structured data and reward design rather than by replacing the backbone (Yang et al., 7 Jul 2026).

2. Automated graph construction and graph-derived supervision

The original SaGe pipeline begins with an automated hybrid annotation engine that transforms flat image-text corpora into hierarchical scene graphs. Each graph is written as $G_i = (V_i, E_i)$. Nodes are not restricted to whole objects; they also include sub-components, which gives the representation an explicitly hierarchical character. The $t$-th node is defined as

$v_t = \bigl(c_t,\; a_t,\; b_t,\; d_t\bigr),$

where $c_t$ is semantic category, $a_t$ attributes, $$b_t = [x_t^{\min}, y_t^{\min}, x_t^{\max}, y_t^{\max}]$$ the 2D bounding box, and $d_t = [z_t^{\min}, z_t^{\max}]$ an estimated depth range (Yang et al., 7 Jul 2026).

Graph construction proceeds in three stages: compositional node mining, depth-aware node modeling, and edge construction via node priors. Qwen2.5-VL-72B detects salient primary objects with bounding boxes and attributes; cropped regions are then reprocessed to mine sub-entities; Depth Anything supplies depth estimates; and SAM refines regions so that depth cues are assigned to segmented entities rather than to loose bounding boxes. Edges encode spatial, interaction, and semantic relations, and the construction pipeline stores the resulting graphs in JSON format. On 1,000 sampled raw graphs evaluated by Gemini-3-Pro, the reported graph quality is 94.5% node precision and 92.9% edge precision; 52.0% of node errors come from mismatches between attributes and visual regions, and 55.6% of edge errors come from invalid relations (Yang et al., 7 Jul 2026).

Using these graphs, SaGe constructs 120K high-quality structured training samples. The corpus is mainly generated from SA-1B, with an additional 10K counting samples from COCO 2014. Its composition is explicitly diversified: 25K attribute-centric samples, 30K spatial reasoning samples, 25K multi-hop reasoning traces, 20K caption samples, 10K grounding samples, and 10K counting samples. Reasoning traces are articulated inside > with <entity>, <bbox>, and <depth> tags, so the model is shown not only what answer to produce but also what node-level evidence supports the answer. A two-round verification procedure filters visual-semantic mismatches and inconsistent reasoning traces, and the final corpus is reported to have a failure rate below 1% under random human cross-validation (Yang et al., 7 Jul 2026).

This data engine is central to the SaGe idea. The graph is not merely a label space; it becomes the source from which questions, captions, traces, and supervision are sampled.

3. Graph-aligned post-training and node-as-proxy rewards

SaGe uses a two-stage post-training schedule: supervised fine-tuning followed by reinforcement fine-tuning with Group Relative Policy Optimization. The supervised stage internalizes graph-structured reasoning patterns by training on graph-derived questions, answers, captions, and node-articulated traces. An important ablation isolates the contribution of the trace schema itself. For SaGe-3B, answer-only supervision yields VStarBench 79.1, CVBench-2D 72.8, and CVBench-3D 66.4; adding <entity> raises VStarBench to 79.9; adding <bbox> raises VStarBench to 80.6 and CVBench-2D to 75.0; adding <depth> produces 83.2, 75.5, and 69.7 respectively. The implication stated in the paper is that entity naming improves semantic attribution, bounding boxes improve 2D spatial reasoning, and depth cues improve 3D reasoning (Yang et al., 7 Jul 2026).

The reinforcement stage addresses a different problem: how to reward graph-consistent reasoning trajectories when edge supervision is ambiguous. SaGe introduces node-as-proxy graph rewards rather than edge-level rewards. The node-grounded reward checks whether a visited node is visually grounded by comparing the <entity> description with the cropped image region corresponding to its bounding box; the implementation uses Qwen3-VL-30B-A3B-Instruct as judge. The node-relevance reward checks whether visited nodes are anchored in the query and answer, or are directly connected and semantically necessary for navigation; the implementation uses Qwen3-30B-A3B-Instruct. The relevance scale is graded: 1.0 for exact query-consistent match, 0.8 for semantically similar but not identical, 0.5 for correct subject with missing or misused query-specific attributes, 0.2 for loosely related but wrong subject selection, and 0.0 for conflicting or irrelevant nodes (Yang et al., 7 Jul 2026).

The reward design is deliberately asymmetrical. The paper reports that node-grounded reward alone can hurt performance, likely because it can be hacked by attending to visually salient but task-irrelevant entities, whereas node-relevance reward alone helps strongly; combining both works best. For SaGe-3B, GRPO only gives VStarBench 85.9, CVBench-2D 76.2, and CVBench-3D 69.3; adding node-relevance reward gives 87.1, 76.9, and 71.5; node-grounded reward alone gives 85.0, 74.6, and 67.6; and combining both yields 89.0, 77.8, and 72.3 (Yang et al., 7 Jul 2026).

A second misconception is that SaGe necessarily requires an explicit scene graph at inference time. The original paper emphasizes the opposite: graph construction is used to create supervision and rewards so that the model learns to behave as if it is traversing a scene graph. The main goal is internalization.

4. Empirical performance

SaGe is evaluated on eight multimodal benchmarks spanning high-resolution fine-grained perception, spatial reasoning, and more general visual tasks. The largest gains occur where structured relational navigation appears to matter most: VStarBench, CVBench-2D relation, and CVBench-3D depth (Yang et al., 7 Jul 2026).

Benchmark	Qwen2.5-VL-3B	SaGe-3B
VStarBench	75.4	89.0
HRBench-4K	66.0	72.0
HRBench-8K	63.1	68.9
CVBench-2D	67.0	77.8
CVBench-3D	66.5	72.3
MMStar	52.2	54.4
RefCOCO	86.3	89.0
ChartQA	82.2	83.8

The same pattern holds at 7B scale. Qwen2.5-VL-7B rises from 76.4 to 89.0 on VStarBench, from 68.8 to 76.5 on HRBench-4K, from 65.3 to 73.4 on HRBench-8K, from 73.6 to 79.4 on CVBench-2D, and from 73.3 to 80.5 on CVBench-3D. On VStarBench’s spatial subset, the gains are especially pronounced: 67.1 to 93.4 for 3B and 75.0 to 94.7 for 7B. On CVBench-2D relation, the gains are 72.6 to 89.9 for 3B and 84.9 to 93.2 for 7B. On CVBench-3D depth, the gains are 73.6 to 79.2 for 3B and 79.8 to 88.8 for 7B (Yang et al., 7 Jul 2026).

The ablations indicate that supervised graph-aligned data produce the largest initial improvement, while reinforcement learning and node-as-proxy rewards further sharpen exploration. The paper also reports that performance improves steadily as training data scale increases, and that adding multi-hop reasoning traces and graph-traversal captions improves VStarBench from 80.6 to 83.2. A notable teacher-comparison result is that SaGe-7B surpasses the Qwen2.5-VL-72B graph-construction teacher on several sub-benchmarks, including VStarBench attribute, VStarBench spatial, CVBench-2D count, CVBench-2D relation, and CVBench-3D depth, though it remains lower on CVBench-3D distance (Yang et al., 7 Jul 2026).

These results define the empirical core of SaGe: graph-aligned supervision improves not only graph-specific tasks but also broader multimodal reasoning, with the strongest effect on structured spatial perception.

5. Broader interpretations and adjacent formulations

The SaGe label is most precise for the graph-aligned post-training paradigm above, but adjacent research uses scene graphs in closely related ways: as retrieval priors, persistent memory, transition systems, event structures, or embodied world models. Several of these works are explicitly described as highly relevant to a Scene Graph Thinking perspective.

System	Scene-graph role	Domain
SaGe (Yang et al., 7 Jul 2026)	Graph-derived supervision and node-as-proxy rewards for MLLM post-training	Multimodal reasoning
SGR3 (Wang et al., 4 Mar 2026)	Retrieved scene graphs as external memory; generated scene graph as output	Semantic 3D scene graph generation
SAGE (Li et al., 26 Sep 2025)	Scene graph transition chain bridging task planning and visuomotor control	Long-horizon manipulation
GraphThinker (Cheng et al., 19 Feb 2026)	Event graph as intermediate thinking process	Video reasoning
ISE (Hu et al., 2022)	Persistent graph updated incrementally rather than regenerated	Scene graph modification

SGR3 is a particularly direct adjacent case. It is described as a training-free framework for semantic 3D scene graph generation from RGB image sequences without explicit 3D reconstruction. Its core move is graph-to-graph retrieval: visually aligned scene graph fragments are retrieved from an external knowledge base and inserted directly into the prompt as structured relational evidence. The paper explicitly states that retrieved scene graphs are both memory and output, and reports a copy ratio of 64.7% and an object-pair copy ratio of 71%, supporting the view that concrete structural reuse, rather than abstract rule extraction, drives much of the gain (Wang et al., 4 Mar 2026).

SAGE, in robotics, uses scene graphs as the explicit intermediate representation between high-level semantic planning and low-level image-conditioned control. It parses the current scene into a graph, asks an LLM to produce a scene graph transition chain in which each transition changes exactly one relationship edge intentionally, converts each target graph into a sub-goal image through layout prediction, inpainting, and composition, and then executes the plan phase by phase with a goal-conditioned policy. On unseen tasks, it reports 25/30 on Sequential Task, 20/30 on Flexible Task, 26/30 on Hybrid Task I, and 22/30 on Hybrid Task II, while several end-to-end baselines score 0/30 across all unseen tasks (Li et al., 26 Sep 2025).

GraphThinker extends the same logic to video reasoning. It constructs an Event-based Video Scene Graph whose top-level nodes are event subgraphs with timestamps and internal triplets, then injects that graph back into the MLLM as an intermediate thinking process. On RexTime, GraphThinker with RL reports mIoU 41.46, Accuracy 71.30, and Accuracy@IoU$\ge 0.5$ 30.75, compared with 13.60, 56.60, and 6.35 for zero-shot Qwen2.5-VL-7B; on VidHalluc it improves especially on Temporal Sequence Hallucination and Scene Transition Hallucination (Cheng et al., 19 Feb 2026).

ISE offers a different but conceptually important variant. It treats scene graph modification as incremental structure expanding rather than conditional regeneration: a source graph persists, the unchanged subgraph is preserved, and the target graph is reached through localized expansion steps. This makes the graph a stateful semantic memory rather than an ephemeral output (Hu et al., 2022).

Other work broadens the same agenda toward more persistent or more universal substrates. Universal Scene Graph proposes a modality-spanning graph that merges modality-invariant and modality-specific scene semantics, while Multiview Scene Graph frames graph construction as recovering place-place and place-object correspondences from unposed images (Wu et al., 19 Mar 2025, Zhang et al., 2024). Incremental 3D Gaussian systems such as OGScene3D and DGSG-Mind push scene-graph-centric reasoning toward online, open-vocabulary, dynamically updated world models with explicit grounding, relocalization, and long-term maintenance (Zhu et al., 17 Mar 2026, Ge et al., 28 May 2026). Taken together, these systems suggest that SaGe is best understood not as a single architecture but as a research direction centered on graph-mediated cognition.

6. Limitations, misconceptions, and open questions

SaGe and its adjacent formulations do not amount to a fully explicit symbolic reasoner. In the original post-training framework, scene-graph reasoning is internalized through data and rewards; in SGR3, retrieved triplets are injected as prompt context but the actual reasoning remains autoregressive inside Qwen3-VL 32B; in GraphThinker, inter-event edges are formally timestamp-based and logical constraints are enforced during refinement rather than by a separate causal solver (Yang et al., 7 Jul 2026, Wang et al., 4 Mar 2026, Cheng et al., 19 Feb 2026). A stronger classical notion of graph reasoning—graph search, logical inference, constraint propagation, or executable graph programs—is therefore only partially represented in this literature.

Graph quality also remains a bottleneck. In the original SaGe pipeline, node precision is 94.5% and edge precision is 92.9%, which is high but not perfect; recall is explicitly described as harder to define because the graphs are open-ended and hierarchical; depth cues are approximate ranges rather than precise geometry; and node-grounded reward alone can produce reward hacking toward salient but irrelevant nodes (Yang et al., 7 Jul 2026). The general scene graph literature adds older but still relevant concerns: long-tail predicate distributions, dataset bias, noisy or incomplete annotations, ambiguity in relation labels, quadratic object-pair scaling, and evaluation metrics that do not fully capture scene-graph faithfulness (Zhu et al., 2022).

A third misconception is that any method with a scene graph is necessarily a SaGe method. The adjacent papers indicate a more restrictive interpretation. Some systems use graphs as offline outputs; others use them as persistent memory, transition chains, or retrieval priors; others still use them only to generate training data. The degree of “thinking with scene graphs” varies with how strongly the graph shapes inference rather than merely describing it afterward. This suggests that the most informative distinction is not whether a graph appears, but whether it acts as an operational substrate for reasoning.

Open directions follow directly from these limits. The evidence across post-training, retrieval-augmented reasoning, manipulation planning, and dynamic 3D mapping suggests that future SaGe-like systems may need stronger uncertainty handling, richer relation ontologies, closed-loop graph updating, and more explicit coupling between graph reasoning and raw perceptual evidence. A plausible implication is that the field is moving toward hybrid systems in which scene graphs are neither purely symbolic nor purely latent, but synchronized with retrieval modules, renderable 3D substrates, and policy-level control interfaces.