Reason-3D: Reasoning in 3D Perception
- Reason-3D is a research program that integrates explicit reasoning signals, such as intermediate rationales and spatial computations, into various 3D perception tasks.
- It encompasses diverse methodologies for visual grounding, segmentation, and scene generation using modalities like point clouds, meshes, and RGB-D videos.
- Empirical findings indicate that using explicit reasoning traces can measurably improve localization, segmentation, and spatial analysis despite challenges in data quality.
Reason-3D denotes a family of reasoning-centric approaches to 3D perception, grounding, segmentation, question answering, and scene generation in which models are expected not only to localize or describe 3D content, but also to expose intermediate structure such as explicit rationales, symbolic predicates, spatial computations, or procedural plans. The term is not standardized across the literature: some works use it as an umbrella idea for reasoning-first 3D intelligence, while others introduce differently named datasets and models—such as Reason3D, ReasonSeg3D, MORE3D, Scene-R1, Reason3DVG, SpatialReasoner, and text-to-scene Reason-3D—whose common thread is the coupling of 3D grounding with explicit reasoning mechanisms (Huang et al., 2024, Yuan et al., 21 Jun 2025, Berdoz et al., 30 Sep 2025).
1. Terminological scope and naming
A central feature of the literature is terminological heterogeneity. “Reason-3D” sometimes names a concrete system, sometimes an organizing research agenda, and sometimes nothing official at all within a given paper.
| Work | Authors’ term | Role of “Reason-3D” |
|---|---|---|
| "Scene-R1: Video-Grounded LLMs for 3D Scene Reasoning without 3D Annotations" (Yuan et al., 21 Jun 2025) | Scene-R1 | Described as an annotation-efficient “Reason-3D” solution |
| "Multimodal 3D Reasoning Segmentation with Complex Scenes" (Jiang et al., 2024) | ReasonSeg3D / MORE3D | “Reason-3D” is not a named component |
| "Reason3D: Searching and Reasoning 3D Segmentation via LLM" (Huang et al., 2024) | Reason3D | Explicit model name |
| "Reasoning Matters for 3D Visual Grounding" (Huang et al., 13 Jan 2026) | Reason3DVG-8B | Reasoning-first 3D grounding pipeline |
| "Text-to-Scene with Large Reasoning Models" (Berdoz et al., 30 Sep 2025) | Reason-3D | Explicit text-to-scene model name |
The most explicit clarification appears in the ReasonSeg3D/MORE3D work, which states that “Reason-3D” does not name any component proposed there, and that the paper’s own contributions are the multi-object 3D reasoning segmentation task, the ReasonSeg3D benchmark, and the MORE3D model (Jiang et al., 2024). By contrast, Reason3D uses the name directly for a point-cloud LLM that produces textual responses and dense 3D masks (Huang et al., 2024), while the text-to-scene system uses Reason-3D for an LRM-based generation pipeline that retrieves assets and places them under physical and contextual constraints (Berdoz et al., 30 Sep 2025).
This suggests that Reason-3D functions less as a single method family with a fixed ontology than as a broad research program centered on explicit reasoning over 3D structure.
2. Core problem settings
Across papers, Reason-3D-style systems target several distinct but related tasks. One major track is open-vocabulary 3D visual grounding: given a scene and a free-form query, the model localizes the referred object or region in 3D. Scene-R1 formulates this in RGB-D video as temporal segment selection, per-frame 2D box prediction, mask tracking, and 2D-to-3D lifting, with optional extension to 3D visual question answering (Yuan et al., 21 Jun 2025). Reason3DVG-8B instead consumes text-serialized 3D detector proposals and outputs a target object ID through a four-stage reasoning process (Huang et al., 13 Jan 2026). ReGround3D generalizes explicit 3D grounding to implicit instructions in ScanReason, where the output is both a textual answer/explanation and a flexible set of target 3D boxes (Zhu et al., 2024).
A second track is reasoning segmentation. Reason3D outputs dense 3D masks from point clouds and text prompts for 3D reasoning segmentation, hierarchical searching, express referring, and 3D QA with mask outputs (Huang et al., 2024). MORE3D addresses multi-object 3D reasoning segmentation: a point cloud and a multi-object query are mapped to multi-object 3D segmentation masks and a textual explanation that captures both user intent and 3D spatial relations (Jiang et al., 2024). PARIS3D shifts the target from whole-scene grounding to part-level segmentation of 3D objects under implicit functional or relational queries, such as localizing the part “you would grab to pour tea from a kettle” (Kareem et al., 2024). Reasoning3D similarly defines zero-shot 3D reasoning segmentation on meshes, rendering multiple views, producing reasoning-driven 2D segmentations, and fusing them back to 3D faces or vertices (Chen et al., 2024).
A third track focuses on explicit spatial reasoning. SpatialReasoner decomposes 3D reasoning into perception, computation, and reasoning stages, each sharing object-level 3D state in a calibrated camera frame (Ma et al., 28 Apr 2025). ReRe treats spatial reasoning from egocentric video as revisitable: an MLLM first forms a hypothesis from the original video and then verifies or revises it after observing a synthesized allocentric video generated from inferred 3D geometry (Ma et al., 10 Jun 2026). MAG-3D organizes open-ended 3D reasoning into a planning agent, a grounding agent, and a coding agent that verifies answers by executable geometric computation (Zheng et al., 10 Apr 2026).
A fourth track extends Reason-3D beyond analysis into generation and task planning. CoRe3D couples semantic chain-of-thought with geometric chain-of-thought over localized octant regions in a 3D latent space for both understanding and generation (Yu et al., 14 Dec 2025). The text-to-scene Reason-3D model uses Large Reasoning Models to retrieve assets, extract explicit and implicit layout constraints, place objects autoregressively, and refine placements with collision-aware reasoning (Berdoz et al., 30 Sep 2025). Relationship-aware hierarchical scene-graph methods and XR-oriented spatial reasoners further serialize 3D environments into object- and relation-centric structures for rule-based or LLM-mediated task reasoning (Puigjaner et al., 2 Feb 2026, Häsler et al., 25 Apr 2025).
3. Representational and algorithmic paradigms
One defining axis of Reason-3D research is how reasoning is made explicit. In Scene-R1, the model is trained to emit visible rationales in > … and structured outputs in <answer>…</answer> or JSON, with temporal grounding and image grounding optimized by GRPO rather than dense 3D supervision (Yuan et al., 21 Jun 2025). After 2D localization, SAM2 propagates masks across frames, and 3D grounding is obtained by back-projecting foreground pixels with depth and poses:
The final 3D prediction is an accumulated point cloud and its axis-aligned 3D box (Yuan et al., 21 Jun 2025).
Reason3D makes the language-to-mask interface explicit through two special LLM-predicted tokens, [LOC] and [SEG], which drive a hierarchical coarse-to-fine 3D mask decoder over superpoints (Huang et al., 2024). MORE3D uses a different mechanism: <SEG> tokens are inserted into generated explanations, their hidden states are indexed through , and the resulting object-specific embeddings are fused with per-point features in a 3D decoder to produce masks and classifications (Jiang et al., 2024). PARIS3D and Reasoning3D offload part of the problem to multi-view 2D segmentation, then lift or vote those outputs back into 3D via superpoints or face-ID mappings (Kareem et al., 2024, Chen et al., 2024).
A second paradigm is explicit geometric state. SpatialReasoner represents each object by 3D position , orientation, and derived quantities used in subsequent reasoning stages (Ma et al., 28 Apr 2025). Its relation tests are deliberately operational: Euclidean distance is , front/behind can be computed with , and left/right with (Ma et al., 28 Apr 2025). This differs from systems that rely on latent cross-modal features without exposing the intermediate geometry.
A third paradigm is symbolic or programmatic reasoning. MAG-3D stores grounded instances, point maps, and retrieved views in a shared scene memory, then lets a coding agent translate spatial-language intents into executable programs for distance, containment, visibility, and counting (Zheng et al., 10 Apr 2026). Relationship-aware scene graphs encode objects, rooms, and object–object relation features into a hierarchical graph that an LLM and VLM can traverse for task reasoning (Puigjaner et al., 2 Feb 2026). The XR Spatial Reasoner similarly maps OBB geometry to symbolic predicates such as on, behind, near, and inside, then evaluates dynamic rules over a spatial knowledge graph (Häsler et al., 25 Apr 2025).
A fourth paradigm is reasoning through view synthesis or hierarchical latent structure. ReRe preserves the MLLM’s native video interface by converting predicted 3D geometry into a synthesized novel-view video, then prompting the model to compare the new evidence against its earlier hypothesis (Ma et al., 10 Jun 2026). ReasonGrounder extends 3D Gaussian Splatting with hierarchical language and instance features conditioned on physical scale, then uses LVLM-guided interpretation and HDBSCAN clustering to localize occluded objects under implicit queries (Liu et al., 30 Mar 2025). CoRe3D decomposes the 3D latent space into 512 localized octants and autoregressively predicts octant tokens under semantic and geometric chain-of-thought, aligning linguistic predicates with spatially localized generation steps (Yu et al., 14 Dec 2025).
4. Datasets and benchmark ecology
The dataset landscape around Reason-3D is unusually broad, spanning indoor RGB-D scans, synthetic scenes, multi-view images, mesh renderings, and medical volumes.
| Dataset / benchmark | Scale | Primary use |
|---|---|---|
| ReasonSeg3D (Jiang et al., 2024) | 1,513 scenes; 20,113 QA-mask triplets | Multi-object 3D reasoning segmentation |
| RPSeg3D (Kareem et al., 2024) | 2,624 objects; over 60k instructions | 3D reasoning part segmentation |
| ScanReason (Zhu et al., 2024) | 12,929 QA-location triplets; 1,456 scenes | 3D reasoning grounding |
| ReasoningGD (Liu et al., 30 Mar 2025) | over 10K scenes; about 2M annotations | Open-vocabulary grounding and amodal perception |
| SPAR-7M / SPAR-Bench (Zhang et al., 29 Mar 2025) | over 7M QA pairs / 7,207 benchmark questions | 2D-to-3D spatial perception and reasoning |
| 3DReasonKnee / ReasonKnee-Bench (Sambara et al., 23 Oct 2025) | 494k quintuples from 7,970 MRI volumes | Grounded reasoning in 3D medical imaging |
ReasonSeg3D is notable for emphasizing free-form multi-object spatial relational descriptions rather than a fixed relation ontology; its answers place <SEG> tokens adjacent to object mentions so that language and masks are directly coupled (Jiang et al., 2024). ScanReason instead operationalizes implicit human-intention reasoning across spatial, functional, logical, emotional, and safety categories, using object categories and 3D boxes from EmbodiedScan plus GPT-4-assisted generation and human verification of the validation set (Zhu et al., 2024).
SPAR-7M takes a different route: it uses 3D ground truth from ScanNet, ScanNet++, and Structured3D to generate over 7 million 2D QA pairs that teach LVLMs depth, distance, cross-view matching, camera-pose inference, and spatial imagination, while still operating on images at inference time (Zhang et al., 29 Mar 2025). ReasoningGD supplies about 100 RGB-D views per scene together with both modal and amodal masks, making occlusion-aware open-vocabulary grounding measurable (Liu et al., 30 Mar 2025). In medicine, 3DReasonKnee introduces 494k quintuples containing a 3D MRI volume, a diagnostic question, a 3D bounding box, clinician-generated reasoning steps, and structured severity assessments aligned with MOAKS (Sambara et al., 23 Oct 2025).
This diversity indicates that Reason-3D is not tied to a single input modality. RGB-D videos, point clouds, meshes, slice stacks, and pure 2D multi-view images are all used, provided that the supervision or inference path exposes 3D structure in a form the model can reason over.
5. Empirical findings
The empirical record consistently shows that explicit reasoning supervision or explicit intermediate structure improves 3D performance, though the form of that improvement varies by task.
Scene-R1 reports, on ScanRefer, overall [email protected] = 38.8% and [email protected] = 17.1%, and states that it outperforms the prior label-free baseline OpenScene by +25.6 and +10.6 points at 0.25 / 0.5 IoU; on SQA3D it reaches EM-1 = 49.4% and EM-R1 = 51.6% (Yuan et al., 21 Jun 2025). Its ablations are especially revealing: temporal grounding mIoU rises from 15.1 in zero-shot mode to 34.3 with GRPO, image grounding on ScanRefer rises from 32.2 / 25.9 to 70.4 / 59.4 at [email protected] / 0.5, and removing the <think> prompt degrades performance across stages (Yuan et al., 21 Jun 2025).
MORE3D reports cIoU = 30.19 and gIoU = 32.01 on the ReasonSeg3D validation set, substantially above OpenScene, PLA, and RegionPLC (Jiang et al., 2024). Its ablations show that using the exact <SEG> token index list is materially better than random hidden-state selection, that supervising both language and masks yields the best result, and that dot-product fusion outperforms addition or concatenation in the 3D decoder (Jiang et al., 2024). PARIS3D reports validation mIoU 55.42 and test mIoU 56.77 on RPSeg3D, exceeding LISA-MV and LISA-MV(ft), while also generating explanations (Kareem et al., 2024).
Reason3DVG-8B demonstrates the strongest direct evidence for the value of reasoning traces in LLM-based 3D grounding. On ScanRefer it reports [email protected] = 38.7 and [email protected] = 34.4, surpassing 3D-GRAND while using only 1.6% of its training data (Huang et al., 13 Jan 2026). In the oracle-class NR3D setting, training without reasoning traces yields 33.5 overall, whereas four-stage reasoning supervision yields 49.3, a gain of +15.8 (Huang et al., 13 Jan 2026).
SpatialReasoner reports 60.3 mean accuracy on 3DSRBench, outperforming Gemini 2.0 by 9.2%, and shows that explicit 3D representations improve SFT performance from 51.9 to 58.3 (Ma et al., 28 Apr 2025). ReRe provides a complementary result at inference time: on VSI-Bench, Qwen3-VL-4B improves from 30.7 to 36.5 and Qwen3-VL-2B from 22.5 to 31.0 after cross-view revisiting, while simple concatenation of original and synthesized videos does not match the two-phase protocol (Ma et al., 10 Jun 2026).
MAG-3D shows that training-free 3D reasoning also benefits from explicit decomposition. On Beacon3D, it improves GPT-4o case-level QA score from 57.1 to 63.5 and Seed-1.6 from 60.2 to 65.0, while improving object-level QA score and Grounding–QA coherence over SceneCOT (Zheng et al., 10 Apr 2026). In generation, the text-to-scene Reason-3D model reports human-preference win rates of 95.2 against Holodeck and 98.4 against LayoutVLM, with Elo 2248 versus 1500 and 1650 respectively (Berdoz et al., 30 Sep 2025).
Taken together, these findings support a narrow but recurring claim: when reasoning is made operational—through explicit intermediate states, structured traces, geometric computation, or revisitable evidence—performance often improves measurably.
6. Limitations, misconceptions, and open directions
A common misconception is that visible reasoning traces are equivalent to correctness. The evidence is more limited. Scene-R1 reports that encouraging thinking improves localization and QA scores, but it also identifies failure modes tied to depth noise, inaccurate poses, SAM2 drift, sparse viewpoints, and the remaining accuracy gap to fully supervised detector-centric systems (Yuan et al., 21 Jun 2025). Reason3DVG-8B shows that detector quality remains the main bottleneck relative to fully supervised models, as oracle-class results are substantially better than ordinary proposal-based evaluation (Huang et al., 13 Jan 2026). SpatialReasoner’s own analysis concludes that 3D perception, rather than 3D computation, is the main bottleneck, with orientation accuracy only 35.5% on a human-verified subset (Ma et al., 28 Apr 2025).
Another misconception is that Reason-3D necessarily requires explicit 3D input representations at inference. “From Flatland to Space” argues the opposite: 3D spatial reasoning can be improved by 3D-grounded 2D data generation while keeping inference image-based, and its SPAR-mix training substantially improves VSI-Bench and CV-Bench 3D results without introducing point-cloud encoders or explicit 3D inputs at inference (Zhang et al., 29 Mar 2025). Conversely, methods such as SpatialReasoner, MAG-3D, and relationship-aware scene graphs show that explicit 3D states, geometry, and symbolic relations can materially improve auditability and generalization (Ma et al., 28 Apr 2025, Zheng et al., 10 Apr 2026, Puigjaner et al., 2 Feb 2026).
Several open problems recur across papers. ReasonSeg3D notes that spatial relations are encoded in free text rather than a formal ontology with explicit thresholds or relation metrics (Jiang et al., 2024). Reasoning3D and PARIS3D remain challenged by small parts, ambiguous implicit queries, and occlusion (Chen et al., 2024, Kareem et al., 2024). ReRe assumes quasi-static geometry and can degrade when monocular reconstruction is noisy or when parallax is insufficient (Ma et al., 10 Jun 2026). MAG-3D remains vulnerable to clutter, reflections, and geometry errors from textureless surfaces (Zheng et al., 10 Apr 2026). Text-to-scene Reason-3D assumes upright asset pre-alignment and approximately correct scales, and it can miss collisions in dense scenes despite refinement (Berdoz et al., 30 Sep 2025). 3DReasonKnee further shows that even when localization and reasoning are jointly annotated at large scale, supervised fine-tuning with CoT did not yet outperform SFT without CoT in its initial experiments (Sambara et al., 23 Oct 2025).
A plausible implication is that the next phase of Reason-3D research will depend less on merely adding longer verbal rationales and more on improving the interfaces between reasoning and geometry: better detector or parser quality, richer relation schemas, stronger verification mechanisms, and benchmarks that jointly score where a model looks, what it computes, and whether its explanation is faithful to the underlying 3D evidence.