GS-Reasoner: Unified 3D Visual Grounding

• The paper introduces GS-Reasoner, the first fully autoregressive 3D LLM that fuses semantic, geometric, and positional cues without relying on external modules.
• It employs a dual-path pooling mechanism that integrates cross-attention driven semantic features with interpolation-based geometric cues to form dense, patch-aligned 3D representations.
• The architecture achieves state-of-the-art performance on benchmarks like ScanRefer and VSI-Bench while introducing the GCoT dataset to enhance grounded chain-of-thought supervision.

The term GS-Reasoner refers to the "Grounded-Spatial Reasoner," a 3D LLM architecture and framework for unified 3D visual grounding and spatial reasoning. It is the first model to achieve fully autoregressive grounding without external modules, constructing a dense patch-based 3D representation that fuses semantic and geometric context for end-to-end reasoning grounded in rich spatial data. GS-Reasoner introduces a dual-path pooling mechanism to resolve the foundational representation challenges in 3D LLMs and is evaluated on novel benchmark datasets and a diverse set of reasoning and captioning tasks, demonstrating state-of-the-art (SOTA) performance across spatial and general 3D vision-language benchmarks (Chen et al., 15 Oct 2025).

1. Unified 3D Visual Grounding and Reasoning: Motivation

GS-Reasoner addresses two fundamental limitations in prior 3D LLMs: the absence of a unified representation that fuses semantic, geometric, and positional context, and the heavy dependence on external grounding modules such as proposal or detection systems. Traditional 3D LLMs either lack the ability to align geometric features with semantic cues (resulting in poor grounding accuracy) or segment the reasoning process with ad hoc, non-differentiable modules (precluding true end-to-end integration). GS-Reasoner's architecture leverages image-patch aligned 3D features across modalities and spatial cues, enabling direct "in-model" 3D reasoning (Sec. 1, 3).

2. Dual-Path Pooling Mechanism

The representation backbone of GS-Reasoner is the dual-path pooling pipeline (Sec. 3.2). Each RGB-D frame is first back-projected into a patch-aligned 3D point map. These are processed as follows:

• Semantic-Aligned Geometric Feature (Cross-Attention): For each image patch, semantic features $s_i$ (from a SigLIP Vision Transformer) and geometric features $\{g_{ik}\}$ (from a Point Transformer v3) are fused using cross-attention:

$$\alpha_{ik} = \mathrm{softmax}_k\left( \frac{(s_i W_q) (g_{ik} W_k)^{\top}}{\sqrt{d_k}} \right)$$

$$\widetilde g_i^{(s)} = \sum_{k=1}^K \alpha_{ik} (g_{ik} W_v)$$

• Position-Aligned Geometric Feature (Interpolation): The center pixel $p_i^c$ samples a 3D point, and its corresponding geometric features are interpolated:

$\widetilde g_i^{(p)} = \mathrm{Interp}(G_i, p^c_i)$

• Hybrid Patch Representation: The two geometric features are concatenated and projected:

$$g_i = W_f[\widetilde g_i^{(s)}; \widetilde g_i^{(p)}]$$

$h_i = W_h[s_i; g_i; E_{\mathrm{pos}}(p_i^c)]$

where $E_{\mathrm{pos}}$ is a 3D sinusoidal positional encoding.

This fusion ensures that both the semantic and geometric structure of 3D scenes are captured on a per-patch level without increasing the token count, which is crucial for scalable LLM reasoning (Sec. 3.2, Fig. 2b).

3. End-to-End GS-Reasoner Architecture

The GS-Reasoner pipeline (Sec. 3, Fig. 2) consists of:

• Semantic Encoder: Extracts patch-level visual features from SigLIP ViT.
• Geometric Encoder: Back-projects depth into point clouds, then globally encodes them with Point Transformer v3.
• Dual-Path Pooling: Produces per-patch hybrid features $h_i$ aligned in semantic, geometric, and positional space.
• Tokenization: Each hybrid patch feature is projected to a single token—no token inflation.
• Video LLM (Qwen2-7B backbone): Consumes the concatenated sequence of tokenized 3D patch features and the tokenized query. It autoregressively predicts (in token space):
  1. All detected/relevant 3D objects with world-frame bounding boxes,
  2. Stepwise spatial reasoning (with explicit reference to grounded objects),
  3. A final answer.

By unifying grounding and reasoning without external modules, GS-Reasoner delivers an end-to-end and extensible architecture for spatial intelligence.

4. Grounded Chain-of-Thought (GCoT) Dataset

To bridge grounding with reasoning, the GCoT dataset (Sec. 4) was introduced:

• Construction: Based on ScanNet, ScanNet++, and ARKitScenes, spatial and temporal QA pairs are generated. Each includes:

  • 3D bounding boxes for all referenced objects,
  • Chain-of-thought explanations that explicitly first ground objects, then reason about the spatial relationships,
  • Both complex multi-step and simple reasoning tasks.
• Format Example:
  • With CoT: > radiator 1 <bbox>(-1.9165,...) </bbox>, ... reasoning steps ... </think><answer>A</answer> > - Without CoT: <think> ... <answer> ... </answer>
• Scale: 156,000 QA pairs, 79% with chain-of-thought annotations.

The GCoT dataset is unique in explicitly embedding the grounding step as part of reasoning, facilitating direct supervision for spatial cognition.

5. Empirical Performance and Ablations

GS-Reasoner achieves SOTA or near-SOTA results on multiple 3D vision-language benchmarks:

• 3D Visual Grounding:
  • On ScanRefer, Acc@25: 60.8 vs. 61.1 (ROSS3D), with GS-Reasoner exceeding all other 3D LLMs that operate without external mesh proposals.
• Spatial Reasoning (VSI-Bench):
  • Achieves 64.7% (predicted depth) and 70.1% (ground-truth depth) average accuracy across 8 complex spatial reasoning tasks.
• General 3D Tasks:
  • Scan2Cap (dense captioning): CIDEr 101.0 (vs. 83.8 prior SOTA).
  • ScanQA/SQA3D: competitive, indicating no trade-off between 3D specialization and general vision-language capacity.

Ablation Studies (Sec. 5.4, 5.5):

Augmentation/Pooling Variant	Acc@25	Acc@50
LLaVA-NeXT (no geo, no pos enc.)	0	0
+ average pos enc.	53.2	29.8
+ max pooling	57.5	35.7
+ cross-attention only	58.9	38.6
+ sampling/interp only	59.3	40.2
Full Dual-Path	60.8	42.2

Grounded CoT supervision results in up to +8.4 points improvement on spatial tasks. Zero-shot variants retain high accuracy when transferred across datasets (e.g., ScanNet→ScanNet++/ARKitScenes), underscoring the representation's generality.

6. Limitations and Prospective Advances

Current limitations include noise introduced by the depth estimation pipeline (VGGT-SLAM + MoGe-2), and occasional overfitting to language bias on narrative-heavy QA tasks. GCoT's reliance on synthetic CoT (from GPT-4o) introduces possible hallucinations, partially mitigated through the use of BEV (bird's-eye view) maps. Prospective directions include:

• Improved depth/SfM pipelines for metric fidelity,
• Integrating reconstructive losses to enforce genuine 3D feature utilization,
• Expanding GCoT with real-world, human-annotated chains-of-thought,
• Application of GS-Reasoner as the reasoning and grounding module in embodied agents for planning and scene interaction,
• Joint finetuning with multimodal action data (Chen et al., 15 Oct 2025).

---

GS-Reasoner establishes a unified, end-to-end 3D LLM framework for visual grounding and spatial reasoning, setting new technical baselines for holistic vision-language intelligence grounded in real-world geometry (Chen et al., 15 Oct 2025).