Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views
Abstract: Though recent advances in vision-LLMs (VLMs) have achieved remarkable progress across a wide range of multimodal tasks, understanding 3D spatial relationships from limited views remains a significant challenge. Previous reasoning methods typically rely on pure text (e.g., topological cognitive maps) or on 2D visual cues. However, their limited representational capacity hinders performance in specific tasks that require 3D spatial imagination. To address this limitation, we propose 3DThinker, a framework that can effectively exploits the rich geometric information embedded within images while reasoning, like humans do. Our framework is the first to enable 3D mentaling during reasoning without any 3D prior input, and it does not rely on explicitly labeled 3D data for training. Specifically, our training consists of two stages. First, we perform supervised training to align the 3D latent generated by VLM while reasoning with that of a 3D foundation model (e.g., VGGT). Then, we optimize the entire reasoning trajectory solely based on outcome signals, thereby refining the underlying 3D mentaling. Extensive experiments across multiple benchmarks show that 3DThinker consistently outperforms strong baselines and offers a new perspective toward unifying 3D representations into multimodal reasoning. Our code will be available at https://github.com/zhangquanchen/3DThinker.
Summary
- The paper presents a novel framework, 3DThinker, that integrates 3D mental representations into VLMs using a two-stage training process.
- It employs supervised learning to align VLM-generated 3D latents with a 3D foundation model and reinforcement learning to refine the reasoning trajectory.
- Experimental results demonstrate improved spatial reasoning and generalization across benchmarks, enhancing applications in autonomous driving and embodied AI.
Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views
Introduction
The paper "Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views" (2510.18632) addresses a significant challenge in the field of Vision-LLMs (VLMs): the ability to understand and reason about 3D spatial relationships from limited visual inputs. Despite the remarkable advancements in VLMs, their capacity to perform spatial reasoning based on 3D geometry remains limited, often relying on textual or 2D visual inputs. The authors propose a novel framework, 3DThinker, designed to enable VLMs to form 3D mental representations during the reasoning process, thereby enhancing their spatial understanding capabilities.
Figure 1: Illustration of our 3DThinker. Existing methods typically perform reasoning based solely on pure text or 2D visual cues, without fully exploiting the rich spatial and geometric information inherent in images.
Methodology
The methodological foundation of 3DThinker lies in its two-stage training process. The first stage involves supervised learning that aligns 3D latent features generated by VLMs with those of a 3D foundation model, such as VGGT, using constructed Chain-of-Thought (CoT) data. This alignment facilitates the integration of 3D spatial reasoning into the model's native processing capabilities without requiring explicitly labeled 3D data. The second training stage employs outcome-driven reinforcement learning (RL) to fine-tune the entire reasoning trajectory, further refining the model's 3D mental representations based on feedback from the output.
Figure 2: The schematic illustration of our 3DThinker, a framework that enables thinking with 3D mentaling.
The integration of a projector is crucial in this process, as it transforms the VLM-generated 3D latent into the feature space compatible with VGGT, maintaining dimensional consistency throughout the transformation.
Figure 3: Illustration of our projector, which transforms VLM-generated 3D latent into the feature space of VGGT.
Experimental Results
The paper presents extensive experimental results across various benchmarks, demonstrating that 3DThinker consistently outperforms existing methods in spatial reasoning tasks. The framework shows impressive improvements across multiple benchmarks, achieving significant gains over the baseline models. The integration of 3D mental representations allows for enhanced interpretability and visualization, as the resulting 3D latents can be projected back into 3D space, providing a clear depiction of the model's reasoning process.
Figure 4: The reasoning process for different cases is presented, along with the visualization of the 3D latent representations.
The authors highlight the generalization capability of 3DThinker, as it performs well across different VLM architectures and parameter scales, further validating the robustness of the proposed framework.
Implications and Future Work
The introduction of 3DThinker carries significant implications for the development of VLMs capable of advanced spatial reasoning without reliance on external annotations or priors. This research opens up possibilities for the application of such models in fields requiring spatial intelligence, such as autonomous driving and embodied AI. Future work could explore the integration of autoregressive 3D latent tokens into the framework, enabling more seamless reasoning across multidimensional contexts. Additionally, iterative 3D mentaling could further enhance the model's ability to refine its understanding of complex spatial scenarios dynamically.
Conclusion
In conclusion, 3DThinker represents a significant step forward in the integration of 3D mental representations into vision-language reasoning. By enabling VLMs to imagine and reason with 3D scenarios using limited views, this framework not only enhances the current capabilities of multimodal models but also sets a foundation for future innovations in spatial intelligence. Its ability to operate without annotated 3D input and external tools underscores its potential for widespread applicability across diverse domains.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper introduces 3DThinker, a new way to help AI models “imagine” 3D spaces from just a few 2D pictures and use that imagination to reason about where things are. The goal is to make vision–LLMs (VLMs)—AI systems that look at images and read text—better at understanding 3D layouts without needing special 3D inputs or extra tools during use.
Key Objectives
The paper focuses on three simple goals:
- Teach AI to picture 3D scenes in its “mind” using only a few normal images.
- Avoid expensive, detailed 3D labels or extra sensors (like depth maps) during training.
- Make this 3D “thinking” an internal skill of the model, so it doesn’t rely on extra tools when answering questions.
Methods and Approach
Think of the model as a student learning to visualize a room from a few photos and then explain the layout. The approach has two main stages:
- Stage 1: Supervised alignment (learning with a teacher)
- The model writes step-by-step reasoning using Chain-of-Thought (CoT). Inside this reasoning, it inserts special placeholders called “3D tokens.” These act like invisible sticky notes where the model stores a compact “mental 3D sketch.”
- A strong 3D expert model (VGGT, a “3D foundation model”) provides high-quality 3D features from the same images.
- A “projector” (like a translator) maps the model’s hidden 3D tokens into the same feature space as the 3D expert. The model’s 3D mental sketch is trained to align (match) the expert’s features.
- At the same time, the model is trained to keep its text reasoning clear and correct, so the 3D tokens don’t break the flow of its explanation.
- Stage 2: Reinforcement learning (learning by trial and reward)
- After the initial alignment, the model improves further by receiving rewards:
- A format reward: ensuring the output follows the right structure (e.g., includes the 3D tokens and final answer tags).
- An answer reward: giving points if the final answer is correct.
- A 3D token reward: encouraging the hidden 3D tokens to stay close to the expert’s 3D features.
- This is like grading each attempt and nudging the model to produce better 3D imagination and reasoning over time, using outcomes only (no extra 3D labels).
Helpful explanations of technical terms:
- Vision–LLM (VLM): An AI that looks at images and reads text together.
- 3D tokens: Hidden “slots” inside the model’s thoughts where it stores a compact 3D representation.
- Latent: A hidden numeric representation (not visible images) that captures important information (like a mental sketch).
- Projector: A small network that converts the model’s hidden 3D tokens to match the 3D expert’s features.
- VGGT: A strong 3D model used as a teacher during training.
- Reinforcement Learning: Training by trial and error, guided by rewards.
Main Findings and Why They Matter
- Better 3D reasoning from limited views: Across several benchmarks (like MindCube and Ego3D), 3DThinker consistently beats strong baseline models that only use text or 2D cues.
- Works with many base models: The approach improves different VLM families (Qwen2.5-VL, InternVL3, LLaVA-OneVision), showing it’s broadly useful.
- No extra tools at test time: Unlike other methods, 3DThinker doesn’t need depth maps, 3D tokens from separate models, or camera parameters during inference.
- More interpretable: Because the hidden 3D tokens can be “projected back” into 3D representations (like point clouds), you can better understand what the model is imagining as it reasons.
In short, the model not only answers spatial questions more accurately but also shows what it’s “thinking” in 3D, which builds trust and makes debugging easier.
Implications and Potential Impact
3DThinker points to a future where AI can:
- Help robots and self-driving cars reason about space from simple camera views.
- Assist in AR/VR, mapping, and indoor navigation by imagining full scenes from a few photos.
- Reduce the need for costly 3D labels or special sensors, making 3D reasoning more accessible.
- Unite 3D understanding with language reasoning, making multimodal AI more powerful and human-like in spatial thinking.
Overall, this research shows a practical path to teach AI to “think in 3D” naturally—much like people do—using ordinary images and smart training, without heavy extra data or tools.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper leaves several aspects underexplored or insufficiently validated. Future research can address the following concrete gaps:
- Training-time dependence on a specific 3D foundation model: The approach hinges on aligning to VGGT features during both supervised and RL stages. It remains unclear how performance and learned latents change if a different 3D foundation model (e.g., DUSt3R, CUT3R, depth/normal estimators) is used, or under domain shifts where VGGT underperforms (e.g., outdoor vs. indoor, low-texture scenes).
- Lack of metric and geometric guarantees: The 3D latents are only aligned to a feature space (via Frobenius loss and cosine similarity), not to metric geometry. There is no explicit constraint on scale, depth ordering, or rigid consistency across views. It is unknown whether the latents encode metrically accurate or even monotonic depth relationships.
- Minimal-view requirements and view-count sensitivity: The method claims to work from “limited views,” but the minimal number of views, camera configuration assumptions, and performance degradation with fewer views are not quantified. Single-image performance and robustness to unknown intrinsics/extrinsics are not reported.
- Generalization beyond the two benchmarks: Experiments are limited to MindCube-Tiny and Ego3D-Bench. Transfer to other spatial QA datasets, real-world photos, different domains (e.g., outdoor driving, cluttered indoor scenes), or downstream tasks (navigation, manipulation, 6DoF pose queries) is not evaluated.
- Quantitative evaluation of “recoverable” 3D interpretability: Although the paper claims point clouds can be recovered from latents via the projector, there is no quantitative assessment (e.g., Chamfer distance, normal consistency, depth ordering accuracy) or qualitative analysis of 3D reconstructions and their stability across reasoning steps.
- RL reward design sparsity and stability: The outcome reward is mostly 0/1 (format and final answer), with a cosine similarity term to VGGT features. There is no study of training stability, sample efficiency, or credit assignment to 3D tokens. Ablations on the contribution of each reward component are missing.
- Projector design and optimization: The projector architecture is introduced but not ablated (e.g., capacity, conditioning on image features, alternative cross-attention designs). Freezing the projector in RL may limit improvements to 3D alignment; the trade-off between freezing vs. fine-tuning is not studied.
- Latent token budget and structure: The number and arrangement of 3D special tokens (), their dimensionality, and sensitivity to token budget are not explored. Guidelines for choosing and impacts on performance and compute are missing.
- Computational footprint and latency: Training costs (VGGT forward passes, RL sampling) and inference overhead from longer token sequences are not reported. Real-time feasibility and memory/latency trade-offs remain unclear.
- Robustness under hard visual conditions: The method’s behavior under low resolution, motion blur, heavy occlusions, extreme viewpoints, textureless surfaces, or lighting variations is not evaluated, despite claims that prior methods fail under such conditions.
- Camera geometry handling: Without camera parameters, it is unclear how viewpoint changes are internally handled or whether latents remain consistent under ego-motion. Temporal/video consistency and cross-frame 3D coherence are not examined.
- Failure modes and uncertainty: The paper does not analyze typical failure cases (e.g., symmetric ambiguities, occluded relations, near/far confusion) or provide mechanisms for uncertainty estimation, confidence calibration, or detection of unreliable “3D mentaling.”
- Dependence on GPT-4o-generated CoT: The quality, biases, and reproducibility of the GPT-4o-generated reasoning chains (with 3D placeholders) are not assessed. Sensitivity to prompt templates and the feasibility of replicating the data generation pipeline are not discussed.
- Rewarding open-ended outputs: The RL framework assumes discrete correctness (e.g., multiple-choice). How to define outcome rewards for open-ended spatial descriptions or continuous metrics is left open.
- Semantic vs. geometric disentanglement: It is unclear whether the aligned latents mostly capture semantics correlated with VGGT features rather than geometry. Controlled tests for viewpoint invariance and geometry-only cues (e.g., textureless scenes) are missing.
- Comparisons to close latent-reasoning baselines: Direct, controlled comparisons against Mirage/LVR under identical settings (same datasets, views, and prompts) are absent, making it hard to attribute gains specifically to 3D mental tokens vs. other factors.
- Cross-lingual generalization: The pipeline and evaluations appear English-centric. Whether the 3D mentaling and formatting generalize across languages is not tested.
- User-facing formatting constraints: The strict requirement for special tags (<|latent_start|> …) may be brittle. At inference, how these tags are handled in user-facing outputs and whether deviations degrade performance are not clarified.
- Scaling laws and data efficiency: The effect of training data size (beyond MindCube’s 10K), quality of multi-view coverage, and the presence/absence of diverse scenes on learned 3D latents is not characterized.
- Biases inherited from VGGT and datasets: Potential propagation of biases from VGGT’s training data and from GPT-4o-generated rationales is not analyzed, nor are mitigations proposed.
- Safety for embodied use: For embodied decision-making, erroneous 3D mentaling could be hazardous. No safety constraints, fail-safes, or interventions are proposed when the model’s 3D inference is uncertain or inconsistent.
Practical Applications
Immediate Applications
Below are concrete use cases that can be deployed with modest engineering effort, leveraging the paper’s core contributions: intrinsic “3D mentaling” from monocular or few-view images, no external priors/tools at inference, and point‑cloud–like interpretability via the latent projector.
- 3D‑aware visual question answering (VQA) and scene Q&A from photos
- Sectors: software, consumer apps, education, real estate
- What it enables: Answering spatial queries (e.g., “Is the chair behind the table?”, “Which object is closer to the door?”) from one or a few images without depth or camera metadata.
- Potential tools/workflows: “3D-CoT” inference mode in VLM chatbots; developer SDK that exposes 3D special tokens and latent-to-point-cloud preview; spatial Q&A widgets in listing platforms.
- Assumptions/dependencies: Trained base VLMs adapted to target domains; scale ambiguity may limit absolute distance estimates; works best on scenes similar to training data and with reasonable lighting/visibility.
- Warehouse and home robotics: monocular spatial reasoning for pick-and-place and navigation
- Sectors: robotics, logistics, smart home
- What it enables: Planning and error-checking from limited RGB views (e.g., “Is the target box reachable without collision?”), improving grasp/placement instructions when depth sensors are unavailable.
- Potential tools/workflows: A “spatial-latent planner” module in robot stacks that queries a 3DThinker‑augmented VLM; visual checklists during execution; lightweight on-robot reasoning without external tools.
- Assumptions/dependencies: Requires careful domain finetuning; not a replacement for motion planners or safety sensors; real-time constraints depend on model size and decoding speed.
- Drone/asset inspections with limited views
- Sectors: energy, infrastructure, agriculture
- What it enables: Spatial relationship reasoning for defects or components from few aerial images (e.g., “Is the cable behind the insulator?”), assisting operators in low-bandwidth environments.
- Potential tools/workflows: On-laptop or on-edge VQA assistant for inspection crews; post‑hoc reporting enriched with spatial relations and latent point-cloud snapshots.
- Assumptions/dependencies: Accuracy affected by occlusions, scale ambiguity, and domain shift (e.g., reflective surfaces, high dynamic range scenes).
- AR‑aided interior layout and furnishing guidance from smartphone photos
- Sectors: consumer AR/VR, e-commerce, home improvement
- What it enables: Advising furniture placement and clearances (“Can this sofa fit between the shelves?”) using imagined 3D relations from photos.
- Potential tools/workflows: Mobile app plugin that answers spatial fit/placement queries; showroom assistants linking catalog items with user room photos.
- Assumptions/dependencies: No guaranteed metric scale without calibration; best for relative comparisons and qualitative guidance.
- Accessibility: spatial descriptions and object localization for blind/low-vision users
- Sectors: assistive technology, public services
- What it enables: Clear relational language (“The mug is to your left, behind the laptop”) from a single photo or brief video snippet.
- Potential tools/workflows: Smartphone assistant that describes spatial layouts and suggests next steps (e.g., “Move 20 cm left to reach the handle” qualitatively).
- Assumptions/dependencies: Must handle safety-critical caveats (avoid overconfident commands); robust UX for uncertainty communication; privacy and on-device processing considerations.
- Real estate and construction pre‑analysis from listing/site images
- Sectors: real estate, AEC (architecture/engineering/construction)
- What it enables: Lightweight floor‑space Q&A and qualitative relationships (e.g., “Does the kitchen island block the oven door swing?”) without full scans.
- Potential tools/workflows: Agent add-ons that answer spatial RFP questions; pre‑walkthrough checklists; marketing material enrichment with spatial relations.
- Assumptions/dependencies: Not a substitute for full measurements or BIM; lighting and lens distortion can affect reliability.
- Security/surveillance operators’ assistant
- Sectors: security, public safety
- What it enables: Rapid spatial queries (“Is the person concealed behind the SUV?”, “Line of sight between cameras A and B?”) from monocular feeds.
- Potential tools/workflows: Console‑integrated Q&A agent that returns a textual answer plus the recovered latent point-cloud for audit.
- Assumptions/dependencies: Requires thoughtful human-in-the-loop; performance may degrade with low-light/noisy feeds.
- E‑commerce and warehouse product arrangement planning
- Sectors: retail, logistics
- What it enables: Advising shelf/aisle placements and access constraints from snapshot images; relative clearance checks without depth.
- Potential tools/workflows: Staff mobile assistant to verify layout constraints; slotting optimization tools that query spatial relations.
- Assumptions/dependencies: Domain-specific finetuning; does not replace full 3D scans when exact measurements are required.
- Model auditing and compliance via latent‑to‑3D interpretability
- Sectors: AI governance, enterprise software
- What it enables: Inspectable 3D latent reconstructions (e.g., point clouds) from the reasoning process to validate whether decisions align with perceived geometry.
- Potential tools/workflows: “Reasoning Inspector” dashboards that display 3D latents alongside answers; red‑team tools that probe for geometric hallucinations.
- Assumptions/dependencies: The projector’s fidelity constrains interpretability; still not a ground-truth 3D reconstruction.
- Research tooling for spatial cognition with fewer labels
- Sectors: academia, R&D labs
- What it enables: Training and benchmarking VLMs for spatial tasks without dense 3D annotations; studying how 3D latents evolve during CoT.
- Potential tools/workflows: Open-source pipelines: Stage‑1 distillation from a 3D foundation model (e.g., VGGT) + Stage‑2 outcome‑driven RL; benchmark harnesses (MindCube/Ego3D).
- Assumptions/dependencies: Access to pre-trained 3D foundation models during training; sufficient compute for supervised + RL stages; licensing for data/models (e.g., GPT‑4o prompts for CoT synthesis).
Long‑Term Applications
These opportunities require further research, scaling, domain adaptation, or reliability guarantees (e.g., calibration, metric scale, regulatory compliance) before deployment in high‑stakes settings.
- Sensor‑light mobile robots that reason with internal 3D mental models
- Sectors: robotics (service, household, intralogistics)
- Potential: Robust navigation/manipulation in cluttered, dynamic environments using few monocular views, reducing dependency on LiDAR or dense depth.
- Dependencies/risks: Safety certification, closed-loop integration with planners and control; handling dynamic occlusions; latency and compute on edge hardware.
- Autonomous driving and ADAS incident analysis from dashcams
- Sectors: automotive
- Potential: Spatial Q&A for edge cases, scene‑understanding auditors, post‑hoc incident triage where only camera footage exists.
- Dependencies/risks: Regulatory constraints; metric scale and calibration challenges; cannot replace sensor fusion for driving decisions without extensive validation.
- Live AR guidance for indoor navigation and spatial tutoring
- Sectors: AR/VR, education, accessibility
- Potential: On‑glasses spatial assistance (“turn right at the second doorway”) and geometry tutoring with real‑time 3D mentaling from a single forward camera.
- Dependencies/risks: On‑device inference efficiency; accurate temporal consistency; user safety and fatigue; privacy.
- Few‑view digital twin bootstrapping
- Sectors: AEC, facilities management, retail
- Potential: Rapid, approximate digital twins initialized from sparse images, refined later by scans—useful for early planning or inventory.
- Dependencies/risks: Metric ambiguity and geometry gaps; workflows for progressive refinement and uncertainty tagging.
- Multi‑agent collaboration via shared 3D latent spaces
- Sectors: robotics, defense, search and rescue
- Potential: Teams of agents exchanging compact 3D mental tokens instead of full maps to coordinate tasks in bandwidth‑constrained settings.
- Dependencies/risks: Standardization of token formats; synchronization and drift handling; robust cross‑agent alignment.
- Medical imaging triage assistants from limited 2D views (exploratory)
- Sectors: healthcare
- Potential: Reasoning about 3D anatomical relations from sparse 2D projections (e.g., ultrasound slices) to assist clinicians in low‑resource settings.
- Dependencies/risks: Requires domain‑specific training and clinical validation; regulatory approval; ethical and safety constraints; may be unsuitable without modality‑specific geometry priors.
- Construction progress monitoring and as‑built verification from casual photos
- Sectors: AEC, infrastructure
- Potential: Detecting spatial deviations (“beam installed too close to conduit”) with approximate 3D relations before full scans are available.
- Dependencies/risks: Lighting/occlusion variability; metric tolerance requirements; integration with BIM workflows.
- Insurance claims and risk assessment from claimant photos
- Sectors: insurance, finance
- Potential: Estimating relative extents, obstructions, or object relations to support adjusters when site scans are absent.
- Dependencies/risks: Need for human verification; fraud risk; varying photo quality.
- Game AI and embodied NPCs with 3D‑aware cognition from screen frames
- Sectors: gaming, simulation
- Potential: NPCs that infer occlusions/cover and spatial tactics from 2D frames, improving realism without full engine access.
- Dependencies/risks: Tight coupling with render pipelines; latency budgets; domain adaptation to stylized graphics.
- Cross‑modal world models that unify 2D, 3D, and language reasoning
- Sectors: foundational AI research
- Potential: Extending latent‑alignment methods to integrate depth, audio, or tactile signals, creating generalist agents that “think in 3D” across tasks.
- Dependencies/risks: Scaling laws, training stability for multi‑latent trajectories, and new evaluation standards.
Common Assumptions and Dependencies Affecting Feasibility
- Training relies on a 3D foundation model (e.g., VGGT) to distill geometric features; inference does not, but domain shift from training data can reduce reliability.
- The approach handles relative spatial relationships well; absolute metric accuracy is limited without camera intrinsics or additional priors.
- Quality of “3D‑CoT” data (often synthesized via GPT‑4o prompts) and reward design in RL critically affects performance; licensing and privacy of training images must be respected.
- Real‑time deployments depend on model size, decoding speed, and hardware (edge vs. cloud).
- Interpretability via the projector aids auditability but is not equivalent to ground‑truth 3D reconstruction; uncertainty communication is essential in high‑stakes uses.
Glossary
- 3D cognitive tokens: Placeholder tokens representing imagined 3D scenes used inside the model’s reasoning. "which operationalize the 3D cognitive tokens "
- 3D foundation model: A large pretrained model that provides general-purpose 3D geometric features for alignment or supervision. "3D foundation model (e.g., VGGT)"
- 3D latent: A compact internal embedding the model generates to represent 3D geometry during reasoning. "align the 3D latent generated by VLM while reasoning with that of a 3D foundation model"
- 3D mentaling: The paper’s term for internally imagining 3D scenes during reasoning without explicit 3D inputs. "our framework is the first to enable 3D mentaling during reasoning"
- 3D special tokens: Special tokens inserted into the reasoning chain to carry 3D latent information. "Chain-of-Thought (CoT) data that incorporates 3D special tokens"
- 3D visual token: The specific latent token sequence that encodes 3D visual information during the trajectory. "To further optimize the 3D visual token in the reasoning process"
- Aggregator: A component (in VGGT) that combines features across inputs or layers to produce final geometry features. "through the last layer of VGGT aggregator"
- Auxiliary modalities: Additional input types beyond RGB images that provide geometric cues. "incorporates auxiliary modalities as additional inputs (e.g., point clouds, camera parameters)"
- Bird’s-eye-view (BEV): A top-down 2D representation of a 3D scene layout. "bird’s-eye-vie (BEV) annotations"
- Camera parameters: Intrinsic and extrinsic calibration data used to relate images to 3D geometry. "point clouds, camera parameters"
- Chain-of-Thought (CoT): Step-by-step reasoning traces used to guide and align the model’s intermediate computations. "Chain-of-Thought (CoT) data"
- Cognitive maps: Abstract spatial representations of environments used for reasoning about layout and relations. "train models to generate cognitive maps of 3D layouts"
- Cosine similarity: A metric to measure alignment between two feature vectors by comparing their orientation. "the cosine similarity between the VGGT features and the projected features is calculated"
- Cross-entropy loss: A standard classification loss used to train correct token predictions and textual coherence. "we employ both 3D latent alignment loss and the cross-entropy loss"
- Depth estimation: Predicting per-pixel distance from the camera to surfaces in the scene. "DepthAnythingv2 for depth estimation"
- DepthAnywherev2: A model used to automatically estimate depth from images. "DepthAnythingv2 for depth estimation"
- Ego-centric: Viewpoints tied to the agent’s own position and orientation. "ego-centric, multi-view observations"
- Extrinsic tool-usage: Methods that call external tools during reasoning to augment visual processing. "Other extrinsic tool-usage methods"
- Feature distillation: Transferring representational knowledge from a teacher/model into another model’s tokens. "distill its features to the 3D special token generated within the VLM reasoning process"
- Frame of reference: The coordinate system relative to the agent that grounds spatial semantics. "spatial semantics tied to the machine’s frame of reference"
- Frobenius loss: A matrix norm-based loss used to align projected 3D latents with geometry features. "we formulate the 3D alignment as the Frobenius loss"
- GRPO (group-relative policy optimisation): A reinforcement learning algorithm that normalizes advantages within sampled groups. "group-relative policy optimisation (GRPO)"
- GroundingDINO: A model for grounding text in images, used here for referring expression tasks. "GroundingDINO for referring expression comprehension (REC)"
- Group-normalized advantage: An RL advantage computed relative to a group’s mean and variance at each step. "The group-normalized advantage, denoted as , is calculated"
- Hidden states: Internal layer activations of the model that carry information for subsequent token generation. "extracted from the last layer hidden states of VLM"
- In-context learning: Improving performance by conditioning on examples and rationales within the prompt. "in-context learning, including intermediate rationales, can significantly enhance the performance of LLMs"
- KL divergence: A regularization term measuring the difference between the current policy and a reference policy. "represents the KL divergence"
- Latent reasoning: Reasoning that operates in continuous embedding spaces instead of only discrete text. "Latent reasoning: some studies have shown that incorporating intermediate hidden representations"
- Mirage: A prior framework that supervises visual latent generation to enable reasoning with images. "Mirage~\cite{mirage} and LVR~\cite{lvr} utilize special visual tokens alongside ordinary text during reasoning"
- Monocular images: Single-camera images lacking stereo or multi-view geometric input. "real-world scenarios where only monocular images are available."
- Multi-hop methods: Approaches that perform iterative, multi-step reasoning with interleaved visual cues. "Some intrinsic multi-hop methods"
- Multimodal trajectory: A sequence of interleaved text and latent/visual tokens produced during reasoning. "facilitating the continuation of a multimodal trajectory"
- Outcome-driven signals: Rewards based solely on final outcomes used to optimize the entire reasoning trajectory. "optimizing the tokens across the entire sampling trajectory based solely on outcome-driven signals"
- Patch-level visual features: Local image descriptors extracted from patches for downstream geometric alignment. "patch-level visual features"
- Point clouds: Sets of 3D points representing scene geometry. "point clouds, camera parameters"
- Projector: A learned module that maps VLM latents into the geometry feature space for alignment. "we employ the projector as illustrated in Fig.~\ref{fig:projector}"
- Referring Expression Comprehension (REC): Understanding and grounding text phrases that refer to specific objects in images. "referring expression comprehension (REC)"
- Reinforcement learning (RL): Training via rewards to optimize policies over sampled trajectories. "We formalize the RL framework as follows."
- Spatial semantics: Meaning about spatial relations embedded in views relative to an agent’s coordinates. "they inherently carry spatial semantics tied to the machine’s frame of reference"
- Supervised learning: Training with labeled signals to teach desired behavior or align features. "we perform supervised learning"
- Trajectory: The full sequence of tokens (text and latent) generated during reasoning. "the reasoning trajectory can be decomposed"
- VGGT: A 3D foundation model whose features are used as geometry supervision targets. "VGGT~\cite{wang2025vggt}"
- Vision–LLMs (VLMs): Models that jointly process images and text for multimodal tasks. "vision–LLMs (VLMs)"
- Visual tokens: Non-text tokens inside reasoning that carry visual/latent information. "utilize special visual tokens alongside ordinary text during reasoning"
