SD-VLM: Spatial Reasoning & Self-Driving VLM

• SD-VLM is a multimodal framework that combines explicit depth positional encoding with vision-language models to enhance 3D spatial reasoning.
• It integrates dual-system architectures where high-level VLM decisions guide low-level diffusion-based planning for improved self-driving trajectories.
• The approach highlights the importance of structured intermediate representations to tackle the limitations of conventional end-to-end VLM systems.

Searching arXiv for the cited SD-VLM-related papers to ground the article in current records. SD-VLM is an overloaded designation in recent arXiv literature. In one usage, it denotes a vision-LLM framework for quantitative 3D spatial reasoning that combines a large-scale spatial dataset with depth positional encoding to improve metric understanding from RGB imagery (Chen et al., 22 Sep 2025). In another usage, the same label is applied in the provided summary to a self-driving vision-LLM system derived from Senna-2, where a vision-LLM supplies high-level driving decisions to an end-to-end diffusion planner through an explicit decision adapter and a consistency-oriented training routine (Song et al., 11 Mar 2026). A related but distinct line, SD-VLA, transfers the “SD” prefix to static-dynamic disentanglement for efficient long-horizon vision-language-action modeling (Qiu et al., 3 Feb 2026). Taken together, these works place SD-VLM at the intersection of multimodal representation learning, spatial reasoning, and embodied decision-making, while also illustrating that the acronym does not yet have a single stabilized meaning across subfields.

1. Terminological scope and research context

The title “SD-VLM” is used explicitly for “Spatial Measuring and Understanding with Depth-Encoded Vision-LLMs,” a framework motivated by the observation that vision-LLMs excel in 2D semantic visual understanding but remain weak at quantitatively reasoning about 3D spatial relationships because 2D images have limited spatial representation ability (Chen et al., 22 Sep 2025). The problem statement in that work is concrete: typical VLMs ingest only a 2D projection of the world, so depth, scale and camera intrinsics are lost in the imaging process, and existing 2D VLMs therefore lack precise spatial priors and training data with ground-truth metric labels (Chen et al., 22 Sep 2025).

The provided material also applies “SD-VLM” to a self-driving VLM system inspired by Senna-2. In that formulation, the system is organized as a dual-system architecture in which Qwen2.5-VL-3B produces a structured meta-action and an end-to-end driving policy generates trajectories conditioned on both visual features and VLM-derived decision embeddings (Song et al., 11 Mar 2026). This usage is conceptually distinct from the spatial-measurement framework, even though both emphasize explicit structure for bridging semantic reasoning and downstream geometric or control tasks.

A third nearby acronym, SD-VLA, refers to “Static-Dynamic Disentanglement” in long-horizon vision-language-action models rather than to spatial depth encoding or self-driving decision alignment (Qiu et al., 3 Feb 2026). This suggests that “SD” currently functions more as a family resemblance marker than as a canonical taxonomy. A plausible implication is that careful disambiguation is necessary when citing or comparing “SD-VLM” systems across robotics, autonomous driving, and spatial VQA.

2. Spatial measuring and understanding with depth-encoded VLMs

The spatial-reasoning SD-VLM is designed to address quantitative 3D reasoning tasks such as exact dimensions, metric separation, and reference-based estimation from images (Chen et al., 22 Sep 2025). Its central claim is that explicit depth-aware tokenization is more effective than relying on plain RGB inputs or ad hoc depth injection schemes.

The framework introduces the Massive Spatial Measuring and Understanding (MSMU) dataset. MSMU is built from 2,000 real-world 3D scenes from ScanNet/ScanNet++, from which 25,000 RGB images are rendered; it contains 700,000 spatial VQA pairs, 2.5 million distinct numerical annotations, and 10,000 chain-of-thought augmented samples (Chen et al., 22 Sep 2025). The task coverage is broad but still tightly specified: Scale Estimation, Object Grounding, Existence Recognition, Counting, Absolute Distance Measurement, Relative Position, Scale Comparison, and Reference-Object Estimation (Chen et al., 22 Sep 2025). MSMU-Bench is a held-out benchmark of approximately 1,000 QA pairs from unseen scenes for quantitative 3D reasoning evaluation (Chen et al., 22 Sep 2025).

The model architecture is built on LLaVA-1.5-7B, with a frozen CLIP-ViT/14 vision encoder, a depth estimation model at inference time when ground-truth depth is unavailable, a lightweight depth positional encoding module inserted after the vision encoder and before tokenization, and LoRA adapters on the 7B-parameter LLM for efficient fine-tuning (Chen et al., 22 Sep 2025). The training setup is also explicit: one epoch of fine-tuning on 8×V100 GPUs, batch size 2/GPU, about 32 GPU-hours, AdamW with learning rates 2e-4 for the LLM adapters and 2e-5 for the visual projector, and standard cross-entropy on next-token prediction for both answer tokens and reasoning tokens when chain-of-thought is used (Chen et al., 22 Sep 2025).

The key architectural novelty is Depth Positional Encoding (DPE), described as a direct analogue of the 2D sinusoidal positional embeddings in Transformers, extended to the depth dimension. Given an RGB image $\mathcal{I}\in\mathbb{R}^{H\times W\times 3}$ and depth map $\mathbf{D}\in\mathbb{R}^{H\times W}$, adaptive mean-pooling produces a patchwise depth map $\mathbf{D}'\in\mathbb{R}^{H'\times W'}$, and sinusoidal embeddings are then computed from $\mathbf{D}'$ and added to the image features before flattening and input to the LLM (Chen et al., 22 Sep 2025). In the formulation provided:

$$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$

$$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$

$$\mathbf{E}^{\text{depth}}(i,j,2t+1)=\cos\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$

and the fused features are

$$\mathbf{E}^{\text{vision}}=\mathbf{E}^{\text{image}}+\mathbf{E}^{\text{depth}}.$$

The stated purpose is to inject an explicit “z-axis” embedding into each spatial token so that the Transformer can reason about absolute scales and distances rather than only 2D layouts (Chen et al., 22 Sep 2025).

3. Empirical performance in quantitative spatial reasoning

On MSMU-Bench, evaluation uses GPT-4 scoring, with qualitative answers receiving a 0/1 mark and quantitative answers judged correct if

$\max\bigl(\hat d/d^*,\,d^*/\hat d\bigr)\le 1.25.$

Under this protocol, SD-VLM achieves an average success rate of 56.31% (Chen et al., 22 Sep 2025). The reported task-level results include Existence at 87.23%, Scale Estimation at 51.35%, Absolute Distance at 40.00%, and Reference Object Estimation at 55.66% (Chen et al., 22 Sep 2025). The same source reports that SD-VLM outperforms GPT-4o and Intern-VL3-78B by 26.91% and 25.56% respectively on MSMU-Bench, with baseline averages of 32.28% for GPT-4o and 33.63% for Intern-VL3-78B (Chen et al., 22 Sep 2025).

Incorporating MSMU-CoT raises the average success rate to 59.19%, with a notable improvement on scale comparison from 55.66% to 69.81% (Chen et al., 22 Sep 2025). External-benchmark performance is also reported: 56.2% success on Q-Spatial++, 33.3% on the quantitative portion of SpatialRGPT-Bench, and 65.5% on the qualitative portion of SpatialRGPT-Bench (Chen et al., 22 Sep 2025).

Ablation results are central to the paper’s argument that the method depends specifically on how depth is integrated. On MSMU-Bench, the baseline without depth reaches 46.73%, “Depth as image” falls to 22.64%, “Depth as prompt” reaches 48.78%, “Depth as token” reaches 35.72%, and DPE in either learnable or sinusoidal/cosine form reaches about 56.2% (Chen et al., 22 Sep 2025). The result that “Depth as image” is harmful is especially important because it argues against treating depth as a generic auxiliary channel. Further studies report only an approximately 5% drop under high Gaussian perturbations and robustness to the choice of depth estimator, including UniDepth and Depth-Anything (Chen et al., 22 Sep 2025).

A representative chain-of-thought example concerns reference-object estimation: given that a blue chair has height 0.7 m, the model reasons that a wooden table appears about twice as tall and concludes that the table height is approximately 1.4 m (Chen et al., 22 Sep 2025). The example is presented to illustrate coherent multi-step reasoning in queries involving scale and reference transfer rather than merely single-shot numeric guessing.

4. Self-driving SD-VLM as a dual-system VLM-policy architecture

In the self-driving usage supplied in the data, SD-VLM is a system inspired by Senna-2 and grounded in the problem of dual-system consistency between a VLM’s high-level decision and an end-to-end driving policy’s low-level planning (Song et al., 11 Mar 2026). The underlying motivation is that generated trajectories may misalign with intended driving decisions, weakening top-down guidance and decision-following ability (Song et al., 11 Mar 2026).

The architecture contains two principal components. The vision-language backbone adopts Qwen2.5-VL-3B as the high-level reasoning engine, taking as input a single front-view RGB frame $I$ and a text prompt $\mathbf{D}\in\mathbb{R}^{H\times W}$0 composed of system prompt, navigation command, and ego speed (Song et al., 11 Mar 2026). Its output is a structured meta-action $\mathbf{D}\in\mathbb{R}^{H\times W}$1, where $\mathbf{D}\in\mathbb{R}^{H\times W}$2 and $\mathbf{D}\in\mathbb{R}^{H\times W}$3 (Song et al., 11 Mar 2026). The stage-1 VLM objective is

$\mathbf{D}\in\mathbb{R}^{H\times W}$4

The end-to-end driving policy consists of a spatio-temporal encoder over multi-view image sequences producing features $\mathbf{D}\in\mathbb{R}^{H\times W}$5, perception heads predicting semantic maps, dynamic agents, and navigation cues, and a DiT-based diffusion planner that generates a residual trajectory $\mathbf{D}\in\mathbb{R}^{H\times W}$6 via denoising (Song et al., 11 Mar 2026). Conditioning is performed by cross-attention and AdaLN, combining $\mathbf{D}\in\mathbb{R}^{H\times W}$7 with VLM-derived features $\mathbf{D}\in\mathbb{R}^{H\times W}$8 (Song et al., 11 Mar 2026). Its stage-1 objective is the standard denoising loss

$\mathbf{D}\in\mathbb{R}^{H\times W}$9

where $\mathbf{D}'\in\mathbb{R}^{H'\times W'}$0 is the expert residual, $\mathbf{D}'\in\mathbb{R}^{H'\times W'}$1 the noisy sample, $\mathbf{D}'\in\mathbb{R}^{H'\times W'}$2 the predicted noise, and $\mathbf{D}'\in\mathbb{R}^{H'\times W'}$3 (Song et al., 11 Mar 2026).

A decision adapter bridges the VLM’s discrete meta-action and the planner’s continuous latent space. It uses a VLM hidden-state token $\mathbf{D}'\in\mathbb{R}^{H'\times W'}$4 together with learnable embeddings $\mathbf{D}'\in\mathbb{R}^{H'\times W'}$5 and $\mathbf{D}'\in\mathbb{R}^{H'\times W'}$6, then fuses them as

$\mathbf{D}'\in\mathbb{R}^{H'\times W'}$7

The overall information flow is summarized as

$\mathbf{D}'\in\mathbb{R}^{H'\times W'}$8

$\mathbf{D}'\in\mathbb{R}^{H'\times W'}$9

$\mathbf{D}'$0

The design intent is stated directly: the VLM supplies interpretable high-level commands, the adapter embeds them as continuous “soft prompts,” and AdaLN injection globally modulates trajectory generation (Song et al., 11 Mar 2026).

5. Three-stage alignment paradigm in self-driving SD-VLM

The self-driving SD-VLM is trained through a consistency-oriented three-stage paradigm (Song et al., 11 Mar 2026). Stage 1 is driving pre-training. The VLM is trained in QA style on $\mathbf{D}'$1 using 360M frames from approximately 10k hours of real driving; the end-to-end policy is pre-trained for diffusion-based planning on expert trajectories; and the adapter is pre-trained by freezing the VLM and jointly fine-tuning the adapter and E2E policy with $\mathbf{D}'$2 only (Song et al., 11 Mar 2026).

Stage 2 is open-loop alignment. A consistency indicator compares the VLM decision $\mathbf{D}'$3 and planner output $\mathbf{D}'$4:

$\mathbf{D}'$5

where $\mathbf{D}'$6 is a kinematic mapping that discretizes $\mathbf{D}'$7 into direction and speed (Song et al., 11 Mar 2026). The alignment loss is

$\mathbf{D}'$8

with $\mathbf{D}'$9 set to $$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$0 (Song et al., 11 Mar 2026). The rationale is to back-propagate only on samples where $$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$1 and $$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$2 disagree, thereby enforcing decision-planning alignment via external supervision (Song et al., 11 Mar 2026).

Stage 3 is closed-loop alignment via hierarchical reinforcement learning in 3DGS environments. The environment construction uses 1,300 high-risk clips converted into 3DGS simulation, with 256 clips reserved for evaluation (Song et al., 11 Mar 2026). The low-level planner is updated using safety and efficiency rewards:

$$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$3

$$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$4

$$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$5

The high-level decision is then updated with

$$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$6

when $$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$7, and the combined stage-3 loss is

$$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$8

with $$\mathbf{D}'(i,j)=\tfrac{1}{|\mathcal{P}_{ij}|}\sum_{(u,v)\in\mathcal{P}_{ij}}\mathbf{D}(u,v),$$9 (Song et al., 11 Mar 2026).

The reported quantitative outcomes are specific: average decision-planning consistency $$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$0 of 0.760 versus 0.637 for Senna, corresponding to a 19.3% improvement; open-loop FDE of 0.597 m versus 0.633 m, corresponding to a 5.7% reduction; and at-fault collision rate of 0.077 versus 0.111, corresponding to a 30.6% reduction (Song et al., 11 Mar 2026). The metrics are defined in the source as decision-planning consistency $$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$1, Final Displacement Error $$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$2, and Average Frame-wise Collision Rate, given as the number of at-fault collisions divided by the number of rollout clips (Song et al., 11 Mar 2026).

6. Relation to SD-VLA and broader design patterns

SD-VLA is not an SD-VLM system, but it is relevant because it shows how the “SD” prefix is being used in multimodal embodied modeling to denote explicit factorization of visual information (Qiu et al., 3 Feb 2026). In SD-VLA, the RGB observation at each timestep is tokenized into static tokens and dynamic tokens, with static tokens further organized hierarchically into levels of temporal persistence (Qiu et al., 3 Feb 2026). The method reduces effective context length from naive $$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$3 to

$$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$4

and reduces the leading attention term to

$$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$5

when static tokens are cached and reused (Qiu et al., 3 Feb 2026).

The model also introduces a recache gate for each static level, trained with Gumbel-softmax relaxation and regularized by

$$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$6

(Qiu et al., 3 Feb 2026). Empirically, SD-VLA reports a +39.8 percentage-point absolute improvement in On-Stove success on the LIBERO-Memory benchmark, a 29.8% reduction in heating-time error, a +3.9 percentage-point gain on SimplerEnv, a 2.26× inference speedup on that benchmark, and 43.4% of the FLOPs relative to the base VLA model (Qiu et al., 3 Feb 2026).

This relation matters because all three systems emphasize explicit structure rather than monolithic multimodal fusion. The spatial SD-VLM injects depth-aware positional structure into visual tokens (Chen et al., 22 Sep 2025). The self-driving SD-VLM inserts a decision adapter between symbolic meta-actions and trajectory planning latents (Song et al., 11 Mar 2026). SD-VLA partitions visual sequences into static and dynamic subspaces for efficient long-horizon control (Qiu et al., 3 Feb 2026). This suggests a broader design pattern in recent multimodal research: architecture-level disentanglement is being used to compensate for weaknesses in generic end-to-end VLM pipelines.

7. Limitations, misconceptions, and prospective directions

A common misconception would be to treat SD-VLM as a single, standardized architecture. The provided record does not support that interpretation. Instead, it supports at least two distinct meanings: one centered on depth-encoded spatial understanding (Chen et al., 22 Sep 2025), and another centered on self-driving decision-planning consistency inspired by Senna-2 (Song et al., 11 Mar 2026). The coexistence of SD-VLA further reinforces that acronym-level ambiguity remains unresolved (Qiu et al., 3 Feb 2026).

For the spatial reasoning SD-VLM, the stated limitations are that MSMU focuses on indoor static scenes, depth estimation errors at inference can degrade performance even though the method is robust to moderate noise, and current training uses a single-view depth map rather than multi-view or temporal integration (Chen et al., 22 Sep 2025). The authors identify possible future directions including multi-camera fusion, explicit incorporation of learned camera intrinsics, and scaling to larger VLM backbones (Chen et al., 22 Sep 2025).

For the self-driving SD-VLM description, the “Potential Extensions” listed in the source include multi-command chaining, fusion of other sensor modalities such as LiDAR or GPS embeddings alongside VLM tokens in $$\mathbf{E}^{\text{depth}}(i,j,2t)=\sin\!\bigl(\mathbf{D}'(i,j)/10000^{2t/d}\bigr),$$7, and transfer to urban or off-road settings via fine-tuning on off-road scene questions and terrain-aware diffusion priors (Song et al., 11 Mar 2026). These are presented as prospective extensions rather than demonstrated results. A plausible implication is that the explicit adapter formulation is intended to remain compatible with broader multimodal control stacks.

Across these usages, the recurring technical theme is that VLMs alone are insufficient when the downstream task requires metric grounding, temporal persistence, or policy-level consistency. The solutions differ—depth positional encoding, decision adapters with alignment losses, and static-dynamic token disentanglement—but each addresses a concrete failure mode of generic multimodal architectures through structured intermediate representations (Chen et al., 22 Sep 2025, Song et al., 11 Mar 2026, Qiu et al., 3 Feb 2026).