S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving

Abstract: Vision-LLMs (VLMs) have demonstrated remarkable potential for high-level reasoning in autonomous driving, yet they fundamentally struggle to generate precise, low-level control actions. This limitation is rooted in a semantic-physical gap caused by the inherent mismatch between discrete language tokens and continuous trajectory planning. While Vision-Language-Action (VLA) architectures attempt to bridge this gap by unifying perception and control into a single policy, this entanglement creates a new bottleneck. Standard VLAs experience a severe spatial representation collapse, which irreversibly degrades the fine-grained spatial and geometric priors essential for safe, boundary-aware navigation. To address this limitation, we propose the S-squared-VLA, which explicitly decouples the semantic and spatial streams in Vision-Language-Action models. The semantic stream leverages hierarchical bridging to extract multi-scale VLM features for robust intent reasoning. In parallel, an independent spatial stream bypasses the autoregressive language bottleneck, directly preserving uncompressed spatial features from the visual encoder. By integrating auxiliary perception supervision, this stream explicitly equips the model with rich spatial and geometric priors. Finally, a dual-stream planning adapter fuses high-level semantic intent with precise spatial constraints via cascaded attention mechanisms. Evaluations on the NAVSIM closed-loop benchmark show that S-squared-VLA achieves a Predictive Driver Model Score (PDMS) of 87.1, establishing a new state-of-the-art for VLA models under a purely supervised fine-tuning (SFT) setting. By mitigating the spatial representation collapse of traditional VLMs, our framework significantly outperforms baselines, achieving the highest No Collision (NC) rate of 98.4 among all evaluated methods.

• The paper introduces a dual-stream VLA architecture that decouples semantic and spatial processing to combat spatial representation collapse.
• It employs a multi-scale semantic stream and a task-driven spatial stream, significantly improving closed-loop navigation metrics like PDMS and safety compliance.
• The dual-stream planning adapter fuses intent reasoning with explicit geometric constraints, ensuring state-of-the-art trajectory planning in complex driving scenarios.

Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving: Analysis of S-squared-VLA

Introduction

S-squared-VLA introduces a dual-stream architecture that explicitly decouples semantic (high-level reasoning) and spatial (low-level geometric) representations in Vision-Language-Action (VLA) models, targeting the core limitations of previous vision-language planning paradigms for autonomous driving. The core motivation derives from the observation that conventional VLMs, and their one-stream VLA extensions, suffer from “spatial representation collapse”—the progressive degradation of spatial priors when reducing high-dimensional perceptual signals through discrete, autoregressive language bottlenecks. This collapse is detrimental for generating physically consistent, safe trajectories in the complex navigation scenarios arising in real-world or simulator environments.

Architectural Overview

The S-squared-VLA architecture processes multimodal driving inputs (navigation commands, ego-state, and front camera imagery) and splits representation learning after the visual encoding stage into two pathways: a multi-scale semantic stream extracting hierarchical, reasoning-rich features from a VLM backbone, and a task-driven spatial stream which preserves uncompressed dense spatial cues and injects explicit geometric priors through auxiliary objectives.

Figure 1: Overview of S-squared-VLA showing traditional E2E (semantic-blind), VLM (discrete token), and entangled VLA approaches, and highlighting the dual-stream, boundary-aware nature of S-squared-VLA.

• The semantic stream uses InternVL3-2B, aggregating action-focused features from sparsely selected transformer layers to maximize intent fusion and reasoning.
• The spatial stream leverages patch-wise ViT-based encoding, reconstructing dense BEV semantic maps and agent states with direct supervision.
• A Dual-Stream Planning Adapter fuses these with cascaded cross-attention, refining planning tokens that first resolve semantic intent and then enforce strict spatial adherence.

  Figure 2: Architecture of S-squared-VLA, showing parallel semantic and spatial encoding, auxiliary perception branches, and dual-cross-attention planning stack.

The separation of streams is reflected not just architecturally but in the training scheme: semantic understanding is improved via SFT on VQA datasets, while auxiliary BEV map and dynamic agent head objectives regularize the spatial branch.

Multi-Scale Semantic Stream

Prior vision-language approaches trend toward using compressed final-layer transformer embeddings, suffering from heavy abstraction and semantic-over-geometry bias. S-squared-VLA instead samples hierarchical representations from a sparse set of layers, combining shallow, detail-rich, and deep, intent-rich features for reasoning. Action queries drive the network explicitly towards task-relevant latent spaces, which are then combined with ego-state embeddings and fed to the planning adapter.

This design is predicated on recent evidence that multimodal transformers with hierarchical or multi-scale fusion outperform single-layer or late-fusion methods, especially in tasks requiring both abstraction and geometric precision [chen2025multimodal] [ciernik2026beyond].

Task-Driven Spatial Stream

The spatial stream circumvents the autoregressive language bottleneck by extracting and maintaining fine-grained, still-untokenized spatial information. A patch-wise ViT encoder, augmented with visual queries, forms the main backbone, and auxiliary supervision is imposed through:

• A Map Head decoding a local BEV semantic segmentation, facilitating explicit road/lane/collision boundary grounding.
• An Agent Head (DETR-like) predicting the configuration of dynamic objects, training the latent with both $L_1$ and BCE losses for accurate agent localization and classification.

  Figure 3: Architecture of the spatial stream, featuring patch extraction, visual queries, map and agent prediction heads for explicit geometric supervision.

The loss design ensures the spatial features encode priors necessary for trajectory planning, while explicit spatial regularization mitigates the geometric degradation typical in one-stream VLA or VLMs.

Dual-Stream Planning Adapter

Fusing semantic and spatial information in a dual-stage cascaded cross-attention decoder, the planning adapter:

• Aligns planning tokens (for future waypoints) first with semantic features and state memories for context-sensitive intent reasoning.
• Refines those tokens by cross-attending to dense spatial features, injecting geometric and physical constraints.

By controlling gating weights, the architecture flexibly balances intent preservation against geometric bonding, guaranteeing generated plans are both logical and physically feasible.

Empirical Evaluation

Closed-Loop Navigation Performance

Evaluated on the NAVSIM benchmark, S-squared-VLA achieves state-of-the-art supervised fine-tuning (SFT) closed-loop results, marked by:

• PDMS (Predictive Driver Model Score): 87.1
• Highest No Collision (NC): 98.4
• High Drivable Area Compliance (DAC): 94.9
• Superior Ego Progress (EP): 81.6

S-squared-VLA outperforms vision-only SFT E2E and VLA frameworks, and even LiDAR-augmented E2E models like ARTEMIS and DRAMA, despite using only monocular imagery. Critically, it posts a +3.0 gain in PDMS over the InternVL3-2B VLM baseline, which, while using the same transformer backbone, suffers via its token-centric spatial limitations (2607.13926).

Qualitative Results

Figure 4: Predicted future trajectories (orange) versus ground truth (green) for S-squared-VLA, Transfuser, InternVL3-2B, and ReCogDrive. S-squared-VLA closely tracks expert paths, maintaining spatial adherence through high-curvature turns and occluded/degraded-lane segments.

• Traditional E2E fusion models with LiDAR (e.g., Transfuser) are observed to violate boundaries due to insufficient semantic reasoning.
• VLMs (InternVL3-2B) deviate in geometric alignment during complex turns, a direct artifact of the quantization and rounding induced by discrete token planning.
• Contemporary VLAs with diffusion planners and VLM-inspired chains (ReCogDrive) show improved intent but still miss long-horizon geometric fidelity.
• S-squared-VLA's explicit spatial stream regularization ensures path smoothness and strict lane/boundary compliance.

Ablation

Ablations isolate the impact of each component:

• Adding hierarchical semantic features (planning adapter): +1.5 PDMS
• Introducing spatial stream: further +0.6
• Enabling auxiliary perception supervision (map and agent heads): additional +0.9

Improvements are most marked in safety, compliance, and long-horizon progress, confirming that explicit spatial supervision coupled with intent-aware planning is essential for the high-complexity, boundary-critical requirements of operational autonomous driving.

Theoretical and Practical Implications

This work exposes the semantic-physical gap in VLMs for driving: discrete, high-level language representations cannot preserve the high-resolution, continuous state information compulsory for safe planning. Existing VLA models, by entangling these domains, create an irretrievable spatial representation collapse. The dual-stream design solves this, supporting intent reasoning while openly retaining geometric fidelity.

Practically, the S-squared-VLA design demonstrates that fine-grained, camera-only spatial clues—if supervised and preserved—can match or exceed LiDAR-dependent pipelines in large-scale, closed-loop simulation, cutting substantial hardware and cost barriers for deployment.

The architecture is modular and training-paradigm-agnostic. This opens direct future integration with more efficient attention modules or closed-loop RL fine-tuning, which would further improve the operational reliability and reduce dependence on heavy backbone computation.

Conclusion

S-squared-VLA decisively demonstrates that decoupling semantic and spatial streams, with explicit dual-path auxiliary regularization, is critical for scaling Vision-Language-Action models to the stringent demands of autonomous driving. It establishes new SFT state-of-the-art results on NAVSIM, highlights the dangers of semantic-geometric entanglement, and motivates the broader adoption of physically-grounded multimodal fusion for AI planning under real-world constraints. Future developments are expected in reducing inference costs and leveraging RL adaptation, while retaining the dual-stream guarantee of intent-aware, spatially precise planning.

(2607.13926)