UrbanVLA: A Vision-Language-Action Model for Urban Micromobility (2510.23576v1)

Abstract: Urban micromobility applications, such as delivery robots, demand reliable navigation across large-scale urban environments while following long-horizon route instructions. This task is particularly challenging due to the dynamic and unstructured nature of real-world city areas, yet most existing navigation methods remain tailored to short-scale and controllable scenarios. Effective urban micromobility requires two complementary levels of navigation skills: low-level capabilities such as point-goal reaching and obstacle avoidance, and high-level capabilities, such as route-visual alignment. To this end, we propose UrbanVLA, a route-conditioned Vision-Language-Action (VLA) framework designed for scalable urban navigation. Our method explicitly aligns noisy route waypoints with visual observations during execution, and subsequently plans trajectories to drive the robot. To enable UrbanVLA to master both levels of navigation, we employ a two-stage training pipeline. The process begins with Supervised Fine-Tuning (SFT) using simulated environments and trajectories parsed from web videos. This is followed by Reinforcement Fine-Tuning (RFT) on a mixture of simulation and real-world data, which enhances the model's safety and adaptability in real-world settings. Experiments demonstrate that UrbanVLA surpasses strong baselines by more than 55% in the SocialNav task on MetaUrban. Furthermore, UrbanVLA achieves reliable real-world navigation, showcasing both scalability to large-scale urban environments and robustness against real-world uncertainties.

• The paper presents UrbanVLA, a route-conditioned VLA framework that aligns high-level route instructions with real-time visual observations for robust urban micromobility navigation.
• It integrates a dual-branched architecture combining VideoQA and navigation tasks, employing supervised fine-tuning on simulation data followed by reinforcement fine-tuning using IQL.
• Experimental results show significant improvements in success rate, social compliance, and generalization, outperforming traditional methods by over 25% in challenging scenarios.

UrbanVLA: A Route-Conditioned Vision-Language-Action Model for Urban Micromobility

Introduction and Motivation

Urban micromobility platforms, including delivery robots and assistive devices, require robust navigation in dynamic, unstructured city environments. Traditional SLAM-based navigation pipelines are constrained by their reliance on high-fidelity maps, limiting scalability and adaptability in large, ever-changing urban spaces. Recent learning-based approaches, particularly those leveraging foundation models and vision-language-action (VLA) architectures, have demonstrated improved generalization but remain challenged by noisy route information and the need for long-horizon, socially compliant navigation.

UrbanVLA introduces a route-conditioned VLA framework that explicitly aligns high-level, noisy route waypoints (roadbooks) with real-time visual observations, enabling trajectory planning that is robust to environmental uncertainty and misaligned navigation cues. The model is trained via a two-stage pipeline: supervised fine-tuning (SFT) on simulated and web video data, followed by reinforcement fine-tuning (RFT) using Implicit Q-Learning (IQL) on a sim-real aggregated dataset.

Figure 1: Overview of UrbanVLA's two-stage training pipeline, integrating VideoQA and urban micromobility demonstrations for SFT and RFT with IQL for real-world robustness.

Model Architecture and Route Encoding

UrbanVLA builds upon a pre-trained navigation foundation model (NavFoM) and incorporates a dual-branched architecture for both navigation and VideoQA tasks. High-level route instructions are encoded as structured linguistic prompts, combining resampled waypoints and block-level turn cues derived from navigation APIs or heuristic trajectory lifting (HTL). Visual observations are captured from multi-view RGB cameras and processed using DINOv2 and SigLIP encoders, with features projected into the LLM backbone (Qwen2) via a cross-modal MLP.

The navigation policy $$\pi(\mathcal{R}, \mathcal{O}_{\text{vis}}) \mapsto \tau$$ predicts a sequence of waypoints $\tau$ in SE(2), directly conditioned on both route and visual context. The action head decodes trajectory tokens, while the language head supports VideoQA for enhanced scene understanding.

Training Strategy: SFT and RFT

Supervised Fine-Tuning (SFT)

SFT leverages expert demonstrations from the MetaUrban simulator and web-scale urban travel videos. The HTL algorithm extracts abstracted route information from raw trajectories, introducing Gaussian noise and segmenting by detected corners to prevent overfitting to idealized routes. This process yields tuples of (route, visual observation, ground-truth trajectory) for MSE optimization, instilling goal-reaching and generalization capabilities.

Reinforcement Fine-Tuning (RFT)

RFT employs IQL for offline RL, optimizing the policy on a hybrid dataset of simulated and real-world teleoperation episodes. The reward function balances trajectory completion, collision avoidance, and deviation penalties, with coefficients tuned for efficient and safe navigation. State representations are derived from midlayer LLM hidden states, empirically shown to stabilize Q-function learning. The action space is vectorized over $N$ waypoints, enabling trajectory-level optimization.

Real-World Deployment and System Integration

UrbanVLA is deployed on a quadruped robot equipped with GPS, Wi-Fi, multi-view cameras, and onboard compute. The system interfaces with a mobile console for real-time monitoring, navigation target assignment, map visualization, and teleoperation data annotation.

Figure 2: Real-world deployment setup of UrbanVLA, illustrating hardware integration and data flow for navigation and RL annotation.

Experimental Results

Quantitative Performance

UrbanVLA achieves 94% SR and 0.91 SPL on the PointNav test set, and 97% SR and 0.95 SPL on unseen environments, outperforming LiDAR-based RL and IL baselines by over 25% in SR. In SocialNav, the model attains 0.87 SNS (test) and 0.85 SNS (unseen), demonstrating superior social compliance and collision avoidance using only RGB inputs. The cumulative cost is bounded despite higher success rates, reflecting effective obstacle avoidance in challenging scenarios.

Qualitative Evaluation

UrbanVLA demonstrates robust trajectory generation in diverse real-world scenarios, including overpass crossings, street turns, pedestrian interactions, and obstacle avoidance over long distances (>500m). The model adapts to illumination changes, dynamic obstacles, and route misalignments, maintaining social norms and route fidelity.

Figure 3: Qualitative results in real-world scenarios, showing UrbanVLA's executive trajectories for overpass crossing, street turning, pedestrian crossing, and obstacle avoidance.

Ablation Studies

• HTL Effectiveness: Ablating HTL improves simulation SR by 4% but drastically reduces real-world route completion (RC drops from 100% to 42%), indicating overfitting to route instructions and poor robustness to noisy navigation cues.
• RFT Impact: Adding RFT to SFT increases SR by 6% and reduces cost by 0.16 on unseen benchmarks, confirming that RL with real-world data enhances generalization and safety-critical behavior.

Implications and Future Directions

UrbanVLA establishes a scalable framework for urban micromobility, integrating high-level navigation tools with vision-language policy learning. The explicit alignment of route and visual context, combined with sim-to-real RL, enables deployment in dynamic pedestrian environments without reliance on high-fidelity maps or LiDAR. The strong numerical results, particularly the 55–56% improvement over baselines in SocialNav, highlight the efficacy of route-conditioned VLA models for long-horizon, socially compliant navigation.

Theoretically, UrbanVLA demonstrates the viability of multimodal foundation models for embodied AI in open-world settings, suggesting future research directions in:

• Incorporating additional modalities (e.g., audio, semantic maps)
• Extending to heterogeneous robot platforms and broader urban contexts
• Advancing RL fine-tuning strategies for improved sim-to-real transfer and safety guarantees

Conclusion

UrbanVLA presents a route-conditioned VLA model for urban micromobility, trained via a two-stage pipeline integrating simulation, web-scale video, and real-world RL. The model achieves state-of-the-art performance in both simulated and real environments, demonstrating robust, scalable, and socially compliant navigation. This work provides a foundation for future embodied AI systems capable of reliable operation in complex, dynamic urban spaces.