AURA: Multimodal Shared Autonomy for Real-World Urban Navigation

Abstract: Long-horizon navigation in complex urban environments relies heavily on continuous human operation, which leads to fatigue, reduced efficiency, and safety concerns. Shared autonomy, where a Vision-Language AI agent and a human operator collaborate on maneuvering the mobile machine, presents a promising solution to address these issues. However, existing shared autonomy methods often require humans and AI to operate within the same action space, leading to high cognitive overhead. We present Assistive Urban Robot Autonomy (AURA), a new multi-modal framework that decomposes urban navigation into high-level human instruction and low-level AI control. AURA incorporates a Spatial-Aware Instruction Encoder to align various human instructions with visual and spatial context. To facilitate training, we construct MM-CoS, a large-scale dataset comprising teleoperation and vision-language descriptions. Experiments in simulation and the real world demonstrate that AURA effectively follows human instructions, reduces manual operation effort, and improves navigation stability, while enabling online adaptation. Moreover, under similar takeover conditions, our shared autonomy framework reduces the frequency of takeovers by more than 44%. Demo video and more detail are provided in the project page.

• The paper presents a hierarchical multimodal framework that integrates high-level human guidance with low-level diffusion-based control for urban navigation.
• It leverages geometric inputs like drafting and arrowing processed by a spatial-aware encoder to reduce trajectory errors by up to 39.8%.
• The system demonstrates practical deployment on urban sidewalks, reducing human intervention by over 44% while ensuring operational safety.

AURA: Multimodal Shared Autonomy for Real-World Urban Navigation

Introduction and Context

This paper proposes AURA, a novel multimodal shared autonomy framework for urban navigation that combines high-level human guidance with low-level autonomous robotic control. The core motivation arises from the increasing deployment of mobile robots (e.g., delivery robots, assistive wheelchairs) in urban pedestrian spaces, where the diverse environment and unpredictable agents limit the practical adoption of full autonomy. Conventional shared autonomy solutions typically require humans and AI to operate in the same low-level action space, imposing significant cognitive burden and limiting efficiency for long-horizon navigation tasks. AURA addresses this limitation by creating a hierarchical system where human participants provide intermittent, abstract, multi-modal instructions, and the robot executes continuous low-level control.

Figure 1: AURA introduces shared autonomy for urban navigation, enabling real-time human guidance via visual and language instructions.

Architecture and Methodology

AURA implements a dual-system Vision-Language-Action (VLA) architecture, consisting of a multimodal encoder and a diffusion-based policy executor.

Multimodal Encoder

The system ingests RGB camera observations and a spectrum of human instructions, encompassing:

• Texting: users provide natural-language intent (“go straight”, “yield to pedestrians”),
• Drafting: users draw future paths via spatial overlays,
• Arrowing: users specify speed and heading through vector annotations.

The core to instruction grounding is the Spatial-Aware Instruction Encoder (SIE), which projects both textual and geometric cues into a joint representation. For drafting, pixel coordinates of the drawn paths are processed via positional encodings, while arrowing instructions are encoded as rotation-invariant vectors. All conditioned tokens are fused through a ViT-based encoder and fed into a pretrained LLM adapted with LoRA adapters.

Figure 2: Schematic overview of the AURA framework, depicting the data flow from multimodal input through SIE encoding and action prediction.

Diffusion Policy Executor

AURA generates robot actions via a diffusion-based transformer (DiT) policy, leveraging anchor-based regression over trajectory primitives derived from real-world demonstrations. At each step, the DiT receives the fused context embedding, the navigation goal, and current observations to infer a distribution over future waypoints. This enables robust, context-sensitive control that closely tracks both human and environmental requirements.

MM-CoS: Dataset and Auto-Labeling Pipeline

AURA’s development is facilitated by MM-CoS, a 50-hour multimodal video dataset collected via teleoperation on real-world urban sidewalks. MM-CoS incorporates diverse environmental conditions and is substantially annotated using a novel auto-labeling pipeline that employs VLMs for both “interesting event” selection and grounded action explanation.

Figure 3: Example annotation triplets from auto-labeling: textual directive, drafting overlay, and arrowing vector with aligned reasoning trace supervision.

The annotation pipeline operates by scoring temporal windows based on motion dynamics and “interestingness” as determined by a large vision-LLM. Informative frames are labeled with three complementary modalities, reflecting the practical instruction paradigms available to operators: concise verbal directives, geometric route sketches, and instantaneous velocity/heading overlays.

Shared Autonomy Problem Formulation

AURA supports two operational modes:

• Autopilot: autonomous navigation to sparse waypoints (e.g., GPS destinations)
• Takeover: upon detection of ambiguous context (e.g., unreliable GPS, corner cases), humans may supply instructions via any supported modality. These instructions have a finite temporal influence window.

Navigation is reframed as sequential decision-making, where each timestep’s action is a function of recent observations and latest instruction tokens.

Training and Adaptation Protocol

The framework employs a two-stage curriculum:

1. Instruction-Conditioned VLM Adaptation: InternVL3-2B is adapted using LoRA, with the SIE module trained to maximize agreement between VLM-generated captions and expert-provided navigation reasoning.
2. Diffusion Policy Learning: The DiT decoder and associated goal/anchor encoders are trained end-to-end, optimizing both classification over trajectory modes and L2 trajectory regression.

Quantitative and Qualitative Results

Instruction Following

AURA’s instruction adherence is evaluated on the MM-CoS test set using both open-loop (ground-truth comparison) and closed-loop metrics. Across baseline models and all instruction modalities, AURA delivers the lowest L2 errors at both 1s and 2s horizons. Notably, geometric modalities such as drafting and arrowing yield the most precise trajectory alignment, with arrowing guidance reducing 2s L2 error by 39.8% relative to re-trained strong baselines.

Figure 4: Visualization of model-predicted future trajectories under three forms of human instruction (text, draft, arrow).

Shared Control Cost

AURA markedly reduces human operation ratio and intervention frequency in a simulated shared-control testbed. Compared to prior systems, the frequency of required human takeovers drops by over 44%. This translates to a ~70% reduction in overall human intervention time for long-horizon navigation, without loss in safety or goal attainment.

Figure 5: Analysis of human intervention cost across control systems, showing substantial reduction achieved by AURA.

Ablation and Component Analysis

Ablations demonstrate that the combined use of SIE (geometric encoding) and visual prompts outperforms either in isolation, especially for instructions with longer time-lag, indicating the value of explicit geometric context for spatial and temporal robustness.

Figure 6: Effectiveness of semantic and geometric features on planning accuracy under temporal delay.

Real-World Deployment

In on-robot experiments (2.8 km, 16 routes), AURA minimizes human operation ratio (HO), normalized intervention rate (NIR), and off-track distance while maximizing time under safe autonomy. The system generalizes to novel sidewalk topologies and lighting/weather conditions, demonstrating practical transfer from the MM-CoS data and annotation regime.

Figure 7: Teleoperation interface during real-world navigation; green polygons show predicted trajectories, red lines indicate robot bounds at takeover.

Figure 8: Real-world hardware and operator interface utilized in experiments.

Implications and Future Directions

AURA represents a significant advance in practical shared autonomy. Hierarchically decoupling human and AI responsibilities while supporting multimodal instruction markedly reduces human workload, making long-horizon navigation feasible in complex urban environments. The introduction of the SIE and structured synthetic annotation establishes a scalable path for future VLA systems to robustly align human intent with low-level control, accommodating both semantic and geometric modalities.

On the theoretical side, the architecture opens avenues for deeper integration of scene understanding, task inference, and safety-driven constraint satisfaction. The modular nature of instruction processing in AURA lends itself to plug-in upgrades as new foundation VLMs or goal representations become available.

Practically, this work leads toward truly assistive sidewalk robots capable of scalable, robust partial autonomy with minimal dependence on online human control or expensive map priors. The successful deployment and online adaptation mechanisms further support the claim of operational readiness in real city environments.

Conclusion

This paper delivers a systematic and rigorously validated framework for vision-language shared autonomy in urban navigation. By uniquely combining multimodal instruction processing, a diffusion-driven control architecture, and scalable human-in-the-loop data annotation, AURA achieves robust, efficient real-world navigation with dramatically reduced human operation cost. The design principles and empirical findings have immediate ramifications for both theoretical development and practical deployment of shared autonomy systems in open, dynamic domains.