OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation (2509.19480v1)

Abstract: Humans can flexibly interpret and compose different goal specifications, such as language instructions, spatial coordinates, or visual references, when navigating to a destination. In contrast, most existing robotic navigation policies are trained on a single modality, limiting their adaptability to real-world scenarios where different forms of goal specification are natural and complementary. In this work, we present a training framework for robotic foundation models that enables omni-modal goal conditioning for vision-based navigation. Our approach leverages a high-capacity vision-language-action (VLA) backbone and trains with three primary goal modalities: 2D poses, egocentric images, and natural language, as well as their combinations, through a randomized modality fusion strategy. This design not only expands the pool of usable datasets but also encourages the policy to develop richer geometric, semantic, and visual representations. The resulting model, OmniVLA, achieves strong generalization to unseen environments, robustness to scarce modalities, and the ability to follow novel natural language instructions. We demonstrate that OmniVLA outperforms specialist baselines across modalities and offers a flexible foundation for fine-tuning to new modalities and tasks. We believe OmniVLA provides a step toward broadly generalizable and flexible navigation policies, and a scalable path for building omni-modal robotic foundation models. We present videos showcasing OmniVLA performance and will release its checkpoints and training code on our project page.

• The paper introduces OmniVLA, an omni-modal vision-language-action model that fuses visual, positional, and language inputs for enhanced robot navigation.
• It achieves state-of-the-art performance with 100% success in image-conditioned tasks and significant gains in pose and language-conditioned navigation.
• The model employs modality dropout and cross-modal learning, yielding robust generalization across diverse environments and facilitating cross-embodiment transfer.

OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigation

Motivation and Problem Statement

Robotic navigation in unstructured environments demands flexible goal specification, analogous to human navigation, which leverages language, spatial coordinates, and visual cues. Existing navigation policies are typically trained on a single modality, resulting in limited adaptability and poor generalization to real-world scenarios where multi-modal goal specification is natural and often necessary. OmniVLA addresses this gap by proposing a unified vision-language-action (VLA) model capable of ingesting and reasoning over goals expressed in multiple modalities—2D poses, egocentric images, natural language, and their combinations—thus enabling more robust, generalizable, and user-friendly navigation policies.

Model Architecture

OmniVLA builds upon a high-capacity VLA backbone (OpenVLA, 7B parameters), integrating knowledge from internet-scale pre-training and cross-embodiment robot actions. The architecture supports flexible goal conditioning via modality-specific encoders for visual, positional, and language inputs, projecting each into a shared token space for the LLM backbone. Modality dropout is employed during training and inference to mask unavailable modalities, ensuring the model learns to attend to all available goal signals and develops cross-modal representations.

Figure 1: Network architectures for multi-modal vision-based navigation, showing modality-specific encoders and shared token projection for LLM-based action generation.

A resource-efficient variant, OmniVLA-edge, is implemented atop ViNT (50M parameters), using early fusion and temporal token stacking for lightweight deployment. Both architectures utilize pre-trained visual and language backbones (e.g., DINOv2, CLIP, Llama2) and support LoRA-based parameter-efficient fine-tuning for large models.

Figure 2: Network architecture of OmniVLA-edge based on vision-based navigation policies, illustrating modality fusion and transformer-based action prediction.

Training Methodology

OmniVLA is trained on a mixture of 13 publicly available datasets, totaling 9,500 hours across 10 robot platforms and diverse environments. The training data encompasses both manually collected and synthetically labeled actions, with reannotation models (e.g., MBRA) used to bridge embodiment gaps and ensure action consistency across platforms. Modality dropout is applied to randomly mask goal modalities per sample, promoting robust cross-modal representation learning and mitigating modality imbalance.

The training objective combines imitation learning loss over $N$-step action sequences, object-reaching loss for language-conditioned navigation, and smoothness regularization:

$\min_\theta J := J_{il} + m_{obj}J_{obj} + J_{sm}$

where $J_{il}$ is the imitation loss, $J_{obj}$ encourages object reaching, and $J_{sm}$ regularizes action deltas.

Experimental Evaluation

Task Setup

Three navigation tasks are considered: (1) language-conditioned navigation, (2) egocentric goal image-conditioned navigation, and (3) 2D goal pose-conditioned navigation. Evaluations are conducted in 40+ real-world environments using FrodoBots ERZ, VizBot, and Unitree Go1 platforms, with diverse and out-of-distribution (OOD) prompts and challenging obstacle configurations.

Figure 3: Overview of the robotic platforms in our evaluation, demonstrating cross-embodiment deployment.

Baselines

OmniVLA is compared against seven baselines, including specialist policies (ViNT, MBRA, NoMaD), VLA models (SmolVLA, MiniVLA, CounterfactualVLA), and methods leveraging internet-scale visual representations (LeLaN, CoW).

Results

Single-Modality Performance: OmniVLA consistently outperforms specialist baselines across all modalities. Notably, it achieves 100% success in image-conditioned navigation, a 9% improvement in pose-conditioned tasks over MBRA-pose, and a substantial gain in OOD language instruction following compared to LeLaN and CounterfactualVLA. Smaller VLA models (SmolVLA, MiniVLA) underperform due to limited capacity and domain mismatch.

Multi-Modal Conditioning: OmniVLA demonstrates the ability to attend to and reason over multiple goal modalities simultaneously. In tasks requiring both 2D pose and behavioral language instructions, OmniVLA achieves 80% success, outperforming all baselines, none of which can handle such compositional goal specification.

Figure 4: Visualization of language-conditioned navigation rollouts, showing successful OOD instruction following in diverse environments.

Ablation Studies: Training on multi-modal data yields clear advantages in generalization and performance, especially for language and satellite image-conditioned navigation. Inclusion of diverse cross-embodiment data (e.g., BDD-V) further enhances robustness.

Adaptation and Transfer: OmniVLA exhibits foundation model qualities, adapting efficiently to new goal modalities (e.g., satellite imagery) and environments with minimal data (1.2 hours for fine-tuning). Cross-embodiment transfer is demonstrated by successful deployment on VizBot and Go1 platforms without retraining.

Figure 3: Overview of the robotic platforms in our evaluation, demonstrating cross-embodiment deployment.

Implementation Considerations

• Computational Requirements: Training OmniVLA requires significant resources (e.g., 8 H100 GPUs, batch size 224), but LoRA-based fine-tuning enables efficient adaptation.
• Deployment: OmniVLA-edge is suitable for resource-constrained platforms, maintaining strong performance with reduced model size.
• Data Diversity: Robust generalization is contingent on diverse, high-quality datasets spanning modalities, embodiments, and environments.
• Modality Dropout: Essential for handling missing or scarce modalities and promoting cross-modal reasoning.

Implications and Future Directions

OmniVLA establishes a scalable recipe for omni-modal navigation policies, serving as a pre-trained foundation for downstream fine-tuning and adaptation. The demonstrated generalization, robustness to modality scarcity, and cross-embodiment transfer have direct implications for real-world deployment in heterogeneous robotic fleets and user-facing applications. The results highlight the importance of large-scale, diverse data and high-capacity models for achieving flexible, generalist navigation.

Future research should focus on improving language-conditioned navigation, particularly in instruction-following and goal-reaching under open-set prompts. Progress will benefit from larger, more curated datasets and advances in VLM architectures. The release of OmniVLA models and code will facilitate community-driven development of scalable, omni-modal robotic foundation models.

Conclusion

OmniVLA presents a unified, omni-modal vision-language-action model for robot navigation, achieving state-of-the-art performance across modalities and demonstrating strong foundation model properties. Its architecture, training methodology, and empirical results provide a robust framework for generalizable, flexible, and scalable navigation policies, with significant implications for future research and deployment in autonomous robotics.