ViNT: A Foundation Model for Visual Navigation (2306.14846v2)

Abstract: General-purpose pre-trained models ("foundation models") have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on singular datasets. ViNT can be augmented with diffusion-based subgoal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or routing commands) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establishes ViNT as an effective foundation model for mobile robotics. For videos, code, and model checkpoints, see our project page at https://visualnav-transformer.github.io.

• The paper introduces ViNT, a Transformer-based model that achieves zero-shot generalization using EfficientNet encoders and 100 hours of diverse navigation data.
• The paper demonstrates ViNT's effective prediction of future navigation actions and its ability to navigate kilometer-scale environments using diffusion-based goal proposals.
• The paper outlines ViNT's flexible adaptation techniques, enabling modality replacement to accommodate new robotic platforms and task specifications.

Overview of ViNT: A Foundation Model for Visual Navigation

The Visual Navigation Transformer (ViNT) introduces a foundation model specifically designed for vision-based robotic navigation, employing the success of general-purpose pre-trained models evident in other domains such as NLP and visual perception. ViNT leverages a Transformer-based architecture to capture navigational affordances and enable effective adaptation to a variety of downstream navigational tasks, standing out as a versatile and robust approach to mobile robotics navigation.

Model Architecture and Training

ViNT encompasses an architecture that integrates a cross-embodiment foundation model capable of zero-shot generalization across multiple environments and robot embodiments. The model's core is a Transformer leveraging EfficientNet encoders to jointly encode observation sequences and goal images. This setup enables the extraction of essential navigational features needed for efficient goal-reaching behavior. Notably, ViNT is trained using a dataset comprising 100 hours of diverse robotic navigation data from multiple platforms, underscoring its robust design capable of handling varied environmental dynamics and camera configurations.

ViNT is formulated to predict future navigation actions conditioned on visual observations and goal images. The training employs a maximum likelihood estimation framework to optimize goal-reaching trajectories, where the temporal distance to the goal and future action sequences are core predictive targets. This architecture and training approach afford ViNT the ability to learn general-purpose navigation policies that demonstrate strong positive transfer properties.

Capabilities and Performance

A standout feature of ViNT is its ability to undertake exploratory tasks in novel environments by leveraging diffusion-based goal proposals. These proposals augment ViNT's capacity to navigate kilometer-scale environments, exemplifying its scalability and applicability to real-world robotic navigation tasks. Experiments have shown ViNT's ability to perform undirected exploration efficiently while maintaining a high success rate relative to existing specialist models.

Furthermore, ViNT accommodates various task specifications through a novel adaptation technique, replacing its goal encoder with encoders of other modalities, like GPS or discrete navigation commands. This facilitates ViNT's adaptation to new robotic platforms and task objectives with minimal additional training data.

Implications and Future Directions

ViNT signifies a meaningful step towards the development of general-purpose robotic foundation models. Its design effectively bridges the gap between domain-specific navigation solutions and those applicable across different environments and robotic platforms. The versatility exhibited by ViNT in adapting to numerous downstream tasks paints a promising picture for future developments in robot autonomy and cross-domain learning.

The exploration of emergent behaviors, such as collision avoidance and adaptation to dynamic environments, highlights ViNT's potential to provide intelligent and adaptable navigation solutions. Future research might focus on increasing the model's capacity and extending its applicability to incorporate additional sensory modalities or to cover a wider array of robotic tasks.

In summary, the ViNT model's ability to combine zero-shot generalization, positive transfer, and flexible adaptation makes it a noteworthy candidate for deploying mobile robotics in diverse environments, paving the way for more integrated and robust autonomous systems.