Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding (2501.04693v3)

Abstract: Interacting with the world is a multi-sensory experience: achieving effective general-purpose interaction requires making use of all available modalities -- including vision, touch, and audio -- to fill in gaps from partial observation. For example, when vision is occluded reaching into a bag, a robot should rely on its senses of touch and sound. However, state-of-the-art generalist robot policies are typically trained on large datasets to predict robot actions solely from visual and proprioceptive observations. In this work, we propose FuSe, a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities for which large datasets are not readily available by leveraging natural language as a common cross-modal grounding. We combine a multimodal contrastive loss with a sensory-grounded language generation loss to encode high-level semantics. In the context of robot manipulation, we show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound in a zero-shot setting, such as multimodal prompting, compositional cross-modal prompting, and descriptions of objects it interacts with. We show that the same recipe is applicable to widely different generalist policies, including both diffusion-based generalist policies and large vision-language-action (VLA) models. Extensive experiments in the real world show that FuSeis able to increase success rates by over 20% compared to all considered baselines.

• The paper presents FuSe, a novel finetuning method that integrates heterogeneous sensors into pre-trained visuomotor policies using language grounding.
• It employs multimodal contrastive and generative losses to align natural language with sensory observations, enhancing semantic inference.
• Real-world experiments on manipulation tasks show over 20% success rate improvement, underscoring FuSe's practical impact.

Fine-Tuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding

The research presented in the paper "Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding" introduces a novel methodology, FuSe, for adapting pre-trained generalist robot policies to utilize diverse sensor modalities. This paper addresses the significant challenge of effectively incorporating non-visual sensory modalities, such as touch and audio, into robotic policies, which predominantly leverage large datasets of visual data. By employing natural language as a cross-modal grounding mechanism, the authors propose a robust finetuning strategy that unlocks new operational capabilities for robots in multimodal environments.

Methodology Overview

The FuSe approach focuses on finetuning pre-trained visuomotor policies. It leverages large, image-based pretrained generalist models and integrates other sensor modalities. The core innovation revolves around utilizing natural language as a common grounding for these multimodal architectures:

• Multimodal Contrastive Loss: A CLIP-style contrastive loss is employed to align language instructions with observations, maximizing mutual information across different modalities.
• Multimodal Generative Loss: An additional generative loss predicts high-level semantics from sensory observations, enhancing the model's semantic understanding across modalities.

These auxiliary losses are key to effectively integrating heterogeneous sensory data into the learned policies. The losses are designed to connect pre-trained semantic knowledge with modalities outside the original training domain, ensuring that the learning process does not over-rely on visual data alone.

Experimental Validation

The practical application of FuSe is validated through extensive real-world experiments using a WidowX robotic platform equipped with vision, touch, audio, and proprioception sensors. The research investigates three distinct manipulation tasks:

1. Tabletop Grasping: A task requiring the robot to select and grasp specific objects with varying textures and visual characteristics.
2. Shopping Bag Environment: A scenario involving occlusion challenges where the robot must select and retrieve items from inside a bag, highlighting the importance of non-visual sensors.
3. Button Pressing with Audio Feedback: Tasks requiring the robot to discern between buttons based on both visual appearance and audio output, showcasing the integration of sound.

The FuSe-trained policies demonstrated superior performance compared to baselines, including models trained from scratch and those relying solely on visual data. In particular, success rates improved by over 20%, especially in environments with partial visual observability, such as the shopping bag task.

Multimodal and Compositional Reasoning

A significant aspect of FuSe is its support for zero-shot multimodal reasoning. The methodology facilitates complex task prompting that requires joint inference across modalities. For example, in instances where visual data is insufficient, tactile or auditory cues can be integrated seamlessly to provide a holistic understanding of the task environment. Moreover, the ability to use compositional language instructions reflecting multiple modalities underscores the model's flexibility and adaptability.

Implications and Future Outlook

This work has broad implications for advancing the autonomy and versatility of robotic systems. By demonstrating successful integration of diverse sensory inputs, FuSe paves the way for deploying robots in real-world environments where pure reliance on visual data is impractical. The approach encourages rethinking how robotic learning models can be structured to become truly generalist in processing and responding to environmental cues. The open-source nature of the dataset, code, and models will catalyze further research into language-grounded multimodal robotics.

Future developments may aim towards increasing the scalability of the approach, ensuring efficiency in deploying large models with diverse input streams. Enhancing training efficiency could facilitate longer context length processing, potentially improving interaction with sparse signals like tactile feedback. Moreover, exploring additional sensory modalities and their impacts on robotic perception and decision-making could further extend the capabilities demonstrated by FuSe.