Develop robust multimodal fusion for generalist robotic manipulation policies

Develop principled multimodal fusion techniques that reliably integrate visual, proprioceptive, and linguistic inputs to improve performance and generalization of generalist robotic manipulation policies.

Background

LBMs and related VLA approaches rely on combining multiple sensory modalities. While the paper fixes an architecture for controlled evaluation, it notes that multimodal fusion remains an open research challenge for generalist policies.

Improved fusion could enhance language grounding, perception-action coupling, and robustness under distribution shift.

References

Despite progress in training generalist policies, challenges such as catastrophic forgetting, data heterogeneity, scarcity of high-quality data, multimodal fusion, handling dexterity, and maintaining real-time inference speed remain open research problems.

— A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation  (2507.05331 - Team et al., 7 Jul 2025) in Section 2.1, Related Work—Robot Learning at Scale

Across a policy trained from scratch, an adapted vision-language-action model, and a world--action model, these mixed ID results show that reliably fusing tactile cues remains an open modeling problem.

— SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation  (2608.18701 - Jing et al., 19 Aug 2026) in Conclusion