Utilizing Vision-Language Models for Physical Robot Control

Determine effective mechanisms to leverage the emergent capabilities of large vision-language models pre-trained on Internet-scale data to reliably control physical robots for real-world manipulation tasks.

Background

The paper addresses open-vocabulary robotic manipulation using vision-LLMs (VLMs) and proposes a mark-based visual prompting approach (MOKA) to bridge VLM predictions on RGB images with robot motions via point-based affordances. Although VLMs exhibit strong conceptual understanding and commonsense reasoning, there remains a critical gap in applying these capabilities to embodied control in physical environments.

The authors explicitly frame this challenge as an open question concerning how to utilize the emergent capabilities of VLMs to control robots, motivating their hierarchical prompting and visual mark-based strategy to scaffold affordance reasoning and motion generation.