Interaction between high-level agents and low-level vision-language-action executors

Develop interaction methods between the high-level planning agent in RoboMemory (the Closed-Loop Planning Module) and low-level vision-language-action executors (e.g., pi_0) that utilize modalities beyond natural language to communicate fine-grained action details such as grasp points, which are difficult to specify via language-only instructions.

Background

RoboMemory introduces a hierarchical agent framework where a high-level Closed-Loop Planning Module generates plans and a low-level Vision-Language-Action (VLA) executor (e.g., pi_0) carries out primitive actions on real robots. Despite performance gains in simulation and real-world tests, the authors note that interfacing between these layers is a fundamental bottleneck.

Most existing hierarchical systems rely on language-only commands from the planner to the executor. However, many action parameters in robotics (e.g., grasp points, motion waypoints, and fine spatial constraints) are better conveyed through visual or other non-linguistic modalities. The paper explicitly identifies improving this multimodal interaction as an unsolved problem critical to robust long-horizon embodied control.