MoTo: Zero-Shot Interaction-Aware Navigation

• MoTo is a zero-shot module that equips mobile manipulators with interaction-aware docking by generating keypoints and optimized arm trajectories.
• It fuses multi-view scene reconstruction with vision-language models to identify actionable target and arm keypoints from RGB-D data.
• MoTo employs Dual Annealing for joint optimization of base and arm trajectories, improving docking success on mobile manipulation benchmarks.

MoTo (“Move and Touch”) is a zero-shot plug-in interaction-aware navigation module for general mobile manipulation. It is designed to be combined with any off-the-shelf manipulation foundation model to empower them with mobile manipulation ability by generating robot docking points from which fixed-base manipulation can be successfully performed. Rather than learning an end-to-end mobile manipulation policy from mobile manipulation expert data, MoTo uses pre-trained vision-LLMs, LLMs, visual foundation models, scene reconstruction, and optimization-based motion planning to infer interaction-relevant keypoints and feasible base-and-arm trajectories in a zero-shot manner (Wu et al., 1 Sep 2025).

1. Problem formulation and design objective

MoTo addresses the “last-mile” problem in mobile manipulation: given a general manipulation policy that assumes a fixed base, how should a wheeled robot move to a docking position from which the arm can successfully execute the manipulation? The method is motivated by three limitations identified in prior systems. End-to-end mobile manipulation methods require large-scale mobile manipulation demonstration data. Fixed-base manipulation foundation models such as OpenVLA, RDT-1B, iDP3, and AnyGrasp cannot decide how the mobile base should move. Classical modular pipelines typically navigate “near” an object, but not necessarily to a pose that satisfies reachability, clearance, and downstream manipulation feasibility (Wu et al., 1 Sep 2025).

The formulation is posed as a joint optimization over base and arm trajectories: $$\min_{\mathbf{o}_{\text{base}}, \{\mathbf{o}_{\text{arm}}\}} O(\mathbf{o}_{\text{base}}, \{\mathbf{o}_{\text{arm}}\}) \quad \text{s.t. } \mathbf{o}_{\text{base}} \in \mathbb{R}^3,\ \mathbf{o}_{\text{arm}} \in \mathbb{R}^n.$$ Here the base pose is optimized together with arm configurations so that a fixed-base manipulation policy can be invoked only after the robot has reached a manipulation-ready docking point (Wu et al., 1 Sep 2025).

A central distinction in MoTo is that navigation is explicitly interaction-aware. The target is not merely an object instance or receptacle location, but the part of the scene where the intended interaction should occur. This makes MoTo a last-mile navigation method rather than a complete mobile manipulation policy, and its zero-shot property refers specifically to the absence of mobile manipulation expert data for the navigation-and-docking component (Wu et al., 1 Sep 2025).

2. System architecture and plug-in workflow

MoTo begins with scene scanning and reconstruction. The robot collects RGB-D views, from which it builds a 3D point cloud $P \in \mathbb{R}^{N \times 3}$ and a 3D scene graph $G=(N,E)$ using ConceptGraphs. In this graph, nodes represent objects with 3D pose, instance mask, and object-centric images, while edges encode spatial relations such as “inside” and “on top of” (Wu et al., 1 Sep 2025).

Given a free-form instruction $T$, an LLM decomposes the task into subtasks: $$\{(T_1, o_1), \dots, (T_n, o_n)\} = \text{LLM}(T, G),$$ where each subtask $T_k$ is paired with a target object $o_k$ in the scene graph. This decomposition lets MoTo operate as a planner for docking and interaction alignment while leaving the actual short-horizon manipulation to a fixed-base manipulation model (Wu et al., 1 Sep 2025).

The plug-in design is a defining feature. MoTo expects language instructions, RGB-D streams, and robot state, and outputs a base trajectory, a terminal docking pose, and a feasible arm configuration at docking. The downstream manipulation stack can be heuristic, diffusion-based, or a generalist vision-language-action model. In the reported system, MoTo is combined with AnyGrasp, iDP3, OpenVLA, and RDT-1B without modifying their internal architectures (Wu et al., 1 Sep 2025).

This division of labor is also how later work situates MoTo. UniLM-Nav characterizes it as a training-free or zero-shot method that uses vision foundation models and MLLMs for open-vocabulary last-mile navigation and identifies it as the previous state-of-the-art last-mile navigation method on the HomeRobot OVMM benchmark (Zhang et al., 7 Jul 2026).

3. Interaction keypoints and multi-view grounding

MoTo’s core representation is the pair of interaction keypoints. The target keypoint $P_T$ is a 3D point on the target object that is actionable for the current subtask, such as a fridge handle or a contact point on a surface. The arm keypoint $P_A$ is the manipulator point that should make contact, such as the end-effector or tool tip (Wu et al., 1 Sep 2025).

Target keypoint generation is a two-stage process. First, MoTo proposes candidate keypoints in each object-centric image using DINOv2 features and SAM masks. Cluster centers of DINOv2 features, normalized with depth information, serve as proposal points. Second, a VLM selects the most relevant keypoint or top $V$ keypoints conditioned on the subtask language. This procedure is repeated across multiple views $P \in \mathbb{R}^{N \times 3}$0 of the target object (Wu et al., 1 Sep 2025).

The selected 2D keypoints are then projected into the global point cloud, producing

$P \in \mathbb{R}^{N \times 3}$1

MoTo resolves view inconsistency by voting over the scene point cloud: $P \in \mathbb{R}^{N \times 3}$2 where $P \in \mathbb{R}^{N \times 3}$3 is the $P \in \mathbb{R}^{N \times 3}$4-th point in the scene point cloud and $P \in \mathbb{R}^{N \times 3}$5 is a Euclidean distance threshold. The point with the maximum votes becomes the final target keypoint in world coordinates (Wu et al., 1 Sep 2025).

Arm keypoint generation is analogous but uses wrist-camera images $P \in \mathbb{R}^{N \times 3}$6. If no specific arm keypoint is returned, MoTo defaults to the end-effector position. For dual-arm robots, the VLM is used as a binary classifier to select which arm should be used based on distances to the target and whether a gripper is empty or holding an object. The arm keypoint is transformed into world coordinates via

$P \in \mathbb{R}^{N \times 3}$7

where $P \in \mathbb{R}^{N \times 3}$8 is the robot-to-world transformation (Wu et al., 1 Sep 2025).

This keypoint-centric design is what makes MoTo interaction-aware. It does not optimize for nearness to an object in the abstract, but for alignment between the target part that matters for the task and the manipulator part that will physically interact with it (Wu et al., 1 Sep 2025).

4. Optimization-based navigation and motion planning

Once the target and arm keypoints are known, MoTo optimizes base and arm trajectories to minimize their separation while maintaining physical feasibility. The main objective is

$P \in \mathbb{R}^{N \times 3}$9

with

$G=(N,E)$0

The transformed base and arm actions are represented by

$G=(N,E)$1

and Pinocchio IK is used to translate desired keypoint displacements into joint-angle changes (Wu et al., 1 Sep 2025).

The collision term samples query points on the robot surface and penalizes any point that enters a safety margin around the scene point cloud: $G=(N,E)$2 The smoothness term penalizes abrupt changes in base pose and arm configuration: $G=(N,E)$3 The margin term keeps the arm in a comfortable operating radius at docking: $G=(N,E)$4 where $G=(N,E)$5 is the horizontal distance between the end-effector and the base center (Wu et al., 1 Sep 2025).

MoTo uses Dual Annealing to optimize these objectives. The optimizer samples candidate base-and-arm states, evaluates the objective, accepts or rejects proposals according to the annealing schedule, and iterates until the improvement falls below a threshold. In this way, docking is generated by inference-time optimization rather than task-specific training (Wu et al., 1 Sep 2025).

5. Empirical performance and operating regime

MoTo is evaluated on the OVMM benchmark and in real-world mobile manipulation experiments. In OVMM, it is tested as a drop-in replacement for the last-mile navigation component within a HomeRobot-style pipeline. The reported comparison includes Home-Robot baselines, L3MVN-based navigation, and MoTo combined with OpenVLA. The strongest reported OVMM result for MoTo is obtained by OpenVLA with MoTo, reaching an Overall Success Rate of $G=(N,E)$6, which is $G=(N,E)$7 higher than the corresponding OpenVLA with L3MVN configuration; Home-Robot with MoTo reaches $G=(N,E)$8, which is $G=(N,E)$9 higher than Home-Robot (RL) (Wu et al., 1 Sep 2025).

The real-world system uses a HEXMOVE wheeled platform, dual PiPER robotic arms, an Orbbec Femto Bolt head RGB-D camera, two Orbbec Gemini 336L wrist RGB-D cameras, and an Intel T265 tracking camera. GroundingSAM is used for object detection, GPT-4o for task planning, and ConceptGraph for scene graph construction. Tasks include “Bring me food”, “Serve me water”, and “Prepare a meal”, with evaluation based on success rate and task completion rate. Across ten runs per task, MoTo consistently improves both success rate and completion rate over L3MVN-style navigation for AnyGrasp, iDP3, and RDT-1B, and the reported real-world success-rate improvement over state-of-the-art mobile manipulation methods is $T$0 (Wu et al., 1 Sep 2025).

Ablation results identify the multi-view fusion and margin term as important. Removing fusion reduces Overall SR to $T$1; using only a single view yields $T$2; removing the margin cost yields $T$3. These results indicate that multi-view voting is not an auxiliary convenience but a major contributor to robust target-keypoint localization, and that arm operating margin is important for successful pick-and-place execution (Wu et al., 1 Sep 2025).

Later comparison on OVMM further places MoTo as a strong zero-shot baseline: FindObj $T$4, Pick $T$5, FindRec $T$6, and Overall SR $T$7. In that comparison, UniLM-Nav reaches $T$8, exceeding MoTo by $T$9 percentage points (Zhang et al., 7 Jul 2026).

6. Limitations, interpretation, and subsequent developments

MoTo is explicitly not a full whole-body mobile manipulation policy. It assumes an initial scene scan and static reconstruction, and the reported limitations include no online scene reconstruction, limited whole-body control, strong observability assumptions, dependence on VLM quality and SLAM, and nontrivial computational overhead from Dual Annealing and multi-view VLM calls (Wu et al., 1 Sep 2025). Failures observed in analysis include SLAM-induced localization errors, optimization loops, insufficient smoothing, and operation failures caused by viewpoint shifts that the manipulation policy cannot absorb (Wu et al., 1 Sep 2025).

A common misconception is to treat MoTo as ordinary object-goal navigation. Its actual function is narrower and more specific: it is a last-mile docking and trajectory-optimization layer that positions the robot so that a fixed-base policy can succeed. Another misconception is to treat it as an end-to-end learned mobile manipulation model; in fact, its mobile manipulation capability is derived from inference-time keypoint grounding and optimization, while the underlying manipulation competence is imported from pre-trained fixed-base systems (Wu et al., 1 Sep 2025).

Subsequent work clarifies both MoTo’s importance and its limits. UniLM-Nav describes MoTo as effective for interaction-centric navigation, but under-exploring task-aware spatial-relation reasoning. In that account, MoTo operates primarily at object level, while later systems explicitly decompose last-mile navigation into view selection, task-conditioned affordance grounding, and geometry-aware base-pose reasoning. This suggests that MoTo occupies a transitional position in zero-shot last-mile navigation: it established interaction-aware docking as a scalable plug-in abstraction, and later work extended that paradigm with more explicit affordance and geometry modeling (Zhang et al., 7 Jul 2026).