---
title: 'MoTo: Zero-Shot Interaction-Aware Navigation'
url: https://www.emergentmind.com/topics/moto
type: topic
---

# MoTo: Zero-Shot Interaction-Aware Navigation

MoTo (“Move and Touch”) is a zero-shot plug-in interaction-aware navigation module for general mobile manipulation. It is designed to be combined with any off-the-shelf manipulation foundation model to empower them with mobile manipulation ability by generating robot docking points from which fixed-base manipulation can be successfully performed. Rather than learning an end-to-end mobile manipulation policy from mobile manipulation expert data, MoTo uses pre-trained vision-language models, large language models, visual foundation models, scene reconstruction, and optimization-based motion planning to infer interaction-relevant keypoints and feasible base-and-arm trajectories in a zero-shot manner [2509.01658].

## 1. Problem formulation and design objective

MoTo addresses the “last-mile” problem in mobile manipulation: given a general manipulation policy that assumes a fixed base, how should a wheeled robot move to a docking position from which the arm can successfully execute the manipulation? The method is motivated by three limitations identified in prior systems. End-to-end mobile manipulation methods require large-scale mobile manipulation demonstration data. Fixed-base manipulation foundation models such as OpenVLA, RDT-1B, iDP3, and AnyGrasp cannot decide how the mobile base should move. Classical modular pipelines typically navigate “near” an object, but not necessarily to a pose that satisfies reachability, clearance, and downstream manipulation feasibility [2509.01658].

The formulation is posed as a joint optimization over base and arm trajectories:
\[
\min_{\mathbf{o}_{\text{base}}, \{\mathbf{o}_{\text{arm}}\}} O(\mathbf{o}_{\text{base}}, \{\mathbf{o}_{\text{arm}}\})
\quad \text{s.t. } \mathbf{o}_{\text{base}} \in \mathbb{R}^3,\ \mathbf{o}_{\text{arm}} \in \mathbb{R}^n.
\]
Here the base pose is optimized together with arm configurations so that a fixed-base manipulation policy can be invoked only after the robot has reached a manipulation-ready docking point [2509.01658].

A central distinction in MoTo is that navigation is explicitly interaction-aware. The target is not merely an object instance or receptacle location, but the part of the scene where the intended interaction should occur. This makes MoTo a last-mile navigation method rather than a complete mobile manipulation policy, and its zero-shot property refers specifically to the absence of mobile manipulation expert data for the navigation-and-docking component [2509.01658].

## 2. System architecture and plug-in workflow

MoTo begins with scene scanning and reconstruction. The robot collects RGB-D views, from which it builds a 3D point cloud \(P \in \mathbb{R}^{N \times 3}\) and a 3D scene graph \(G=(N,E)\) using ConceptGraphs. In this graph, nodes represent objects with 3D pose, instance mask, and object-centric images, while edges encode spatial relations such as “inside” and “on top of” [2509.01658].

Given a free-form instruction \(T\), an LLM decomposes the task into subtasks:
\[
\{(T_1, o_1), \dots, (T_n, o_n)\} = \text{LLM}(T, G),
\]
where each subtask \(T_k\) is paired with a target object \(o_k\) in the scene graph. This decomposition lets MoTo operate as a planner for docking and interaction alignment while leaving the actual short-horizon manipulation to a fixed-base manipulation model [2509.01658].

The plug-in design is a defining feature. MoTo expects language instructions, RGB-D streams, and robot state, and outputs a base trajectory, a terminal docking pose, and a feasible arm configuration at docking. The downstream manipulation stack can be heuristic, diffusion-based, or a generalist vision-language-action model. In the reported system, MoTo is combined with AnyGrasp, iDP3, OpenVLA, and RDT-1B without modifying their internal architectures [2509.01658].

This division of labor is also how later work situates MoTo. UniLM-Nav characterizes it as a training-free or zero-shot method that uses vision foundation models and MLLMs for open-vocabulary last-mile navigation and identifies it as the previous state-of-the-art last-mile navigation method on the HomeRobot OVMM benchmark [2607.06537].

## 3. Interaction keypoints and multi-view grounding

MoTo’s core representation is the pair of interaction keypoints. The **target keypoint** \(P_T\) is a 3D point on the target object that is actionable for the current subtask, such as a fridge handle or a contact point on a surface. The **arm keypoint** \(P_A\) is the manipulator point that should make contact, such as the end-effector or tool tip [2509.01658].

Target keypoint generation is a two-stage process. First, MoTo proposes candidate keypoints in each object-centric image using DINOv2 features and SAM masks. Cluster centers of DINOv2 features, normalized with depth information, serve as proposal points. Second, a VLM selects the most relevant keypoint or top \(V\) keypoints conditioned on the subtask language. This procedure is repeated across multiple views \(\{I_k^1,\dots,I_k^m\}\) of the target object [2509.01658].

The selected 2D keypoints are then projected into the global point cloud, producing
\[
P_K \in \mathbb{R}^{(V \times m) \times 3}.
\]
MoTo resolves view inconsistency by voting over the scene point cloud:
\[
V(P_K, P) = \arg\max_i \sum_{j=1}^{V \times m} \mathbb{I}\big( \| P(i) - P_K(j) \|_2 - \tau \big),
\]
where \(P(i)\) is the \(i\)-th point in the scene point cloud and \(\tau\) is a Euclidean distance threshold. The point with the maximum votes becomes the final target keypoint in world coordinates [2509.01658].

Arm keypoint generation is analogous but uses wrist-camera images \(\{s^w\}\). If no specific arm keypoint is returned, MoTo defaults to the end-effector position. For dual-arm robots, the VLM is used as a binary classifier to select which arm should be used based on distances to the target and whether a gripper is empty or holding an object. The arm keypoint is transformed into world coordinates via
\[
P_{A,t} = \mathcal{F}(\text{VLM}(T_k, \{s^w\}), E_t),
\]
where \(E_t\) is the robot-to-world transformation [2509.01658].

This keypoint-centric design is what makes MoTo interaction-aware. It does not optimize for nearness to an object in the abstract, but for alignment between the target part that matters for the task and the manipulator part that will physically interact with it [2509.01658].

## 4. Optimization-based navigation and motion planning

Once the target and arm keypoints are known, MoTo optimizes base and arm trajectories to minimize their separation while maintaining physical feasibility. The main objective is
\[
\arg\min_{ \{\mathbf{o}_{\text{base},t}, \{\mathbf{o}_{\text{arm},t}\}\}_{t=1}^T }
\sum_{t=1}^{T}
\left\| P_T - P_{A,t}(\mathbf{o}_{\text{base},t}, \{\mathbf{o}_{\text{arm},t}\}) \right\|_2
+ C_t(\mathbf{o}_{\text{base},t}, \{\mathbf{o}_{\text{arm},t}\}),
\]
with
\[
C_t(\mathbf{o}_{\text{base},t}, \{\mathbf{o}_{\text{arm},t}\}) = F_C + F_S + F_M.
\]
The transformed base and arm actions are represented by
\[
\mathbf{a}_{\text{base},t} = \mathcal{A}(\mathbf{o}_{\text{base},t}), \quad
\{\mathbf{a}_{\text{arm},t}\} = \mathcal{A}(\{\mathbf{o}_{\text{arm},t}\}),
\]
and Pinocchio IK is used to translate desired keypoint displacements into joint-angle changes [2509.01658].

The collision term samples query points on the robot surface and penalizes any point that enters a safety margin around the scene point cloud:
\[
F_C = \sum_{j=1}^{N_q} \max(0, \epsilon_0 - D(q_{j,t}, P)).
\]
The smoothness term penalizes abrupt changes in base pose and arm configuration:
\[
F_S =
\left\| \mathbf{o}_{\text{base},t+1} - \mathbf{o}_{\text{base},t} \right\|_2^2 +
\left\| \mathbf{o}_{\text{arm},t+1} - \mathbf{o}_{\text{arm},t} \right\|_2^2.
\]
The margin term keeps the arm in a comfortable operating radius at docking:
\[
F_M = \left| r_{\min} + r_{\max} - r_t \right|,
\]
where \(r_t = g(\mathbf{o}_{\text{arm},t})\) is the horizontal distance between the end-effector and the base center [2509.01658].

MoTo uses Dual Annealing to optimize these objectives. The optimizer samples candidate base-and-arm states, evaluates the objective, accepts or rejects proposals according to the annealing schedule, and iterates until the improvement falls below a threshold. In this way, docking is generated by inference-time optimization rather than task-specific training [2509.01658].

## 5. Empirical performance and operating regime

MoTo is evaluated on the OVMM benchmark and in real-world mobile manipulation experiments. In OVMM, it is tested as a drop-in replacement for the last-mile navigation component within a HomeRobot-style pipeline. The reported comparison includes Home-Robot baselines, L3MVN-based navigation, and MoTo combined with OpenVLA. The strongest reported OVMM result for MoTo is obtained by OpenVLA with MoTo, reaching an Overall Success Rate of \(20.64\%\), which is \(2.68\%\) higher than the corresponding OpenVLA with L3MVN configuration; Home-Robot with MoTo reaches \(18.32\%\), which is \(3.52\%\) higher than Home-Robot (RL) [2509.01658].

The real-world system uses a HEXMOVE wheeled platform, dual PiPER robotic arms, an Orbbec Femto Bolt head RGB-D camera, two Orbbec Gemini 336L wrist RGB-D cameras, and an Intel T265 tracking camera. GroundingSAM is used for object detection, GPT-4o for task planning, and ConceptGraph for scene graph construction. Tasks include “Bring me food”, “Serve me water”, and “Prepare a meal”, with evaluation based on success rate and task completion rate. Across ten runs per task, MoTo consistently improves both success rate and completion rate over L3MVN-style navigation for AnyGrasp, iDP3, and RDT-1B, and the reported real-world success-rate improvement over state-of-the-art mobile manipulation methods is \(16.67\%\) [2509.01658].

Ablation results identify the multi-view fusion and margin term as important. Removing fusion reduces Overall SR to \(15.19\%\); using only a single view yields \(17.61\%\); removing the margin cost yields \(17.87\%\). These results indicate that multi-view voting is not an auxiliary convenience but a major contributor to robust target-keypoint localization, and that arm operating margin is important for successful pick-and-place execution [2509.01658].

Later comparison on OVMM further places MoTo as a strong zero-shot baseline: FindObj \(66.67\%\), Pick \(60.95\%\), FindRec \(49.87\%\), and Overall SR \(20.64\%\). In that comparison, UniLM-Nav reaches \(23.77\%\), exceeding MoTo by \(3.13\) percentage points [2607.06537].

## 6. Limitations, interpretation, and subsequent developments

MoTo is explicitly not a full whole-body mobile manipulation policy. It assumes an initial scene scan and static reconstruction, and the reported limitations include no online scene reconstruction, limited whole-body control, strong observability assumptions, dependence on VLM quality and SLAM, and nontrivial computational overhead from Dual Annealing and multi-view VLM calls [2509.01658]. Failures observed in analysis include SLAM-induced localization errors, optimization loops, insufficient smoothing, and operation failures caused by viewpoint shifts that the manipulation policy cannot absorb [2509.01658].

A common misconception is to treat MoTo as ordinary object-goal navigation. Its actual function is narrower and more specific: it is a last-mile docking and trajectory-optimization layer that positions the robot so that a fixed-base policy can succeed. Another misconception is to treat it as an end-to-end learned mobile manipulation model; in fact, its mobile manipulation capability is derived from inference-time keypoint grounding and optimization, while the underlying manipulation competence is imported from pre-trained fixed-base systems [2509.01658].

Subsequent work clarifies both MoTo’s importance and its limits. UniLM-Nav describes MoTo as effective for interaction-centric navigation, but under-exploring task-aware spatial-relation reasoning. In that account, MoTo operates primarily at object level, while later systems explicitly decompose last-mile navigation into view selection, task-conditioned affordance grounding, and geometry-aware base-pose reasoning. This suggests that MoTo occupies a transitional position in zero-shot last-mile navigation: it established interaction-aware docking as a scalable plug-in abstraction, and later work extended that paradigm with more explicit affordance and geometry modeling [2607.06537].

Source: https://www.emergentmind.com/topics/moto