- The paper introduces a training-free, three-stage MLLM framework that selects informative views, grounds task-specific affordances, and computes geometry-aware base poses for zero-shot last-mile navigation.
- UniLM-Nav achieves 23.77% Overall Success Rate on the HomeRobot OVMM validation set, outperforming MoTo at 20.64% and training-based MoManipVLA at 15.80%.
- The results show that explicit 3D geometry, decomposed MLLM calls, and geometrically computed headings improve performance, while spatial reasoning for relational placement remains the primary challenge.
Motivation and problem setting
In open-vocabulary mobile manipulation (OVMM), object-goal navigation typically terminates once the target lies within a coarse near-target radius of 1–2 meters. This proximity does not guarantee a manipulation-ready base pose—one that provides reachability, orientation, and clearance for the intended manipulation. The paper addresses this "last-mile navigation" problem: given egocentric RGB-D observations, proprioceptive state, task instructions, and an obstacle map, select a collision-free, reachable base pose b=(x,y,θ) suitable for the downstream manipulation.
Prior approaches fall into two camps. Training-based methods—monolithic navigation-manipulation policies or learned distributions over manipulation-conditioned poses—are data-intensive and generalize poorly to open-vocabulary settings. Zero-shot methods such as MoTo leverage vision foundation models and MLLMs but rely on object-level cues, which the authors argue are insufficient for fine-grained spatial constraints (e.g., placing a bottle in front of a monitor requires identifying a receptacle region satisfying a spatial relation, not merely localizing the monitor). UniLM-Nav targets this gap with a fully zero-shot, training-free framework built on a single shared MLLM backend.
Method
UniLM-Nav decomposes last-mile navigation into three MLLM-driven stages:
View selection. The framework maintains a short-term memory buffer over the last K=5 observations before object navigation terminates, each paired with its robot state. The MLLM selects one reference view according to two criteria: clear visibility of the target object or receptacle, and evidence of a feasible approach path. This matters because the terminal observation may be occluded or lack spatial context for manipulation.
Task-conditioned affordance grounding. Given the selected view and instruction, the MLLM predicts a normalized image-space affordance point (u,v)—a graspable region for pick tasks, or a safe, unoccupied interior placement location on the receptacle surface for place tasks. This goes beyond object-level localization by requiring joint reasoning over semantics, layout, and manipulation constraints.
Geometry-aware base-pose reasoning. Rather than asking the MLLM to visually infer metric distances from raw pixels—a capability the authors identify as unreliable—the grounded 2D point is lifted to a robot-centric 3D affordance coordinate using aligned depth and camera parameters. The affordance point is also visually marked on the image as a visual prompt. Conditioned on this marked observation, the lifted 3D target, robot configuration, and task instruction, the MLLM predicts a base position (x,y) in the local frame; the heading is computed geometrically by orienting toward the affordance point rather than predicted by the model. The pose is transformed to the global frame and executed by a low-level navigation policy. For placement, the same call additionally predicts arm extension and lift height.
Benchmark results
On the HomeRobot OVMM validation set, UniLM-Nav with Gemini-3-Flash-Preview achieves 23.77% Overall SR, surpassing the previous state-of-the-art zero-shot method MoTo (20.64%) by 3.13 percentage points and the training-based MoManipVLA (15.80%)—notable given UniLM-Nav is entirely training-free. It also attains the highest FindObj success rate (69.47%), which the authors attribute to more reliable target approach enabled by the last-mile strategy. A lightweight RoboBrain-2.5-4B backend still reaches 19.19% Overall SR, outperforming most baselines and offering a deployability/efficiency trade-off.
Ablations and analysis
Ablations on a scene-stratified 20% subset of OVMM establish that all components contribute:
| Variant |
Overall SR |
| Full UniLM-Nav (Gemini-3-Flash) |
25.42% |
| w/o view selection |
20.42% |
| w/o base-pose reasoning |
substantially lower |
| w/o last-mile navigation |
below 5% |
Removing last-mile navigation entirely collapses performance below 5%, confirming that turning toward the affordance is insufficient when the current pose is not physically executable. Removing geometry-aware base-pose reasoning—replacing it with direct visual grounding of a floor point—degrades both Pick and Overall SR, supporting the paper's central design claim that explicit geometric context outperforms implicit visual metric estimation.
The MLLM backend comparison yields two findings worth highlighting. First, within the Qwen3-VL family, scaling helps (3.75% at 4B to 15.83% at 32B), yet Qwen3-VL-235B-A22B-Instruct plateaus at 17.50%. Second, and more strikingly, RoboBrain-2.5-4B achieves 20.50% Overall SR, exceeding GPT-5.4 (19.17%) and the 235B-parameter Qwen3-VL variant, which the authors attribute to robotics-oriented embodied spatial-reasoning fine-tuning. Backend differences concentrate at the placement stage: FindObj rates are nearly identical across backends (~67–69%), while Overall SR varies widely, indicating placement is the discriminating bottleneck.
Appendix ablations reinforce the decomposition argument. Merging view selection and affordance grounding into a single MLLM call degrades Overall SR across all tested backends (e.g., −4.17 points for Gemini-3-Flash), suggesting current MLLMs cannot reliably perform both subtasks jointly. Geometrically computed heading outperforms MLLM-predicted heading by 8.34 points of Overall SR (25.42% vs. 17.08%). Finally, swapping in a thinking-mode model only for base-pose reasoning raises Overall SR from 9.58% to 20.83%, whereas adding an explicit CoT prompt to the instruct model does not (9.17%), and thinking mode actually hurts affordance grounding—indicating that extra deliberation benefits spatially demanding stages but not direct perception stages.
Error analysis and real-world deployment
Manual failure analysis shows navigation errors dominate failed episodes (61.0%), followed by last-mile failures (20.9%) and manipulation errors (18.1%). Last-mile failures split into three modes: affordance points predicted near receptacle boundaries causing unstable placement, view selections blocked by obstacles despite better alternatives existing in memory, and base poses predicted too close to walls, limiting manipulator workspace. These cases indicate that stage-specific embodied decision making remains difficult even for strong MLLMs.
Real-world experiments on a Unitree B2 quadruped with a 6-DoF Z1 arm and eye-in-hand Orbbec Gemini 335 camera, using LIO-SAM for odometry and ROS 2 Navigation2 for execution, evaluate four tasks repeated ten times each. Simple pick/place tasks succeed at 7/10 and 6/10, while tasks requiring fine-grained spatial reasoning—"in front of the monitor" and viewpoint-conditioned "bottom-left corner"—drop to 4/10 each (52.5% total). The authors explicitly identify MLLM spatial reasoning as the bottleneck here. Deployment also includes a re-grounding step with the wrist camera after reaching the base pose, compensating for odometry drift, depth noise, and calibration error—an acknowledgment that single-shot grounding is not robust end-to-end.
Limitations and open questions
The framework assumes object navigation delivers the robot to a state where the target appears in recent observations; active local exploration before last-mile navigation is left unaddressed. Failure modes in view selection, affordance grounding, and base-pose reasoning persist, and the authors propose prompting improvements and rechecking mechanisms without demonstrating them. Evaluation beyond OVMM and additional last-mile benchmarks remains future work. More fundamentally, the real-world results raise an open question: how far MLLM spatial reasoning can be improved for viewpoint-conditioned and relational placement constraints, since these currently halve success relative to simpler tasks.
Conclusion
UniLM-Nav demonstrates that a shared MLLM backend, organized into view selection, affordance grounding, and geometry-aware base-pose reasoning, can solve last-mile navigation zero-shot and achieve state-of-the-art OVMM performance without task-specific training. The ablations provide actionable design guidance—explicit geometric context beats visual inference, decomposed calls beat merged ones, and embodied fine-tuning can outweigh scale—and the real-world deployment validates practical applicability while exposing MLLM spatial reasoning as the principal remaining constraint.
