MoMani Benchmark: Mobile Manipulation

• MoMani Benchmark is a large-scale, automated system that defines long-horizon mobile manipulation tasks by integrating vision, language, and action into cohesive, multi-phase trajectories.
• It employs a three-stage pipeline—MLLM-guided expert trajectory planning, iterative feedback-driven refinement in simulation, and real-robot demonstration—to stress-test agent memory, planning, and closed-loop perception.
• The framework rigorously evaluates performance using metrics like success rate and mobile-manipulation score, emphasizing its capability to handle complex, diverse, and extended task sequences.

MoMani (Mobile Manipulation) Benchmark is an automated, large-scale framework for the training and evaluation of vision-language-action (VLA) agents on long-horizon mobile manipulation tasks. Introduced as a core contribution of the EchoVLA system, MoMani addresses the limitations of existing short-horizon, tabletop manipulation benchmarks by generating expert-level, multi-phase trajectories encompassing navigation, manipulation, and return behaviors in both simulated and real-world settings (Lin et al., 22 Nov 2025). The benchmark integrates multimodal LLM (MLLM)-guided planning, iterative feedback-driven refinement, and real-robot validation to support the study of memory, closed-loop perception, and complex planning in service-robot contexts.

1. Objectives and Scope

MoMani is purpose-built to stress-test VLA architectures under the conditions required for realistic mobile manipulation in unstructured environments. Traditional benchmarks have focused on isolated pick-and-place or short-horizon tasks, typically within static, designated workspaces. In contrast, MoMani mandates:

• Global Navigation: Locating and reaching objects distributed throughout a scene.
• Diverse Fine-Grained Interaction: Operations such as opening doors, actuating knobs, pulling drawers, and pushing or pulling objects.
• Return/Goal Navigation: Completing tasks by navigating to a specified endpoint (e.g., returning to the starting position).

Episodes are constructed to comprise all three phases within a single trajectory, producing average horizon lengths ranging from 630 to over 750 low-level action steps—orders of magnitude longer than in traditional settings. This length, combined with task variety and spatial context variation, places heavy demands on agent memory, planning, and closed-loop visual feedback (Lin et al., 22 Nov 2025).

2. Data Generation Workflow

MoMani’s dataset is synthesized through a three-stage pipeline:

1. Expert Trajectory Planning (MLLM-guided) Multimodal LLMs (MLLMs) are prompted with natural language task instructions (e.g., "Go to the cabinet, open the drawer, place the red block inside, then return to start.") and visual scene observations such as point-cloud images. The MLLM outputs a sequence of symbolic subgoals and macro-actions (e.g., NAVIGATE_TO, OPEN_HANDLE, GRASP_OBJECT, PLACE_AT). These are converted by a low-level motion planner into executable action sequences.
2. Feedback-Driven Trajectory Refinement Generated trajectories are executed in simulation (RoboCasa). A validation system flags failures (e.g., collision, failed grasp), automatically generating failure reports. The episode and failure report are fed back to the MLLM, prompting it to revise subgoals or motion parameters (e.g., reorient end-effector). This closed feedback loop iterates up to a fixed budget (typically $K=2$–$3$) or until the entire trajectory succeeds. The process is depicted in the following pseudocode:

$3$1

1. Real-Robot Demonstration Collection A representative subset of the refined simulated trajectories is transferred and executed on a real TidyBot++ mobile manipulator equipped with a RealSense D435i camera. Four household apparatuses are covered: refrigerator (upper/lower doors), side-opening microwave with knob, pull-out cabinet drawer, and sink rack, each with 30–50 real-world episodes recorded for dense, multi-view arm and object coverage.

3. Dataset Composition

MoMani is subdivided into:

• MoMani-Sim (Simulation)
  • TOF (Task: Open–Fetch)
  • PnPS2C (Pick-and-Place, Source→Counter)
  • PnPC2S (Pick-and-Place, Counter→Shelf)
  • TOS (Task: Open–Switch)

Each family contains approximately 1,000 successful trajectories (~4,000 total). Horizon lengths (average number of low-level action steps) are highest for PnPS2C at 755.56, with other families in the 630–700 range. Robocasa, by comparison, averages 278.00, and a navigation-only baseline 103.62.

• MoMani-Real (Real Robot) Includes four manipulation tasks: Open Drawer (OD), Close Microwave (CM), Turn Cabinet Knob (TC), Open Refrigerator (OR), with real-world horizon lengths from 126.2 to 191.0 steps per episode. An additional “Rotate Knob” (RK) evaluation consists of 20 episodes, supporting sim-to-real analysis.

Portion	Task Family / Task	Trajectories	Avg. Steps
MoMani-Sim	TOF	~1,000	630.99
	PnPS2C	~1,000	755.56
	PnPC2S / TOS	~1,000	650-700
MoMani-Real	OD, CM, TC, OR	30–50 each	126.2–191
	RK	20	--

4. Evaluation Protocols and Metrics

MoMani emphasizes two primary quantitative metrics:

• Success Rate (SR)

$$\mathrm{SR} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\text{episode}_i\, \text{ succeeded}\}$$

This measures the fraction of successfully completed episodes.

• Mobile-Manipulation Score ($\pi_\alpha$)

$$\pi_{\alpha} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl(d^{(i)}_{\mathrm{nav}} \leq \alpha \;\wedge\; d^{(i)}_{\mathrm{manip}} \leq \alpha\bigr)$$

Here, $d_\mathrm{nav}$ and $d_\mathrm{manip}$ are normalized navigation and manipulation errors, and $\alpha$ (commonly $0.5$) is the threshold for “closeness.” Success requires both the robot base and end-effector to reach task goals within margin.

Secondary analyses examine average horizon length and per-segment failure rates to diagnose bottleneck modalities (navigation vs. manipulation) (Lin et al., 22 Nov 2025).

5. Innovations and Distinctions from Prior Work

MoMani introduces several departures from previous mobile manipulation benchmarks:

• Extended Horizons: Tasks average $\sim$700 low-level steps, 2–3× longer than those in datasets such as Robocasa or BEHAVIOR/BH‐1k.
• Task Diversity: Episodes integrate multiple interaction modalities per trajectory (knobs, doors, drawers, pushing/pulling), going beyond isolated “pick-and-place.”
• MLLM-Driven Expert Generation: Multimodal LLMs synthesize expert demonstrations from textual and visual input, minimizing manual scripting overhead.
• Closed-Loop Feedback: Automated reports enable dynamic correction and denoising of expert demonstrations through iterative refinement.
• Real-World Validation: The MoMani-Real subset, using TidyBot++ and consumer household items, directly evaluates sim-to-real robustness across diverse interaction surfaces and physical parameters.

These design decisions produce tasks in which agents must recall long sequences of subgoals, perform fine-grained multi-view perception, and exhibit recovery strategies after intermediate failures.

6. Quantitative and Qualitative Benchmark Utility

Comparative results demonstrate MoMani’s challenge and utility:

• Horizon Comparison: PnPS2C trajectories (755.56 steps) are 2.7× longer than in Robocasa (278.00).
• Real-World Complexity: The Open Refrigerator (OR) task averages 191.0 steps, reflecting multi-joint, multi-phase operation.
• Success Rates: On the RK (Rotate Knob) real-robot task, MoMani’s scoring discriminates methods: baseline $3$0 0.40, diffusion policy 0.10, DP3 0.00, EchoVLA 0.50.
• Qualitative Behavior: Trajectories manifest navigation→interaction→return structures, engagement with complex affordances (rotary knobs, hinged doors), and emergent recovery maneuvering during or after failed grasps or misaligned approaches.

A plausible implication is that MoMani’s rich task structure and iterative data generation pipeline encourage the development of agents with more robust memory and planning faculties, distinct from those optimized on static, short-horizon, single-modality datasets. This is supported by performance gains in long-horizon metrics when integrating EchoVLA’s memory mechanisms (Lin et al., 22 Nov 2025).

7. Role in Advancing Embodied Intelligence Research

MoMani establishes a new standard for VLA benchmarking in mobile manipulation. By heavily compounding trajectory horizon, integrating diverse motor skills, leveraging closed-loop MLLM expert synthesis, and incorporating real-robot demonstration, it enables systematic evaluation of learning, memory, planning, perception, and transfer. The approach distinguishes itself from simulation-only, static-scene, or teleoperation-heavy datasets (see MobileManiBench (Wang et al., 5 Feb 2026) for a simulation-first alternative with a similar focus on scalability and controlled experimentation).

MoMani is therefore an essential testbed for investigating the intersection of multi-modal agent memory, language-conditioned planning, and real-world manipulation skill transfer. Its design elements position it as a critical resource for developing generalizable, robust, and data-efficient autonomous service robots in unconstrained domestic or industrial environments (Lin et al., 22 Nov 2025).