TiPToP: A Modular Open-Vocabulary Planning System for Robotic Manipulation

Abstract: We present TiPToP, an extensible modular system that combines pretrained vision foundation models with an existing Task and Motion Planner (TAMP) to solve multi-step manipulation tasks directly from input RGB images and natural-language instructions. Our system aims to be simple and easy-to-use: it can be installed and run on a standard DROID setup in under one hour and adapted to new embodiments with minimal effort. We evaluate TiPToP -- which requires zero robot data -- over 28 tabletop manipulation tasks in simulation and the real world and find it matches or outperforms $π_{0.5}\text{-DROID}$, a vision-language-action (VLA) model fine-tuned on 350 hours of embodiment-specific demonstrations. TiPToP's modular architecture enables us to analyze the system's failure modes at the component level. We analyze results from an evaluation of 173 trials and identify directions for improvement. We release TiPToP open-source to further research on modular manipulation systems and tighter integration between learning and planning. Project website and code: https://tiptop-robot.github.io

• The paper demonstrates that a modular integration of foundation models with GPU-parallelized task and motion planning achieves high success rates in multi-step robotic manipulation without robot-specific training data.
• It details a three-part architecture—perception, planning, and execution—that efficiently combines zero-shot scene interpretation with concrete trajectory generation to handle open-vocabulary tasks.
• Empirical evaluations reveal up to 100% success in distractor-rich scenarios and faster task completions, underscoring TiPToP's potential for scalable, cross-platform deployment.

TiPToP: A Modular Open-Vocabulary Planning System for Robotic Manipulation

System Architecture and Pipeline

TiPToP is proposed as a fully modular, extensible system for language-conditioned robotic manipulation that unifies contemporary foundation models for perception with efficient task and motion planning (TAMP) to execute complex, multi-step tasks over everyday objects and open-vocabulary language instructions. The system forgoes any robot training data or embodiment-specific fine-tuning, instead leveraging inference-time foundation models as perception frontends, while planning is centralized in a GPU-parallelized TAMP backend. The architecture is partitioned into three principal modules: (1) a Perception Module integrating vision and LLMs for scene interpretation and goal grounding, (2) a cuTAMP-based Planning Module for discrete-continuous task synthesis, and (3) an Execution Module responsible for accurate trajectory tracking.

Figure 1: System overview demonstrating the flow from stereo RGB images and language to planned robot trajectories.

Perception: Foundation Models and Open-World Semantics

The perception stack operates at $t=0$ (single-shot, open-loop), taking as input a calibrated stereo wrist-camera image pair and an unconstrained natural language command. The 3D Vision Branch utilizes FoundationStereo for dense, robust zero-shot depth estimation, outperforming device-specific stereo matching—specifically on specular, transparent, or low-texture surfaces—followed by unprojection to a world-frame-aligned scene point cloud.

Grasp prediction is performed via M2T2 on the full scene point cloud, providing 6-DoF candidate grasps, which are subsequently associated with object-level masks. The Semantic Branch queries a VLM (Gemini Robotics-ER 1.5) to extract precise 2D bounding boxes, object labels, and a formal goal graph (currently over $On(a, b)$ predicates), integrating vision and language for task-relevant object grounding and semantic disambiguation. Segment Anything Model v2 (SAM-2) refines bounding boxes to pixel-level segmentation, extracting individual object geometries.

Figure 2: Perception results with depth estimation (left), neural grasping confidence (middle), and semantic goal specification (right).

Object-centric scene construction combines instance masks with point cloud geometry, reconstructing watertight convex hull object meshes for downstream collision checks, and assigns predicted grasps using spatial proximity filtering.

Figure 3: SAM-2 segmentation masks generated from VLM-detected bounding boxes, supporting mesh extraction and perception-planning integration.

Planning: GPU-Accelerated Task and Motion Optimization

Task specification, in the form of a predicate conjunction over detected objects, is translated into a PDDL-style symbolic goal. cuTAMP enumerates feasible plan skeletons (e.g., via pick-and-place compositions and primitives), instantiates these skeletons by sampling continuous parameters from perception output and heuristics, and performs differentiable batch optimization over particle populations per skeleton to resolve discrete-continuous coupling and constraint satisfaction (e.g., collision-freedom, stable placements, kinematic feasibility).

Particles that survive optimization initiate parallel GPU motion planning via cuRobo, ultimately producing a timed joint-space and gripper trajectory for deterministic execution. The modular planning interface admits straightforward extension with additional primitives or symbolic predicates.

Execution: Trajectory Tracking and Open-Loop Limitations

The produced trajectory is tracked with a bespoke joint impedance controller, empirically found to yield higher precision than open-source alternatives such as DROID’s Polymetis. Importantly, the system executes open-loop (no visual feedback during or after primitive completion), meaning demarcation between perception, plan synthesis, and execution is strictly enforced. This determinism enables deep failure tracing but also exposes the system to errors from unrecovered execution failures (e.g., grasp reattempts are not supported in the current implementation).

Empirical Results and Comparative Evaluation

TiPToP is evaluated in 28 unique rearrangement and pick-and-place scenarios (simulation and real-world tabletop environments) via 165 trials, with extensive comparisons against $\pi_{0.5}$-DROID, a state-of-the-art vision-language-action (VLA) policy model fine-tuned on 350 hours of demonstration data on the same hardware. Critical evaluation axes included:

• Open-vocabulary semantic composition, i.e., instructions such as "serve peanut butter crackers on each tray," requiring nuanced cultural and referential reasoning.
• Generalization to distractor-rich, multi-step, and semantically ambiguous tasks.
• Zero-shot cross-embodiment deployment: The same TiPToP software stack was ported with minimal engineering effort to multiple robot platforms (e.g., UR5e, Trossen WidowX AI).

Strong numerical results were observed:

• On semantic, distractor, and multi-step tasks, TiPToP unambiguously outperforms $\pi_{0.5}$-DROID, e.g., success rates on hard distractor tasks up to 100% versus 20% for the VLA baseline, and in multi-step tasks 75% versus 52%.
• Task progress is systematically higher for TiPToP, and in many cases, non-successful runs still achieve most subgoals, evidencing robustness to partial failures.
• Average completion time for successful trials favors TiPToP: e.g., 14–18s for TiPToP versus 32–45s for $\pi_{0.5}$-DROID on real-world tasks.

Failure Analysis and Module Attribution

173 additional targeted experiments yielded granular failure diagnoses via module-level tracing:

Figure 4: Module-level failure analysis identifying grasping, mesh completion, VLM prediction, and TAMP optimization as bottlenecks.

• Grasping failures (e.g., non-contact or unstable picks) dominate (31/55 failures), caused by imperfect predictions of M2T2 (scene-level), heuristic grasp generation for missing objects, and lack of closed-loop re-attempting.
• Scene completion and mesh approximation errors, especially convex hulls over concave or occluded geometries (bananas), lead to infeasible plans or excessive collision conservatism.
• VLM errors (incorrect or missing detections/bounding boxes); these affect symbolic grounding and mask extraction.
• cuTAMP failures: inability to find feasible plans within compute budget, usually in heavily cluttered scenes.

The strengths of the modular design are most evident here: each failure is attributable to an independent block, supporting targeted research and engineering improvements.

System Modularity, Extensions, and Implications

The architectural separation facilitates both rapid extension and practical deployment. The addition of new low-level skills (e.g., whiteboard wiping, demonstrated in the paper) requires only local changes: new symbolic predicates, action primitives in TAMP, and semantic branch prompt extensions.

Figure 5: Extension to wiping (beyond pick-and-place), pairing language, perception, and motion control.

Furthermore, the ability to deploy on arbitrary robots is demonstrated, contingent only on URDF and controller integration—no retraining or perception/planning modifications are required. This supports credible claims of enhanced reproducibility and ease of benchmarking in robotics.

Theoretical and Practical Implications

TiPToP serves as direct evidence that late-binding modular systems, constructed from powerful foundation models and planners, can approach or exceed the task coverage of large, embodiment-specific, end-to-end VLA policies, without reliance on robot training data, demonstration collection, or joint policy tuning. The work highlights the tradeoff between explicit symbolic/geometric reasoning (supporting robust open-vocabulary and compound task generation; rapid debugging) and reactive closed-loop policies (supporting robust error recovery, compliant actuation, and real-world execution drift). The integrated approach advocated—modular planning systems informed by foundation models—sets a viable direction for tightly coupling learning and planning, and for benchmarking modular versus monolithic architectures.

For future work, the authors underscore:

• Closed-loop planning or belief-space TAMP to recover from execution mistakes and support reactivity;
• Improved shape reconstruction, e.g., multi-view perception, neural implicit meshes (SAM-3D);
• Training or adaptation of grasping policies for higher success on challenging objects;
• Hybridization with VLA policies as closed-loop reactive primitives;
• Automated predicate induction and policy abstraction via learning.

This direction is expected to underpin the next generation of generalizable, high-assurance robotic manipulation systems.

Conclusion

TiPToP establishes a new reference architecture for planning-based manipulation via modular integration of vision, language, and planning foundation models. The system demonstrates that high-level compositionality, transferable deployment, and open-vocabulary semantics are feasible at scale, without training data, and are competitive with (and often superior to) leading VLA systems—particularly in semantic, distractor, and multi-step tasks. The modular structure is vital for identifying, reproducing, and correcting system-level failures, and promises fluid adaptation as improved vision and policy backends emerge. The open-source release and cross-platform compatibility make TiPToP a robust foundation for future modular manipulation research and for exploring synergies between structured planning and data-driven control.