A Rapid Deployment Pipeline for Autonomous Humanoid Grasping Based on Foundation Models

Abstract: Deploying a humanoid robot to manipulate a new object has traditionally required one to two days of effort: data collection, manual annotation, 3D model acquisition, and model training. This paper presents an end-to-end rapid deployment pipeline that integrates three foundation-model components to shorten the onboarding cycle for a new object to approximately 30 minutes: (i) Roboflow-based automatic annotation to assist in training a YOLOv8 object detector; (ii) 3D reconstruction based on Meta SAM 3D, which eliminates the need for a dedicated laser scanner; and (iii) zero-shot 6-DoF pose tracking based on FoundationPose, using the SAM~3D-generated mesh directly as the template. The estimated pose drives a Unity-based inverse kinematics planner, whose joint commands are streamed via UDP to a Unitree~G1 humanoid and executed through the Unitree SDK. We demonstrate detection accuracy of [email protected] = 0.995, pose tracking precision of $σ< 1.05$ mm, and successful grasping on a real robot at five positions within the workspace. We further verify the generality of the pipeline on an automobile-window glue-application task. The results show that combining foundation models for perception with everyday imaging devices (e.g., smartphones) can substantially lower the deployment barrier for humanoid manipulation tasks.

• The paper demonstrates a rapid, fully integrated pipeline leveraging YOLOv8, SAM 3D, and FoundationPose to achieve near real-time, high-precision humanoid grasping with minimal manual annotation.
• It details a three-stage workflow—from offline asset preparation to online perception and sim-to-real deployment—that achieves [email protected] of 0.995 and sub-millimeter pose accuracy.
• The pipeline’s modular design and robust performance in tasks like grasping and glue-application highlight its potential for versatile, industrial-scale robotic manipulation.

Rapid Object Onboarding for Autonomous Humanoid Manipulation via Foundation Models

Pipeline Architecture

The proposed pipeline achieves rapid deployment of humanoid grasping tasks by integrating three foundational perception modules: (1) Roboflow-assisted annotation with YOLOv8 for object detection, (2) 3D mesh reconstruction via Meta SAM 3D, and (3) zero-shot 6-DoF pose estimation using FoundationPose. These components form a perception stack tightly coupled with a Unity-based control pipeline and the Unitree SDK, facilitating fast, data-efficient onboarding for new objects and manipulation tasks.

The three-stage deployment workflow consists of offline asset preparation, high-frequency online perception, and a sim-to-real execution loop. Offline, images are rapidly annotated leveraging SAM-based automation, YOLOv8n is fine-tuned on this lightweight dataset, and a high-fidelity object mesh is reconstructed from a single RGB photo. Online, RGB-D frames feed through YOLOv8 for detection and FoundationPose for pose tracking, with pose output made available over HTTP at 30 Hz. The Unity-based manipulation planner consumes these estimates, generating joint command streams transmitted to the physical robot via a C++ bridge.

Figure 1: The three-stage pipeline for rapid deployment of humanoid grasping.

Perception System Assessment

YOLOv8n, trained using minimally supervised Roboflow annotation, consistently achieves [email protected] of 0.995, [email protected]:0.95 of 0.858, precision of 1.0, and recall of 0.997 on held-out validation data. The total model size is 3M parameters, with a 32 fps single-frame inference rate on CPU. Training time is approximately 11 minutes for 50 epochs, and model deployment overhead (including auto-annotation and mesh reconstruction) is minimal. These results confirm that foundation-model-powered pipelines can supplant decades-old, labor-intensive dataset curation and calibration techniques for object detection, even when constrained to commodity hardware.

Figure 2: YOLOv8n training losses and mAP curves demonstrate robust convergence and stability.

Figure 3: YOLOv8n predictions on the validation set show high accuracy across pose and viewpoint variation.

Figure 4: YOLOv8n confusion matrix indicating minimal confusion between target objects and background.

FoundationPose, using SAM 3D-generated meshes as templates, achieves sub-millimeter precision ($\sigma_{xyz} = 1.05$ mm) in static conditions and maintains 100% tracking success under both dynamic and partial occlusion conditions. Rotation error is dominated by object-axis ambiguity, which does not adversely affect grasping. End-to-end tracking throughput is maintained at >15 FPS (debug mode) and scalable to 30 FPS, meeting the closed-loop control rate needs for real-world bimanual manipulation.

End-to-End Robotic Execution

In a series of five grasping trials distributed through a workspace grid, the Unitree G1 demonstrates robust 6-DoF object grasp execution without any pipeline-specific retraining, as the pipeline accurately transfers perception outputs to a Unity-optimized IK-based motion planner and ultimately to the hardware execution stack. The sampling of joint-angle sequences from Unity simulation, without explicit sim-to-real fine-tuning or domain randomization, translates to stable grasp acquisition even with subtle real-world variation. The demonstrated extension to a window glue-application scenario, involving only class/model refinement and revised waypoint trajectories, substantiates the task-general applicability of the architectural paradigm.

Figure 5: Unity simulation of the bottle-grasping task, visualizing sim-to-real pipeline alignment.

Figure 6: Real-world Unitree G1 grasping demonstrations at multiple workspace locations.

Pipeline Generality: Glue-Application Extension

Task generality is rigorously tested via deployment on an automobile window glue-applicator task, replacing only the mesh and class definitions. The pipeline’s unchanged architecture proves sufficient for robust execution in a substantially altered manipulation context, with validation occurring first in Unity simulation and then in real hardware.

Figure 7: Overview of the gluing task sequence.

Figure 8: Unity simulation of the glue-application trajectory planning for window assembly.

Figure 9: Sample sequence of the physical execution of the glue-application task.

Numerical Outcomes and Deployment Efficiencies

The complete onboarding pipeline, from initial image capture to deployment-ready grasping, is accomplished within approximately 30 minutes per object. This includes 5 minutes for data acquisition, 5 minutes for annotation, 10 minutes for YOLO training, 5 minutes for mesh reconstruction, and minor overhead for Unity scene configuration. These metrics represent an expedited alternative to traditional procedures, which typically require days for manual annotation and laser-scanning workflows. The pipeline achieves strong perception and manipulation metrics without per-object or task-specific model reengineering.

Limitations and Future Directions

The pipeline's main limitations include the suboptimal mesh quality obtained by SAM 3D for reflective and transparent objects, which sometimes necessitates fallback on manual CAD modeling. Tracking robustness, although strong under partial occlusion, degrades upon prolonged or complete occlusions exceeding three seconds, due to the simplistic YOLO-triggered tracker reset mechanism. Precise quantification of the sim-to-real discrepancy, beyond empirical success rates, remains for future work. Promising directions include enhanced mesh reconstruction for challenging materials, multi-view pose reasoning, and analytical metrics for sim-to-real fidelity.

Theoretical and Practical Implications

This modular, data-efficient approach challenges vision-language-action model paradigms by decoupling perception and control, leveraging interpretable, foundation-model-based stacking for real-world deployment. The results indicate that practical, rapid robot deployment in unstructured environments can be achieved without heavy handcrafted datasets or specialized sensors, marking a theoretical advance in methods integration and practical reduction of barriers to robot manipulation in industrial and service domains.

Conclusion

The pipeline details a tightly integrated foundation-model-based strategy for rapid, high-reliability deployment of humanoid robotic manipulation, validated in grasping and assembly tasks on real hardware. Its robustness, speed, and minimal resource requirements set a new standard for practical robot adaptation, while open limitations invite further investigation into mesh quality and occlusion recovery. The modular structure invites extension to diverse manipulation contexts, positioning it as a reference platform for foundation-model-driven robotics research and industrial application.