Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset (2409.17126v1)

Abstract: Generative AI systems have shown impressive capabilities in creating text, code, and images. Inspired by the rich history of research in industrial ''Design for Assembly'', we introduce a novel problem: Generative Design-for-Robot-Assembly (GDfRA). The task is to generate an assembly based on a natural language prompt (e.g., ''giraffe'') and an image of available physical components, such as 3D-printed blocks. The output is an assembly, a spatial arrangement of these components, and instructions for a robot to build this assembly. The output must 1) resemble the requested object and 2) be reliably assembled by a 6 DoF robot arm with a suction gripper. We then present Blox-Net, a GDfRA system that combines generative vision LLMs with well-established methods in computer vision, simulation, perturbation analysis, motion planning, and physical robot experimentation to solve a class of GDfRA problems with minimal human supervision. Blox-Net achieved a Top-1 accuracy of 63.5% in the ''recognizability'' of its designed assemblies (eg, resembling giraffe as judged by a VLM). These designs, after automated perturbation redesign, were reliably assembled by a robot, achieving near-perfect success across 10 consecutive assembly iterations with human intervention only during reset prior to assembly. Surprisingly, this entire design process from textual word (''giraffe'') to reliable physical assembly is performed with zero human intervention.

• The paper introduces a novel generative design approach that converts natural language prompts into viable robotic assembly designs using advanced VLMs and iterative simulations.
• The paper integrates a six-axis robot with automated reset, achieving 99.2% correct block placements after simulation-based perturbation redesign.
• The paper demonstrates significant potential for autonomous manufacturing with a Top-1 design recognition accuracy of 63.5%, bridging conceptual design and physical execution.

Generative Design-for-Robot-Assembly (GDfRA) with Blox-Net

The paper introduces Blox-Net, a system designed to solve the Generative Design-for-Robot-Assembly (GDfRA) problem by leveraging advanced capabilities of generative AI, vision LLMs (VLM), and physics simulations. Recognizing the potential for generative AI systems in domains beyond traditional applications, the paper extends generative design paradigms to the field of robotic assembly, thereby defining the novel task of GDfRA. This task entails creating an assembly based on a natural language prompt and an image of available physical components, such as 3D-printed blocks.

Blox-Net Architecture

Blox-Net is constructed with three main phases, each introducing distinct innovations:

1. VLM Design Generation and Selection: This phase employs a VLM, specifically GPT-4o, to translate textual descriptions into viable assembly designs. Through iterative prompting, Blox-Net generates multiple design candidates, simulating their stability to refine selections. For example, when tasked with producing a structure resembling a "giraffe," the system elaborates on the prompt to understand essential features before devising a constructible design using the available components.
2. Simulation-Based Perturbation Redesign: Recognizing that idealized designs may face real-world physical constraints, this phase employs simulations to evaluate each design, factoring in robot constructability. Perturbation redesign iteratively enhances assembly reliability by adjusting elements to prevent collisions and instability during robotic execution.
3. Robotic Assembly and Evaluation: The final phase transfers the simulated designs to a physical setting, utilizing a six-axis robot arm. This involves grasping and placing blocks as per the refined design, and testing its constructability by automating the assembly process with a reset mechanism, enabling iterative testing with minimal human intervention.

Results and Implications

Blox-Net achieved a notable Top-1 accuracy of 63.5% for the VLM-based recognition of generated designs. The research revealed that the system reliably assembled complex structures across multiple iterations without human assistance during the assembly phase, scoring 99.2% in correct block placements post-simulated perturbation analysis. These results underscore the potential for automating design processes that bridge conceptual design (verbal descriptions) with physical execution (robotic assembly), highlighting a significant advancement in applying LLMs to physical tasks.

Theoretical and Practical Implications

The theoretical implications of this research illustrate an innovative intersection between natural language processing, AI-generated design, and robotic manipulation. The practical applications extend to autonomous manufacturing, where similar systems could revolutionize design cycles, decrease reliance on human oversight, and increase the adaptability and efficiency of industrial robotic systems. By demonstrating the feasibility of using VLMs in generating executable assembly plans, the paper opens pathways for further integration of AI in complex design and manufacturing tasks.

Future Directions

Speculative future developments could involve expanding Blox-Net's capabilities to work with a wider array of components, including deformable parts, and enhancing its design intuition to increase recognizability and fidelity for intricate designs. Additionally, future iterations might also incorporate more robust feedback loops between physical trials and machine learning models to improve adaptability and resilience in real-world environments. Further exploration could investigate the application of such systems in various industries, from automotive assembly to intricate architectural models, offering a vision for AI-driven assembly processes with little to no human intervention.

In summary, this work contributes a significant stride toward fully autonomous robotic assembly processes, balancing generative AI's creative capacities with practical constraints of robotic execution. Future research inspired by this paper can continue refining AI-driven design processes, potentially transforming robotic applications in manufacturing and beyond.