Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Luban: Building Open-Ended Creative Agents via Autonomous Embodied Verification (2405.15414v1)

Published 24 May 2024 in cs.AI

Abstract: Building open agents has always been the ultimate goal in AI research, and creative agents are the more enticing. Existing LLM agents excel at long-horizon tasks with well-defined goals (e.g., `mine diamonds' in Minecraft). However, they encounter difficulties on creative tasks with open goals and abstract criteria due to the inability to bridge the gap between them, thus lacking feedback for self-improvement in solving the task. In this work, we introduce autonomous embodied verification techniques for agents to fill the gap, laying the groundwork for creative tasks. Specifically, we propose the Luban agent target creative building tasks in Minecraft, which equips with two-level autonomous embodied verification inspired by human design practices: (1) visual verification of 3D structural speculates, which comes from agent synthesized CAD modeling programs; (2) pragmatic verification of the creation by generating and verifying environment-relevant functionality programs based on the abstract criteria. Extensive multi-dimensional human studies and Elo ratings show that the Luban completes diverse creative building tasks in our proposed benchmark and outperforms other baselines ($33\%$ to $100\%$) in both visualization and pragmatism. Additional demos on the real-world robotic arm show the creation potential of the Luban in the physical world.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Do as i can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691, 2022.
  2. Video pretraining (vpt): Learning to act by watching unlabeled online videos. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  3. Groot: Learning to follow instructions by watching gameplay videos. In The Twelfth International Conference on Learning Representations, 2024.
  4. Blender Online Community. Blender - a 3d modelling and rendering package, 2018.
  5. A.E. Elo. The USCF Rating System: Its Development, Theory, and Applications. United States Chess Federation, 1966.
  6. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  7. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  8. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05371, 2023.
  9. Geometric deep learning for computer-aided design: A survey, 2024.
  10. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  11. Instruct2act: Mapping multi-modality instructions to robotic arm actions with large language model, 2024.
  12. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning, 2022.
  13. Holodiffusion: Training a 3D diffusion model using 2D images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
  14. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4), jul 2023.
  15. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  16. Free2cad: Parsing freehand drawings into cad commands. ACM Trans. Graph. (Proceedings of SIGGRAPH 2022), 41(4):93:1–93:16, 2022.
  17. Advances in 3d generation: A survey, 2024.
  18. Code as policies: Language model programs for embodied control. In Workshop on Language and Robotics at CoRL 2022, 2022.
  19. Steve-1: A generative model for text-to-behavior in minecraft (abridged version). In NeurIPS 2023 Workshop on Goal-Conditioned Reinforcement Learning, 2023.
  20. Mcu: A task-centric framework for open-ended agent evaluation in minecraft. In Second Agent Learning in Open-Endedness Workshop, 2023.
  21. A comprehensive survey on 3d content generation, 2024.
  22. Freecad for osh automated documentation, February 2023.
  23. Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM, 65(1):99–106, dec 2021.
  24. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
  25. OpenAI. Gpt-4 technical report, 2023.
  26. Self-driven grounding: Large language model agents with automatical language-aligned skill learning. ArXiv, abs/2309.01352, 2023.
  27. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023.
  28. PrismarineJS. Mineflayer: Create minecraft bots with a powerful, stable, and high level javascript api, also usable from python. https://github.com/PrismarineJS/mineflayer, 2013.
  29. Variational inference with normalizing flows. In International conference on machine learning, pages 1530–1538. PMLR, 2015.
  30. 3d-GPT: Procedural 3d modeling with large language models, 2024.
  31. Llama 2: Open foundation and fine-tuned chat models, 2023.
  32. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  33. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024.
  34. Describe, explain, plan and select: Interactive planning with llms enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  35. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv: 2311.05997, 2023.
  36. Hierarchical neural coding for controllable cad model generation. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
  37. Creative agents: Empowering agents with imagination for creative tasks, 2023.
  38. Cadparser: A learning approach of sequence modeling for b-rep cad. In Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 1804–1812. International Joint Conferences on Artificial Intelligence Organization, 8 2023. Main Track.
  39. Ghost in the minecraft: Hierarchical agents for minecraft via large language models with text-based knowledge and memory, 2024.

Summary

  • The paper presents a novel AI agent that uses autonomous embodied verification to accomplish creative Minecraft building tasks.
  • It employs a two-stage process: generating 3D CAD models with visual verification and ensuring functional construction through pragmatic verification.
  • Luban achieved superior results with high quality ratings and a 100% pragmatic verification pass rate in diverse building challenges.

Luban: An AI Agent for Creative Minecraft Building Tasks

Introduction

Have you ever wondered if AI could handle creative tasks that lack clear-cut goals? Well, that's precisely what the Luban agent addresses. Traditional AI agents thrive on tasks with well-defined objectives, like mining diamonds in Minecraft. But when it comes to more inventive tasks, such as building a structurally sound and visually appealing house, the goals are often abstract. Luban is designed to bridge this gap by introducing autonomous embodied verification techniques inspired by human design practices.

Overview of Luban's Approach

Luban is an AI agent designed to perform creative building tasks in Minecraft without predefined goals. It does this through two stages of autonomous embodied verification:

  1. 3D Structural Speculation with Visual Verification: This stage uses CAD programming to create 3D models based on initial task instructions. The visual aspects of these models are then verified using Visual LLMs (VLMs).
  2. Construction with Pragmatic Verification: Once the structure is visually verified, the next step involves constructing it within the Minecraft environment. After the construction, the agent verifies the functionality of the structure, such as ensuring that doors open properly or that a bridge is walkable.

Key Features

3D Structural Speculation

The first stage involves:

  • Decomposing: Breaking down the task into smaller, manageable subcomponents.
  • Subcomponent Generation: Converting these subcomponents into 3D CAD models.
  • Assembling: Putting the subcomponents together into a complete 3D object.

Visual Verification

Multiple CAD models are generated and evaluated to filter out inappropriate designs. This ensures that only the most accurate models proceed to the next stage.

Pragmatic Verification

Here, the focus is on ensuring that the structure is functional within the Minecraft environment. For example, it verifies if players can enter the house through doors or cross a bridge without falling off.

Significant Numerical Results

Luban was tested on five different Minecraft tasks: arrow tower, bridge, Chinese ancient house, stair, and two-story house. The results were impressive:

  • Quality Ratings: Luban received high ratings across various dimensions, such as Appearance, Complexity, and Aesthetics, outperforming other baselines.
  • Elo Ratings: In one-to-one comparisons against other methods, Luban achieved an almost perfect winning rate with an Elo rating significantly higher than others.
  • Pragmatic Verification Pass Rates: Luban had a 100% pass rate in pragmatic verification, indicating that its creations were highly functional.

Implications

Practical Implications

Luban's techniques could revolutionize the way AI handles creative tasks not just in digital environments like Minecraft but also in real-world applications. For example:

  • Architecture: Imagine AI autonomously designing and verifying the functionality of complex structures.
  • Robotics: Luban's methodologies could be adapted for use in robotics, enabling more intelligent and adaptive machines.

Theoretical Implications

From a theoretical standpoint, Luban’s autonomous verification approach contributes significantly to the development of more advanced AI. It opens the door to creating agents capable of handling abstract, open-ended tasks that require iterative feedback and improvement.

Future Developments

The paper suggests several promising avenues for future research:

  • Real-World Applications: Extending Luban’s pragmatic verification to real-world tasks, potentially leading to AI agents that can perform creative building tasks in physical environments.
  • Enhanced Libraries: Developing extensive libraries that can bridge VLMs with the physical world, fostering the emergence of agents with spatial intelligence.

Final Thoughts

Luban is a step forward in making AI more adept at handling creative, open-ended tasks. By incorporating human-like iterative verification and refinement processes, it shows promise in both virtual and real-world applications. As research continues, we can expect even more sophisticated and functional AI agents in various domains.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown