YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals (2406.16273v1)

Abstract: 3D generation guided by text-to-image diffusion models enables the creation of visually compelling assets. However previous methods explore generation based on image or text. The boundaries of creativity are limited by what can be expressed through words or the images that can be sourced. We present YouDream, a method to generate high-quality anatomically controllable animals. YouDream is guided using a text-to-image diffusion model controlled by 2D views of a 3D pose prior. Our method generates 3D animals that are not possible to create using previous text-to-3D generative methods. Additionally, our method is capable of preserving anatomic consistency in the generated animals, an area where prior text-to-3D approaches often struggle. Moreover, we design a fully automated pipeline for generating commonly found animals. To circumvent the need for human intervention to create a 3D pose, we propose a multi-agent LLM that adapts poses from a limited library of animal 3D poses to represent the desired animal. A user study conducted on the outcomes of YouDream demonstrates the preference of the animal models generated by our method over others. Turntable results and code are released at https://youdream3d.github.io/

• The paper introduces a diffusion-based text-to-3D pipeline that integrates 2D pose control to achieve anatomically consistent animal models.
• It leverages TetraPose ControlNet and a multi-agent LLM to automatically generate and refine 3D poses from textual descriptions.
• Experimental results and user studies demonstrate that YouDream outperforms existing baselines with high-quality, geometrically accurate outputs.

An Insight into "YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals"

The paper, "YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals," authored by Sandeep Mishra, Oindrila Saha, and Alan C. Bovik, introduces an innovative approach to generating 3D animal models that are guided by textual descriptions and 3D poses. This method leverages a text-to-image (T2I) diffusion model and integrates it with a mechanism for controlling the output using 2D views of 3D poses, thereby addressing limitations in existing methods of text-to-3D generation.

Methodology Overview

YouDream fundamentally rests upon the integration of text guided generative models with precise 3D pose control. Key contributions include:

1. TetraPose ControlNet: A network trained specifically on tetrapod animals across various families to facilitate the generation of anatomically consistent animals. This model utilizes a 2D pose-controlled diffusion model that ensures the generated images adhere to 2D views of 3D poses.
2. Multi-agent LLM: A multi-agent LLM setup that automates the generation of 3D poses for novel animals. This framework consists of three agents: Finder, Observer, and Modifier. Together, these agents adapt poses from a limited library of animal 3D poses to represent desired novel animals.
3. Pipeline for 3D Generation: An entire automated pipeline for creating geometrically and anatomically consistent animals based on textual descriptions. This includes a user-friendly tool for creating or modifying 3D poses and an initial shape generator to ensure more accurate NeRF initialization.

Experimental Outcomes

A series of experiments highlight the robustness and efficiency of YouDream in generating high-quality, anatomically plausible 3D animal models:

Common Animal Generation: The comparison with baselines (HiFA, Fantasia3D, and 3DFuse) shows that YouDream excels in creating geometrically consistent animals. Images generated with YouDream show remarkable anatomical consistency compared to the baselines, which often produce distorted or incomplete animals.

Unreal Creatures: YouDream's ability to generate non-existent creatures is particularly noteworthy. This capability relies on its pose-editing tool and the multi-agent LLM framework facilitating a broader range of creative outputs that cannot be purely expressed through text prompts. Results show geometrically consistent, imaginative creatures following detailed skeletal structures.

User Study: A formal user paper was conducted to evaluate "Naturalness" and "Text-Image Alignment." Users overwhelmingly preferred YouDream, with a significant margin over other models, indicating its superior perceptual quality and alignment with textual descriptions.

Ablations: Various ablations presented in the paper underscore the importance of components like the initial shape and pose control, which contribute significantly to the anatomical accuracy and geometric consistency of the generated models.

Style Variation: The method's flexibility extends to generating animals with varied styles, facilitated through control scheduling and guidance scheduling strategies. This ensures consistent geometry initially and varied sufficient stylistic details later during training.

Implications of Research

The practical and theoretical implications of YouDream are broad and compelling:

1. Practical Applications: YouDream presents a powerful tool for artists, game developers, and AI enthusiasts to create and manipulate 3D animal models with a high degree of control and fidelity. The ability to produce anatomically consistent animals or imaginative creatures opens new avenues in digital content creation.
2. Theoretical Contributions: This research underscores the potential of combining LLMs and 3D generative models to improve consistency and control in generated assets. It demonstrates the importance of structured guidance (via poses) in overcoming the limitations of textual descriptions alone.

Future Developments

The paper suggests several avenues for future work:

• Improving Detail and Sharpness: Integrating additional techniques to enhance sharpness and fine details in the generated models could further improve the visual realism of the assets produced by YouDream.
• Expanding Pose Libraries: Increasing the diversity and number of pre-defined 3D pose libraries can enhance the tool's ability to generate more varied and complex animal models.
• Shortening Training Time: Optimizing the computational efficiency of the model to reduce training time while maintaining model quality offers significant practical benefits.

In conclusion, YouDream represents an impressive advancement in anatomically controllable 3D animal generation, offering robust performance against existing baselines and providing significant practical and theoretical contributions to the field of AI-driven content creation.