C3DAG: Pose-Controlled Text-to-3D Animal Generation

• C3DAG is a pose-controlled text-to-3D animal generation framework that utilizes a two-stage pipeline combining NeRF initialization and SDS refinement to ensure anatomical and geometric accuracy.
• It employs an automatic 3D shape creator tool with 18 keypoints to generate an initial 'balloon' mesh, providing a robust geometric prior for subsequent depth- and pose-guided optimization.
• The framework integrates text prompts with 2D pose control via a tetrapod-pose ControlNet, achieving rapid, detailed 3D animal asset production with minimal distortions.

The C3DAG framework is a pose-controlled text-to-3D animal generation system designed to produce anatomically and geometrically consistent animal models that adhere strictly to a specified 3D pose. It employs an efficient two-stage pipeline that integrates text-to-3D guidance with explicit geometric pose control, overcoming prior limitations in animal asset generation such as distorted anatomy and inconsistent geometry. Central innovations include the use of an automatic balloon animal mesh generator, depth- and pose-guided Score Distillation Sampling (SDS), NeRF initialization, and refinement via a tetrapod-pose ControlNet.

1. Framework Architecture and Pipeline

C3DAG utilizes a two-stage pipeline to generate controlled 3D animal models:

1. Stage 1 – NeRF Initialization
   • An automatic 3D shape creator tool, interfaced through a web-based platform, assembles a mesh using simple geometric primitives (spheres, cylinders, cones) around a set of 18 keypoints that define the skeletal structure of the animal.
   • The initial “balloon animal” mesh provides geometric prior for the subsequent stages.
   • Depth maps are generated from this mesh using a depth-guided ControlNet. These maps condition the SDS process, which is used to pre-train a Neural Radiance Field (NeRF) to produce plausible geometry.
2. Stage 2 – Pose-Controlled Refinement
   • The optimized mesh is refined further by projecting the keypoints into 2D pose control images from various camera angles. These serve as control signals for the tetrapod-pose ControlNet, trained on diverse animal keypoint datasets.
   • SDS gradients are computed based on both text prompt and 2D pose control, resulting in a NeRF fine-tuned for both appearance (from text) and anatomical pose (from 3D keypoints).
   • The iterative optimization uses a cosine annealing schedule to gradually shift from ControlNet-based guidance to classifier-free diffusion guidance, with $ω = ω_{init} \cdot (1 + \frac{iter}{total\_iters})$, where $ω_{init} = 50$.

The integration of these modules produces 3D assets that are visually detailed and pose-accurate.

2. Pose Specification and Control Mechanisms

Pose control in C3DAG is achieved through a parametric skeleton representation and 2D diffusion-guided conditioning:

• 3D Skeleton Representation: Each animal is encoded by 18 keypoints—including anatomical landmarks such as eyes, nose, neck, individual limbs, and tail end—and their bone connectivity. This abstraction generalizes across tetrapods (mammals, reptiles, birds, amphibians).
• Projection and ControlNet Conditioning: The 3D skeleton is projected into 2D images corresponding to user-specified viewpoints. These control images are provided to a tetrapod-pose ControlNet, whose output conditions diffusion-based generative modeling via SDS. Robustness is enhanced through extensive data augmentation (rotations, translations, scale changes) during ControlNet training.

The modified SDS gradient is formulated as:

$$\nabla_θ L_{SDS}(φ, x) = E_{t, ε} [w(t) \cdot (ε_φ(x_t; y, t, c) - ε) \cdot (\frac{\partial z_t}{\partial x}) \cdot (\frac{\partial x}{\partial θ})]$$

where $c$ denotes the 2D pose image used for conditioning.

This mechanism circumvents anatomical distortions and ensures the fidelity of the generated animal’s pose.

3. Automatic 3D Shape Creator Tool

The shape creator tool is a pivotal component for both pose specification and mesh instantiation:

• Interface and Functionality: Built on THREE.js, it offers an intuitive UI for users to manipulate the 18 keypoints interactively. After configuration, the tool generates a mesh by algorithmically assembling geometric primitives around the skeleton.
• Mesh Generation: Spheres represent the head, cones the nose, cylinders the limbs and tail. Mesh initialization is automatic upon user command (such as “create mesh” button).
• Role in Pipeline: The mesh acts as anatomical prior for NeRF training, providing initial geometric plausibility and a foundation for subsequent depth- and pose-based optimization.

This tool facilitates rapid, flexible pose specification and mesh initialization, obviating manual mesh modeling requirements.

4. Training Strategies: NeRF Initialization and SDS Refinement

C3DAG’s NeRF training is divided into distinct phases:

• Pre-training: Using depth maps from the shape creator tool and depth-controlled ControlNet, the NeRF is pre-trained via SDS, focusing on reconstructing volumetric geometry.
• Fine-tuning: The quadruped-pose-controlled SDS phase incorporates both text and pose signals. 2D pose projections are fed to the ControlNet, which through SDS, guides the optimization toward anatomical and geometric precision.

The combination of robust initialization and pose-driven fine-tuning reduces extraneous artifacts and ensures anatomical correctness. Model hyperparameters such as guidance weight $ω$ are scheduled to shift toward classifier-free guidance, enhancing model fidelity over training iterations.

5. Comparative Evaluation

C3DAG demonstrates notable improvements over existing text-to-3D and parametric model frameworks:

Framework	Anatomical Accuracy	Efficiency (A100/80GB)	Mesh Artifacts
C3DAG	High	~20 min	Minimal; pose-controlled
DreamFusion	Lower	(not specified)	Multiple heads/limbs; distortions
HiFA	Lower	~7 hours	Frequent anatomical inconsistencies
3DFauna	Moderate	(not specified)	Limited detail; suboptimal pose

C3DAG produces anatomically and geometrically consistent results, with rapid convergence and fine details such as tail structure or limb articulation. The pose control mechanism substantially reduces distortion errors present in prior frameworks.

6. Applications and Broader Implications

C3DAG’s capabilities have wide-ranging applications and implications:

• Industry Use Cases: Animation, gaming, and digital art benefit from fast, controllable animal asset generation, reducing manual modeling burdens.
• Simulation and VR: Accurate animal models enable immersive environments and biologically realistic training scenarios.
• Education and Research: Users can visualize animal structures and motions without reliance on expensive 3D scanning or manual mesh construction; applicable to biology, zoology, and veterinary education.
• Generalization Potential: The combination of geometric modeling and diffusion-guided control may extend to other articulated figures beyond tetrapod animals.

A plausible implication is that diffusion-guided, keypoint-conditioned text-to-3D pipelines such as C3DAG could become baseline methodologies for controllable 3D asset generation in other domains, provided adequate pose parameterizations are defined.

7. Integration with Model-Driven Component Systems

C3DAG’s pipeline demonstrates strengths in modularity and reusability central to component-and-connector (C&C) paradigms. By leveraging abstract model specification (3D skeleton, mesh initialization), late binding of configuration (text, pose, viewpoint), and generative development (multi-stage NeRF optimization), it aligns with principles from model-driven engineering approaches as articulated in related literature (Ringert et al., 2014). This suggests that C3DAG’s design could benefit further from explicit component model libraries and systematic transformation mechanisms, supporting future scalability and platform adaptation.