GANFusion: Text-to-3D Synthesis via Diffusion and GANs

• The paper introduces GANFusion, a groundbreaking framework that fuses GAN and diffusion techniques for text-driven 3D object synthesis using dual-training and feed-forward strategies.
• A dual training approach employs score distillation sampling with multi-view supervision while a feed-forward pipeline leverages latent triplane representations for efficient volumetric rendering.
• Experimental results demonstrate that GANFusion achieves superior texture fidelity, geometric accuracy, and prompt adherence without relying on test-time optimization.

Generative Adversarial Network Fusion (GANFusion) for Text-to-3D synthesis designates a family of frameworks that combine adversarial training and denoising diffusion probabilistic models to enable text-prompted generation of 3D objects—either through a dual training regimen employing large-scale text-to-image models with explicit multi-view rendering supervision (Chen et al., 2023), or via end-to-end feed-forward sampling in a latent triplane representation space learned with single-view 2D supervision (Attaiki et al., 2024). These models are motivated by the complementary strengths and weaknesses of GAN-based and diffusion-based generative approaches—GANs being efficient with 2D supervision and offering high-fidelity outputs but limited in conditional expressivity, and diffusion models providing more robust conditional sampling but suffering from optimization and supervision difficulties when constrained to 2D datasets.

1. Architectural Paradigms and Pipeline Structure

GANFusion in the context of text-to-3D synthesis is realized in two dominant architectures:

1. Diffusion–GAN Dual Training (within DreamFusion/IT3D): The process begins with a coarse 3D model generated by a text-to-3D diffusion-prior (e.g., DreamFusion). Explicit multi-view renderings of this model are refined via image-to-image diffusion models (such as ControlNet + Stable Diffusion) to synthesize a small, high-variance posed image dataset. A 3D-aware discriminator is trained to discriminate between these refined images ("real") and new renders from the current 3D model ("fake"). Model optimization jointly leverages gradients from both diffusion priors (score distillation) and adversarial losses, with regularizer terms inherited from Stable-DreamFusion. The training loop alternates discriminator and generator updates, decays the adversarial loss weight, and employs the discriminator to align multi-view distributions even under view inconsistency (Chen et al., 2023).
2. Feed-Forward Diffusion in GAN Space (two-stage triplane pipeline): The method trains an unconditional 3D-aware GAN (StyleGAN2 architecture) to generate triplane feature representations from Gaussian latent vectors using only single-view 2D images as supervision. The triplane features are decoded via a small MLP and rendered volumetrically (NeRF style), with an EG3D-like upsampler for higher-resolution outputs. Captioned renderings of GAN-generated triplanes are then used to train a diffusion model in triplane latent space, conditioned on text extracted via BLIP VQA captions. At inference, a feed-forward DDIM sampler produces triplanes conditioned on open-vocabulary prompts, which are decoded/rendered by the pretrained decoder modules (Attaiki et al., 2024).

2. Representation, Rendering, and Supervision

Both variants employ volumetric scene representations and GAN-based discriminators, but the feed-forward approach further refines the latent scene encoding:

• Triplane Representation: Feature tensors composed of three orthogonal 2D planes $$(T_{xy}, T_{xz}, T_{yz}) \in \mathbb{R}^{n \times h \times w}$$ are stacked to form $T \in \mathbb{R}^{3n \times h \times w}$. Queries at a 3D position $x$ are mapped by bilinear interpolation from each plane, concatenated, and processed by a 4-layer MLP to predict density and color (Attaiki et al., 2024).
• Volumetric Rendering: The MLP output $(d, c)$, with $d$ mapped to density $\sigma = \mathrm{sigmoid}(d)$, is rendered via a NeRF-style ray marcher across sample points with transmittance weighting $a_i = 1-\exp(-\sigma_i \delta_i)$, aggregating pixel colors by recursive alpha compositing.
• GAN Discrimination: For single-view 2D supervision, a StyleGAN2 adversarial loss is imposed over rendered images against a 2D image discriminator; no explicit 3D ground truth is used.

3. Loss Formulations and Training Strategies

Dual Training (Diffusion–GAN)

• Score Distillation Sampling (SDS): The DreamFusion gradient is used, with no explicit reconstruction loss:

$$\nabla_\theta \mathcal{L}_{\mathrm{SDS}}(\phi, g(\theta)) = \mathbb{E}_{t,\epsilon}\left[ w(t) \cdot (\epsilon_\phi(x_t; y, t) - \epsilon) \cdot \partial x / \partial \theta \right]$$

Here, $x = g(\theta)$, $x_t = x + \sigma(t)\epsilon$, and $y$ is the text embedding (Chen et al., 2023).

• Adversarial Loss:

$$\mathcal{L}_D = -\mathbb{E}_{x \sim D'}\left[\log D_{3D}(x)\right] - \mathbb{E}_{x_\text{fake} \sim g(\theta)}\left[\log (1-D_{3D}(x_\text{fake}))\right]$$

$$\mathcal{L}_{G_\mathrm{adv}} = -\mathbb{E}_{x_\text{fake} \sim g(\theta)}[\log D_{3D}(x_\text{fake})]$$

• Total Loss:

$$\mathcal{L}_\text{total}(\theta) = \mathcal{L}_{\mathrm{SDS}}(\theta) + \lambda_\text{adv}(t)\mathcal{L}_{G_\mathrm{adv}}(\theta) + \sum \lambda_\text{reg} \mathcal{L}_\text{reg}(\theta)$$

with $\lambda_\text{adv}$ linearly decayed during refinement.

Feed-Forward GANFusion (Diffusion in Triplane Latent Space)

• GAN Loss: Classic StyleGAN2 losses with single-view 2D render/discriminator pair.
• Diffusion Loss:

$$\mathcal{L}_\mathrm{diff} = \mathbb{E}_{t, Y, T^0, \epsilon}\left[ \| k_\chi(T^t, Y, t) - T^0 \|_2^2 \right]$$

where forward diffusion follows $$T^t = \sqrt{\alpha_t}T^0 + \sqrt{1-\alpha_t}\epsilon$$ for $\epsilon \sim \mathcal{N}(0, I)$, with a sigmoid noise schedule.

• Classifier-Free Guidance: Implemented by randomly substituting $Y \to \emptyset$ at training with probability 0.2, and applying interpolated guidance at sampling.

4. Data Preparation and Conditioning

• Dataset Generation: Models can be trained with only single-view 2D images, which may be synthetic (e.g., 300k images rendered with StableDiffusion 1.5, with SMPL pose/conditioning), or real, depending on the intended domain (Attaiki et al., 2024).
• Automatic Captioning: For diffusion model conditioning, synthetic triplane renderings are automatically captioned using BLIP VQA, filling prompts with attributes such as gender and clothing color. Captions are encoded via CLIP text encoders and injected through cross-attention into the U-Net diffusion model.

5. Experimental Validation and Comparative Results

Quantitative and qualitative results show that GANFusion mechanisms—either as a dual-training loop or as a latent-space diffusion process—achieve superior texture detailing, geometric correction, and prompt-faithful 3D synthesis relative to conventional approaches:

Method	FID (↓)	CLIP Score (↑)	Text-Control	Test-Time Optimization
AG3D (uncond., w/ up)	35.8	—	✗	—
AG3DC+text (CLIP loss)	~92	0.24	✓, low fidelity	—
RenderDiffusion	~136	0.263	✓, blurry	—
GANFusion (low-res)	~88	0.296	✓, high fidelity	No
GANFusion (hi-res)	68.8	0.293	✓	No

FID and CLIP-score are evaluated on rendered images of held-out sets, with the GANFusion model matching AG3D fidelity while enabling open-vocabulary text control and dispensing with test-time optimization (Attaiki et al., 2024). In user studies, the IT3D dual-training strategy using GANFusion was preferred in 89.9% of pairwise comparisons over baseline DreamFusion outputs (Chen et al., 2023).

6. Qualitative Observations and Analysis of Limitations

Key observations include:

• Dual-trained models (SDS + GAN) yield sharper, less over-smoothed textures and better pose/geometry stability than SDS-only or GAN-only models.
• GAN discriminators correct for view inconsistencies inherent in high-variance, per-view synthesis pipelines, by aligning the 3D model render distribution to that of the multi-view real set (Chen et al., 2023).
• In the two-stage feed-forward model, any artifacts or distributional collapse in the initial GAN are inherited ("burned") into the subsequent diffusion prior (Attaiki et al., 2024).
• Both approaches are subject to prompt-ambiguity failure modes, and the diversity in conditioning (limited by the BLIP caption pipeline) constrains open-domain applicability.

Current models are category-specific (e.g., humans, faces, cats) and generalizing to arbitrary classes or using alternative representations (e.g., point clouds, meshes) remains open. Additionally, scaling captioning and dataset diversity is expected to improve generality.

7. Future Directions and Open Problems

Extending GANFusion to arbitrary object categories and closing fidelity gaps relative to state-of-the-art text-to-2D diffusion models represent principal challenges. Integrating more expressive representations, leveraging broader-scale datasets, or coupling with pretrained general-purpose 3D diffusion priors are potential strategies. Further research into addressing the inheritance of GAN artifacts in diffusion priors and enriching text conditioning pipelines may further advance text-to-3D synthesis beyond current limitations (Chen et al., 2023, Attaiki et al., 2024).