Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

Abstract: A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.

• The paper introduces Score Jacobian Chaining, which adapts 2D diffusion models for 3D generation by chaining predicted gradient fields through a differentiable renderer.
• The paper details the PAAS technique that perturbs inputs and averages scores to address distribution mismatches in non-noisy rendered images.
• The paper validates the approach across multiple datasets and model architectures, demonstrating enhanced output quality with optimized noise scheduling.

Score Jacobian Chaining: Advancing 3D Generation with Pretrained 2D Diffusion Models

The paper "Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation" explores a novel method for extending the capabilities of pretrained 2D diffusion models to generate three-dimensional (3D) assets. This research builds on the idea of interpreting diffusion models as predictors of a gradient field and effectively employs the chain rule on these predicted gradients. The process involves back-propagating the score through the Jacobian of a differentiable renderer, particularly using a voxel radiance field for this purpose.

Core Methodology

The central thesis of the research is the conversion of 2D diffusion models into 3D generative models without explicit 3D data. By aggregating 2D scores from multiple camera viewpoints, the method generates a 3D score. This approach leverages pretrained 2D diffusion resources, such as Stable Diffusion, a model trained on the large-scale LAION 5B dataset, for tackling the 3D generation task.

A primary challenge addressed by the authors is the distribution mismatch that emerges when pretrained denoisers are applied to non-noisy rendered images in the optimization process. The authors propose the Perturb-and-Average Scoring (PAAS) method to resolve this issue. The approach estimates the score for non-noisy images through a process that perturbs the input images by noise and averages the score results over multiple evaluations.

Empirical Results and Evaluation

The authors validate their method across different experimental setups, including unconditioned diffusion models from FFHQ and LSUN Bedroom datasets and Stable Diffusion—a pretrained large-scale model. The empirical results demonstrate the viability of PAAS in resolving the out-of-distribution (OOD) problem and reveal various subtleties of employing Annealed versus Random sigma schedules during optimization. Specifically, for high guidance scale settings with Stable Diffusion, Random sigma scheduling produces consistently high-quality outputs.

Contributions and Technical Insights

The paper's contributions include:

1. A method—Score Jacobian Chaining (SJC)—for adapting 2D diffusion models for 3D generation via chaining scores and incorporating the renderer Jacobian.
2. PAAS, a novel method for handling the OOD challenge when using diffusive models in a 3D rendering context.
3. Detailed insights into hyperparameter settings and their impact on diffusion processes, validated across multiple diffusion model architectures and datasets.

The authors demonstrate the theoretical underpinnings of their proposed method with substantial mathematical rigor and provide a comprehensive treatment of the underlying score-based view of diffusion models, facilitating the understanding and application of these concepts in future AI developments.

Implications and Prospective Developments

The research opens avenues for leveraging existing, robust 2D diffusion models in new domains such as 3D content creation without the necessity of specializing the model for 3D data structures. This approach promises broadened utilization of pretrained resources, sparing the need for extensive 3D datasets, which are often expensive and labor-intensive to produce. The implementation of SJC could influence both theoretical explorations in 3D generative modeling and pragmatic application areas, including gaming, virtual reality, and digital design.

The research further highlights an ongoing challenge in the field—how to effectively maneuver between various representations of high-dimensional distributions (moving from 2D to 3D) and maintain fidelity and coherency across dimensions. The insights provided by the authors into noise scheduling and denoising progression significantly improve understanding and may guide future probabilistic models and generative methods.

In conclusion, "Score Jacobian Chaining" offers a sophisticated addition to the toolkit for those seeking to extend 2D diffusion models into 3D rendering territories. The methodologies developed herein and their subsequent findings form a bedrock for upcoming explorations in AI-driven content generation.