Enhancing High-Resolution 3D Generation through Pixel-wise Gradient Clipping (2310.12474v4)
Abstract: High-resolution 3D object generation remains a challenging task primarily due to the limited availability of comprehensive annotated training data. Recent advancements have aimed to overcome this constraint by harnessing image generative models, pretrained on extensive curated web datasets, using knowledge transfer techniques like Score Distillation Sampling (SDS). Efficiently addressing the requirements of high-resolution rendering often necessitates the adoption of latent representation-based models, such as the Latent Diffusion Model (LDM). In this framework, a significant challenge arises: To compute gradients for individual image pixels, it is necessary to backpropagate gradients from the designated latent space through the frozen components of the image model, such as the VAE encoder used within LDM. However, this gradient propagation pathway has never been optimized, remaining uncontrolled during training. We find that the unregulated gradients adversely affect the 3D model's capacity in acquiring texture-related information from the image generative model, leading to poor quality appearance synthesis. To address this overarching challenge, we propose an innovative operation termed Pixel-wise Gradient Clipping (PGC) designed for seamless integration into existing 3D generative models, thereby enhancing their synthesis quality. Specifically, we control the magnitude of stochastic gradients by clipping the pixel-wise gradients efficiently, while preserving crucial texture-related gradient directions. Despite this simplicity and minimal extra cost, extensive experiments demonstrate the efficacy of our PGC in enhancing the performance of existing 3D generative models for high-resolution object rendering.
- ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint, 2022.
- High-performance large-scale image recognition without normalization. In ICML, 2021.
- Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models. arXiv preprint, 2023.
- Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint, 2023a.
- Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, 2023b.
- Delta denoising score. arXiv preprint, 2023.
- Denoising diffusion probabilistic models. In NeurIPS, 2020.
- Debiasing scores and prompts of 2d diffusion for robust text-to-3d generation. arXiv preprint, 2023.
- Avatarfusion: Zero-shot generation of clothing-decoupled 3d avatars using 2d diffusion. arXiv preprint, 2023a.
- Dreamtime: An improved optimization strategy for text-to-3d content creation. arXiv preprint, 2023b.
- Revisiting image pyramid structure for high resolution salient object detection. In ACCV, 2022.
- Revisiting gradient clipping: Stochastic bias and tight convergence guarantees. In ICML, 2023.
- Dreamhuman: Animatable 3d avatars from text. arXiv preprint, 2023.
- Focaldreamer: Text-driven 3d editing via focal-fusion assembly. arXiv preprint, 2023.
- Tada! text to animatable digital avatars. arXiv preprint, 2023.
- Magic3d: High-resolution text-to-3d content creation. arXiv preprint, 2022.
- Microsoft coco: Common objects in context. In ECCV, 2014.
- Zero-1-to-3: Zero-shot one image to 3d object. In ICCV, 2023a.
- Syncdreamer: Learning to generate multiview-consistent images from a single-view image. arXiv preprint, 2023b.
- Realfusion: 360 {{\{{\\\backslash\deg}}\}} reconstruction of any object from a single image. arXiv preprint, 2023.
- Latent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint, 2022.
- Tomáš Mikolov. Statistical language models based on neural networks. PhD thesis, Brno University of Technology, 2012.
- Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021.
- Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint, 2023.
- Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
- Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint, 2023.
- High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
- Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
- Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint, 2023.
- Mvdream: Multi-view diffusion for 3d generation. arXiv preprint, 2023.
- Jiaxiang Tang. Stable-dreamfusion: Text-to-3d with stable-diffusion, 2023. https://github.com/ashawkey/stable-dreamfusion.
- Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint, 2023.
- Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint, 2023.
- Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021.
- Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint, 2023.
- Hd-fusion: Detailed text-to-3d generation leveraging multiple noise estimation. arXiv preprint, 2023.
- Texture generation on 3d meshes with point-uv diffusion. arXiv preprint, 2023.
- Improved analysis of clipping algorithms for non-convex optimization. In NeurIPS, 2020.
- Why gradient clipping accelerates training: A theoretical justification for adaptivity. In ICLR, 2019.
- Adding conditional control to text-to-image diffusion models. arXiv preprint, 2023.
- Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint, 2023.