GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds (2206.07255v2)

Abstract: Recent works have shown that 3D-aware GANs trained on unstructured single image collections can generate multiview images of novel instances. The key underpinnings to achieve this are a 3D radiance field generator and a volume rendering process. However, existing methods either cannot generate high-resolution images (e.g., up to 256X256) due to the high computation cost of neural volume rendering, or rely on 2D CNNs for image-space upsampling which jeopardizes the 3D consistency across different views. This paper proposes a novel 3D-aware GAN that can generate high resolution images (up to 1024X1024) while keeping strict 3D consistency as in volume rendering. Our motivation is to achieve super-resolution directly in the 3D space to preserve 3D consistency. We avoid the otherwise prohibitively-expensive computation cost by applying 2D convolutions on a set of 2D radiance manifolds defined in the recent generative radiance manifold (GRAM) approach, and apply dedicated loss functions for effective GAN training at high resolution. Experiments on FFHQ and AFHQv2 datasets show that our method can produce high-quality 3D-consistent results that significantly outperform existing methods. It makes a significant step towards closing the gap between traditional 2D image generation and 3D-consistent free-view generation.

• The paper presents a novel framework that integrates radiance manifolds with 3D super-resolution to achieve high-resolution 1024×1024 image generation.
• It utilizes 2D CNNs on 3D manifolds to efficiently process data, significantly reducing memory usage and inference time compared to previous models.
• Experimental results show improved FID scores and strong multiview consistency, paving the way for advanced 3D-aware image synthesis applications.

GRAM-HD: Enhancing 3D-Consistent Image Generation

The paper entitled "GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds" presents a novel method aimed at improving the state of 3D-aware generative adversarial networks (GANs). The principal contribution of this work is a technique capable of generating high-resolution images (up to 1024×1024 pixels) while maintaining strong 3D consistency across multiple views. This is achieved by leveraging generative radiance manifolds (GRAM) in combination with an innovative 3D super-resolution strategy. The work addresses critical shortcomings of existing methods, such as the high computational cost of neural volume rendering and the 3D inconsistency introduced by 2D convolutional neural networks (CNNs) used for image-space upsampling.

Key Innovations and Approach

1. 3D Super-Resolution Through 2D CNNs: The paper introduces an approach that preserves 3D consistency by performing super-resolution directly in 3D space. This is implemented by using 2D CNNs on a set of radiance manifolds, effectively reducing computational burden while retaining multiview image consistency.
2. Radiance Manifold Extension: Using the GRAM methodology, the authors generate a set of surface manifolds from a scalar field, achieving a more computationally efficient representation than traditional neural radiance fields (NeRF). These manifolds are then sampled and processed by 2D CNNs to generate high-resolution images.
3. GAN Training Framework: The authors employ a two-stage training strategy. Initially, a low-resolution model is trained, followed by a high-resolution model that refines the generated images using the stored manifold representations. Custom loss functions, including an adversarial loss and a pose loss, are utilized to ensure high-quality image generation and consistent multiview geometries.

Experimental Results

Comprehensive experiments demonstrate that GRAM-HD achieves superior image quality compared to traditional 3D-aware GANs such as StyleNeRF, StyleSDF, and EG3D. Notably, the paper shows that GRAM-HD significantly reduces memory consumption and inference time by 76% and 58%, respectively, while lowering the Fréchet Inception Distance (FID) on FFHQ by 21%. These results indicate a remarkable advance in the practicality and effectiveness of high-resolution 3D-consistent image generation.

Implications and Future Directions

The proposed method offers significant practical implications, particularly in fields requiring high-quality 3D image synthesis such as virtual reality, video game design, and digital content creation. The ability to generate realistic images that are readily usable for animations and video synthesis opens new avenues for interactive applications.

From a theoretical perspective, GRAM-HD bridges the gap between conventional 2D image generative models and 3D integrity, presenting a pathway for future research exploring the use of manifold representations in other dimensions or restructuring pixel synthesis paradigms. However, challenges remain in terms of scalability to more complex geometries and enhancing view extrapolation capabilities, positioning these as potential future research directions.

Conclusion

GRAM-HD marks a notable development in the landscape of 3D-aware image synthesis, showing promise in high-resolution, geometrically consistent image production with computational efficiency. It sets a solid foundation for future explorations of manifold-based representations in generative models, aiming for even broader applications and improved synthesis fidelity.