PartNeRF: Generating Part-Aware Editable 3D Shapes without 3D Supervision (2303.09554v3)

Abstract: Impressive progress in generative models and implicit representations gave rise to methods that can generate 3D shapes of high quality. However, being able to locally control and edit shapes is another essential property that can unlock several content creation applications. Local control can be achieved with part-aware models, but existing methods require 3D supervision and cannot produce textures. In this work, we devise PartNeRF, a novel part-aware generative model for editable 3D shape synthesis that does not require any explicit 3D supervision. Our model generates objects as a set of locally defined NeRFs, augmented with an affine transformation. This enables several editing operations such as applying transformations on parts, mixing parts from different objects etc. To ensure distinct, manipulable parts we enforce a hard assignment of rays to parts that makes sure that the color of each ray is only determined by a single NeRF. As a result, altering one part does not affect the appearance of the others. Evaluations on various ShapeNet categories demonstrate the ability of our model to generate editable 3D objects of improved fidelity, compared to previous part-based generative approaches that require 3D supervision or models relying on NeRFs.

• The paper introduces a part-aware generative model that uses local NeRFs with affine transformations to enable precise, editable 3D shape generation.
• It implements a novel hard assignment of rays to parts to ensure local edits do not affect the whole object.
• The approach eliminates the need for 3D supervision, achieving high shape fidelity and flexibility for applications in gaming, animation, and VR.

"PartNeRF: Generating Part-Aware Editable 3D Shapes without 3D Supervision" addresses a significant challenge in 3D shape generation by enabling local control and editing of 3D shapes without the need for explicit 3D supervision. The motivation behind this work is to combine the quality of implicitly represented 3D shapes with the flexibility of part-aware models, ultimately unlocking various content creation applications.

The core contribution of PartNeRF is a novel part-aware generative model. Here are some key elements and innovations of the approach:

1. Local NeRFs and Affine Transformations:
   • The model generates objects as a collection of locally defined Neural Radiance Fields (NeRFs). Each local NeRF is associated with an affine transformation.
   • This setup allows for a variety of editing operations, such as applying transformations to individual parts and mixing parts from different objects.
2. Hard Assignment of Rays to Parts:
   • One of the primary technical innovations of PartNeRF is the hard assignment of rays to parts, ensuring that the color of each ray is determined by only one specific NeRF.
   • This hard assignment is crucial as it guarantees that modifying one part of the object does not influence the appearance of other parts, thereby enhancing the precision of local edits.
3. No Need for 3D Supervision:
   • Unlike previous approaches, PartNeRF does not require any explicit 3D supervision. This is a significant advantage because collecting 3D ground truth data can be resource-intensive and challenging.
4. Improved Part Fidelity:
   • Evaluations on various ShapeNet categories demonstrate that PartNeRF can generate editable 3D objects with improved fidelity compared to existing part-based generative models that rely on 3D supervision or NeRFs.
   • The model achieves high-quality shape generation while supporting complex local edits, thus offering a level of flexibility and usability previously unattainable.

By eliminating the dependency on 3D supervision and introducing a sophisticated mechanism for part-aware 3D shape generation and manipulation, this work represents a substantial advancement in the field. It has broad implications for content creation, potentially simplifying workflows in industries such as gaming, animation, and virtual reality, where the capability to easily generate and edit 3D shapes is critical.