NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing (2207.11911v1)

Abstract: Very recently neural implicit rendering techniques have been rapidly evolved and shown great advantages in novel view synthesis and 3D scene reconstruction. However, existing neural rendering methods for editing purposes offer limited functionality, e.g., rigid transformation, or not applicable for fine-grained editing for general objects from daily lives. In this paper, we present a novel mesh-based representation by encoding the neural implicit field with disentangled geometry and texture codes on mesh vertices, which facilitates a set of editing functionalities, including mesh-guided geometry editing, designated texture editing with texture swapping, filling and painting operations. To this end, we develop several techniques including learnable sign indicators to magnify spatial distinguishability of mesh-based representation, distillation and fine-tuning mechanism to make a steady convergence, and the spatial-aware optimization strategy to realize precise texture editing. Extensive experiments and editing examples on both real and synthetic data demonstrate the superiority of our method on representation quality and editing ability. Code is available on the project webpage: https://zju3dv.github.io/neumesh/.

• The paper introduces NeuMesh, a method that disentangles geometry and texture using mesh-based neural representations for precise 3D editing.
• It leverages vertex-bound codes, learnable sign indicators, and a two-phase training scheme to overcome limitations of traditional neural rendering.
• Experimental results on synthetic and real-world datasets demonstrate superior PSNR, SSIM, and LPIPS metrics, validating NeuMesh’s impact for advanced content creation.

Overview of NeuMesh: Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing

The paper presents NeuMesh, a mesh-based representation for neural implicit fields that allows for geometry and texture editing. This work addresses limitations in existing neural rendering methods that offer only basic editing functionalities and often fail in fine-grained manipulation of 3D objects. NeuMesh distinguishes itself by encoding geometry and texture information into disentangled codes on mesh vertices, enabling detailed geometry deformation and textured outcomes tailored to creator desires.

Technical Contributions

NeuMesh leverages a mesh scaffold where vertex-bound codes represent the scene, circumventing issues seen in purely MLP or voxel-based approaches. This vertex-bound method effectively translates mesh deformations into corresponding neural representations, offering a precise control over both geometry and textures. Notably, the model accommodates non-uniform geometry through the following advancements:

1. Learnable Sign Indicators: These indicators increase spatial distinguishability in the implicit field, overcoming challenges linked to non-watertight geometries. By enabling the network to adjust indicators based on optimization objectives, NeuMesh supports diverse geometry typologies.
2. Distillation and Fine-Tuning: The use of a pre-trained implicit field as a teacher model allows for smoother training and prevents convergence issues often tied to spatially localized features. This two-phase training scheme ensures NeuMesh maintains high fidelity in rendering while integrating into a flexible mesh-based workflow.
3. Spatial-Aware Optimization for Texture Editing: NeuMesh introduces targeted optimization, allowing precise control over texture painting and filling. This spatial-aware method retains performance across different views by limiting editing influence to specific texture codes only, which helps avoid overfitting to single-view fine-tuning.

Experimental Validation

Experiments illustrate that NeuMesh excels in both synthetic and real-world datasets, outperforming competing methods like NeuTex and NeuS in terms of rendering precision and mesh quality. Strong numerical outcomes in PSNR, SSIM, and LPIPS metrics demonstrate superior visual fidelity and effectiveness in diverse tasks ranging from view synthesis to comprehensive object editing.

Implications and Future Work

NeuMesh's advances in disentangling geometry and texture representations offer compelling implications for AI-driven content creation. This framework aligns well with existing 3D modeling workflows, potentially influencing other AI applications in computer graphics, such as automated scene generation or mixed-reality technologies.

Future developments could explore enhancing neural representations to incorporate lighting effects and material properties, supporting broader realism across varying environments. Additionally, refining spatial constraints to further improve rendering speed without compromising detail remains a promising area for ongoing research.

In conclusion, NeuMesh presents an innovative approach to neural rendering by effectively bridging mesh-based 3D modeling with neural implicit representations, building towards more interactive and artist-friendly graphical environments.