SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement (2408.00653v1)

Abstract: We present SF3D, a novel method for rapid and high-quality textured object mesh reconstruction from a single image in just 0.5 seconds. Unlike most existing approaches, SF3D is explicitly trained for mesh generation, incorporating a fast UV unwrapping technique that enables swift texture generation rather than relying on vertex colors. The method also learns to predict material parameters and normal maps to enhance the visual quality of the reconstructed 3D meshes. Furthermore, SF3D integrates a delighting step to effectively remove low-frequency illumination effects, ensuring that the reconstructed meshes can be easily used in novel illumination conditions. Experiments demonstrate the superior performance of SF3D over the existing techniques. Project page: https://stable-fast-3d.github.io

• The paper introduces SF3D, a novel method that rapidly generates high-quality 3D meshes with UV-unwrapping and illumination disentanglement in just 0.5 seconds.
• It employs an enhanced transformer backbone and probabilistic material estimation to mitigate artifacts and improve texture fidelity.
• The method outperforms existing techniques by producing meshes with lower polygon counts and smoother surfaces, benefiting AR/VR, gaming, and e-commerce applications.

SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement

Abstract

The paper introduces SF3D, a novel approach to rapid and high-quality textured object mesh reconstruction from a single image in just 0.5 seconds. This method leverages advanced techniques such as UV unwrapping, illumination disentanglement, and the prediction of material parameters to enhance visual fidelity and practical utility.

Introduction and Problem Statement

3D reconstruction from a single image remains a challenging inverse problem due to the need for accurate shape and texture inference from limited 2D data. Despite recent advancements driven by transformer models and large synthetic datasets, existing methods often produce suboptimal 3D assets that require extensive post-processing for real-world applications. SF3D addresses several key issues of current fast 3D reconstruction models, including light bake-in, vertex coloring inefficiencies, and marching cubes artifacts. The proposed approach aims to deliver high-quality 3D meshes with lower polygon counts, making them more suitable for applications in gaming, AR/VR, and e-commerce.

Methodology

The SF3D pipeline comprises multiple novel components to overcome the limitations of existing methods:

1. Enhanced Transformer Backbone:
   • SF3D employs a modified transformer architecture based on DINOv2 for generating higher resolution triplanes (96x96 resolution, enhanced to 384x384 using pixel shuffling), which significantly reduces aliasing artifacts and improves texture fidelity.
2. Material Estimation:
   • A probabilistic approach is utilized to predict non-spatially varying material properties such as metallic and roughness values, enhancing the visual realism of reflective surfaces. This is achieved through a separately trained Material Net employing a Beta distribution to stabilize training.
3. Illumination Modeling:
   • The Light Net component predicts spherical Gaussian illumination maps from the triplanes, enabling effective delighting and ensuring that the reconstructed objects can be re-lit under novel conditions. A lighting demodulation loss ensures consistency with training data illumination.
4. Mesh Extraction and Refinement:
   • Differentiable Marching Tetrahedrons (DMTet) are used to generate the initial mesh, with subsequent refinement through learned vertex offsets and normal maps to produce smoother surfaces free of staircase artifacts.
5. Fast UV Unwrapping:
   • A highly efficient, parallelizable cube projection-based UV unwrapping technique is introduced, reducing the UV unwrapping time to 150ms, contributing to the total generation time of 0.5s.

The training pipeline involves pre-training on NeRF tasks followed by mesh fine-tuning with differentiable rendering and several regularization losses to ensure smooth and accurate mesh outputs.

Experimental Results

SF3D is evaluated on the GSO and OmniObject3D datasets, demonstrating superior performance compared to state-of-the-art methods such as TripoSR, OpenLRM, and others in both geometric accuracy (Chamfer Distance and F-score) and visual quality. Notably, SF3D achieves these results while maintaining lower polygon counts, which is advantageous for practical application scenarios. The method produces detailed textures and smoother shading without the marching cubes artifacts prevalent in competing approaches.

Discussion and Implications

SF3D represents a significant improvement in the fidelity and usability of 3D assets reconstructed from single images. The combination of high-resolution triplanes, effective material estimation, and robust illumination modeling contribute to high-quality outputs that require minimal post-processing. The fast UV unwrapping mechanism further enhances the practical utility by enabling rapid integration into graphics pipelines.

The results suggest promising directions for future research, including:

• Extending the material prediction to spatially varying properties to handle heterogeneous objects.
• Training on real-world datasets to improve generalization beyond synthetic data.
• Further optimizing the UV unwrapping process using real-world dataset insights.

Conclusion

SF3D offers a comprehensive solution for rapid and high-quality 3D object generation from single images. By addressing both speed and quality, it advances the state-of-the-art in single-image 3D reconstruction and provides practical benefits for various downstream applications. Future work could expand on its robustness and versatility to meet the growing demands of real-time 3D asset generation in diverse industries.