NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2212.05231v3)

Abstract: Recent methods for neural surface representation and rendering, for example NeuS, have demonstrated the remarkably high-quality reconstruction of static scenes. However, the training of NeuS takes an extremely long time (8 hours), which makes it almost impossible to apply them to dynamic scenes with thousands of frames. We propose a fast neural surface reconstruction approach, called NeuS2, which achieves two orders of magnitude improvement in terms of acceleration without compromising reconstruction quality. To accelerate the training process, we parameterize a neural surface representation by multi-resolution hash encodings and present a novel lightweight calculation of second-order derivatives tailored to our networks to leverage CUDA parallelism, achieving a factor two speed up. To further stabilize and expedite training, a progressive learning strategy is proposed to optimize multi-resolution hash encodings from coarse to fine. We extend our method for fast training of dynamic scenes, with a proposed incremental training strategy and a novel global transformation prediction component, which allow our method to handle challenging long sequences with large movements and deformations. Our experiments on various datasets demonstrate that NeuS2 significantly outperforms the state-of-the-arts in both surface reconstruction accuracy and training speed for both static and dynamic scenes. The code is available at our website: https://vcai.mpi-inf.mpg.de/projects/NeuS2/ .

• The paper introduces multi-resolution hash encodings and optimized second-order derivatives that drastically reduce training times for neural implicit surface reconstruction.
• The paper employs a progressive learning and incremental training strategy that enhances both convergence speed and detail fidelity in dynamic scenes.
• The paper validates NeuS2's superior performance over existing methods on static and dynamic datasets, indicating its potential for real-time AR/VR applications.

NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction

The paper "NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction" presents a novel approach to improving the efficiency and scalability of neural implicit surface reconstruction, especially in dynamic scenarios. The authors focus on reducing computational overhead while maintaining high-quality reconstruction, thus broadening the applicability of neural surface reconstruction techniques.

Addressing Limitations in Current Methods

Existing methods, notably NeuS, have demonstrated impressive reconstruction capabilities but are encumbered by extensive training times, making them unsuitable for dynamic scenes. NeuS, for example, requires around 8 hours to train, rendering it impractical for sequences with many frames.

NeuS2 introduces key innovations to overcome these bottlenecks:

• Multi-resolution Hash Encodings: Drawing inspiration from Instant-NGP, NeuS2 incorporates multi-resolution hash encodings that significantly accelerate the training process while retaining reconstruction quality. These encodings help reduce the depth of neural networks needed, thus expediting convergence.
• Second-order Derivative Optimization: A notable technical advancement in NeuS2 is the simplified calculation of second-order derivatives tailored for ReLU-based MLPs. This computational efficiency is achieved through CUDA parallelization, optimizing runtime by leveraging GPU capabilities.
• Progressive Learning Strategy: The introduction of a progressive learning strategy allows the model to learn from coarse to fine resolutions, further stabilizing training and enhancing convergence speed and quality.

Extending to Dynamic Scenes

NeuS2 extends its applicability to dynamic scenes through:

• Incremental Training Strategy: By leveraging temporal coherence between consecutive frames, NeuS2 reduces training time by refining parameters frame-by-frame, utilizing the previous frame as an initial approximation. This strategy significantly reduces computational requirements without sacrificing accuracy.
• Global Transformation Prediction: The authors introduce a mechanism to predict global transformations, to handle large scene movements and deformations efficiently. This component ensures that the network remains robust against local minima during frame-to-frame adaptations.

Experimental Validation

Experiments demonstrate NeuS2's superiority over existing state-of-the-art methods, including NeuS, Instant-NGP, and Instant-NSR, on both static and dynamic datasets:

• Static Reconstruction: On the DTU dataset, NeuS2 achieves top scores in Chamfer Distance and PSNR, evidencing superior geometry and appearance fidelity.
• Dynamic Scenes: NeuS2 outperforms D-NeRF and TiNeuVox in novel view synthesis and geometry reconstruction across synthetic and real-world sequences, notably achieving significant runtime reductions.

Implications and Future Directions

The accelerated training of NeuS2 holds potential for real-time applications in AR/VR, telepresence, and interactive media. The integration of efficient second-order derivative computation and encoding strategies also sets a precedent for future advancements in neural rendering and surface modeling.

Looking forward, research could explore dense surface correspondences in dynamic sequences to provide temporal coherence across frames. Moreover, parameter compression could further lower memory footprints, enhancing scalability.

Overall, NeuS2 represents a meaningful advancement in neural surface reconstruction, offering a balance of efficiency and high-quality output that paves the way for broader adoption in dynamic scene rendering.