GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

Abstract: While recent advances in neural radiance field enable realistic digitization for large-scale scenes, the image-capturing process is still time-consuming and labor-intensive. Previous works attempt to automate this process using the Next-Best-View (NBV) policy for active 3D reconstruction. However, the existing NBV policies heavily rely on hand-crafted criteria, limited action space, or per-scene optimized representations. These constraints limit their cross-dataset generalizability. To overcome them, we propose GenNBV, an end-to-end generalizable NBV policy. Our policy adopts a reinforcement learning (RL)-based framework and extends typical limited action space to 5D free space. It empowers our agent drone to scan from any viewpoint, and even interact with unseen geometries during training. To boost the cross-dataset generalizability, we also propose a novel multi-source state embedding, including geometric, semantic, and action representations. We establish a benchmark using the Isaac Gym simulator with the Houses3K and OmniObject3D datasets to evaluate this NBV policy. Experiments demonstrate that our policy achieves a 98.26% and 97.12% coverage ratio on unseen building-scale objects from these datasets, respectively, outperforming prior solutions.

• The paper introduces an RL-based NBV policy that expands the action space to a 5D free space, enabling flexible viewpoint selection.
• It employs a multi-source state embedding that fuses geometric, semantic, and action information and optimizes the policy via PPO.
• Experiments show superior performance with coverage ratios up to 98.26%, highlighting robust generalization across diverse 3D scenes.

GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

Introduction

The paper "GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction" addresses the challenge of automating the image-capturing process for large-scale scene reconstruction, which continues to be labor-intensive and time-consuming despite advances in neural radiance fields. Traditional Next-Best-View (NBV) policies face limitations due to reliance on hand-crafted criteria and constrained action spaces. GenNBV introduces an RL-based framework that expands the action space to a 5D free space, allowing drones to interact with unseen geometries during training and scan from any viewpoint.

Figure 1: To determine the best view for 3D reconstruction, previous methods only chose from hand-crafted action space or based on object-centric capturing, lacking the ability to generalize to unforeseen scenes (Left). With our end-to-end trained, generalized free-space policy, it can generalize to unseen objects, enabling the captured drone to image from any viewpoint (Right).

Methodology

GenNBV reformulates the NBV problem as a Markov Decision Process (MDP), proposing both a novel large action space and a multi-source state embedding to ensure effective policy generalization across diverse object geometries.

Action Space

The action space in GenNBV is composed of 3D position coordinates and 2D rotation angles (yaw and pitch), enabling free space exploration, unlike previous methods that confined drones to predefined surfaces such as hemispheres.

State Embedding

The state embedding incorporates geometric, semantic, and action representations. A probabilistic 3D grid is employed to track voxel occupancy, differentiating between occupied and unscanned but empty regions. Semantic embeddings are derived from multi-frame RGB images, aiding in identifying environmental context through visual clues.

Policy Network

GenNBV uses a 3-layer MLP to generate actions sampled from the stochastic policy informed by state embeddings. The policy is optimized using PPO, which leverages parallelized sampling for increased efficiency.

Figure 2: Overview of our proposed framework GenNBV. Our end-to-end policy takes the historical multi-source observations as input, transforms them into a more informative scene representation, and produces the next viewpoint position. A reward signal will be returned at training time to optimize the end-to-end policy for maximizing the expected cumulative reward in one episode. Specifically, the signal is the increased coverage ratio after collecting a new viewpoint.

Experiments

Experimental validation showed that GenNBV achieved superior performance on the Houses3K and OmniObject3D datasets, with a 98.26% and 97.12% coverage ratio respectively, demonstrating robust generalization capabilities.

Figure 3: The visualization results of unseen 3D objects reconstructed by Scan-RL~\citep{peralta2020next}.

Results

The proposed GenNBV policy significantly outperformed heuristic, information-gain based, and other RL-based baselines on building-scale object reconstruction, as evidenced by improved AUC and coverage metrics.

Figure 4: The visualization results of an unseen 3D outdoor scene with enormous details from Objaverse, reconstructed by Uncertainty-guided, Scan-RL and our model. Compared to the uncertainty-guided method and Scan-RL, the scene reconstructed by our method is more watertight and has fewer holes on the ground and building surface, especially in the region highlighted by the red box.

Conclusion

GenNBV demonstrates a scalable approach to active 3D reconstruction in complex environments through its generalizable and efficient view planning policy. Its implementation in diverse datasets highlights its potential for real-world applications where robust scene understanding is requisite. Future work may explore optimizing reward functions or extending generalizability to more complex scenarios.

Figure 5: The curve of coverage ratio with the increasing number of training objects on unseen OmniObject3D house category.