Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation (2404.07933v1)

Abstract: Inferring scene geometry from images via Structure from Motion is a long-standing and fundamental problem in computer vision. While classical approaches and, more recently, depth map predictions only focus on the visible parts of a scene, the task of scene completion aims to reason about geometry even in occluded regions. With the popularity of neural radiance fields (NeRFs), implicit representations also became popular for scene completion by predicting so-called density fields. Unlike explicit approaches. e.g. voxel-based methods, density fields also allow for accurate depth prediction and novel-view synthesis via image-based rendering. In this work, we propose to fuse the scene reconstruction from multiple images and distill this knowledge into a more accurate single-view scene reconstruction. To this end, we propose Multi-View Behind the Scenes (MVBTS) to fuse density fields from multiple posed images, trained fully self-supervised only from image data. Using knowledge distillation, we use MVBTS to train a single-view scene completion network via direct supervision called KDBTS. It achieves state-of-the-art performance on occupancy prediction, especially in occluded regions.

• The paper introduces a novel self-supervised method that distills robust multi-view 3D geometry into a single-view model for complete scene reconstruction.
• It combines a Multi-View Behind the Scenes architecture with knowledge distillation, improving occupancy and depth predictions on benchmarks like KITTI-360.
• The approach reduces computational needs while enhancing accuracy, paving the way for practical applications in autonomous driving and robotics.

Overview of "Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation"

The paper, "Boosting Self-Supervision for Single-View Scene Completion via Knowledge Distillation," addresses the problem of obtaining accurate 3D geometry from images in computer vision, particularly focusing on the task of scene completion. This involves reasoning about both visible and occluded regions. Scene completion extends the classical depth prediction tasks by inferring geometry in occluded areas, thus providing a more comprehensive understanding of a scene's geometry.

Key Contributions

The authors propose a novel method that employs knowledge distillation techniques to improve single-view scene reconstruction by leveraging information from multiple views. The method involves two significant components:

1. Multi-View Behind the Scenes (MVBTS): This is an extension of the existing "Behind the Scenes" (BTS) architecture for density field prediction, which is adapted to a multi-view setting. It is trained in a self-supervised manner using image data alone. This component effectively integrates information from multiple views to enhance the accuracy of occupancy predictions, especially in regions that are occluded from single-view observations.
2. Knowledge Distillation from Multi-View to Single-View (KDBTS): The second component derives from the concept of knowledge distillation between neural networks. Here, the robust scene reconstructions obtained from MVBTS are used to supervise a single-view model, called KDBTS. This method achieves state-of-the-art performance in single-view occupancy prediction by distilling knowledge to predict the scene's complete geometry from just a single view.

Methodology

The MVBTS approach introduces a multi-view neural network architecture capable of fusing density fields obtained from several images into a single coherent scene representation. A core part of their method is leveraging a self-supervised training scheme that relies solely on image data without necessitating ground truth 3D geometry. The process involves a carefully designed volumetric rendering pipeline that reconstructs images by leveraging estimated geometry to enforce photometric consistency.

In the knowledge distillation phase, predictions from the multi-view setup serve as pseudo ground-truth to train a single-view model. This distillation achieves impressive results in predicting complete 3D occupancy maps from a single view, arguably bridging the gap between self-supervised learning capabilities and the demand for minimal input data during inference.

Numerical and Experimental Results

The experimental evaluation shows that the proposed multi-view and single-view models surpass previous methods in occupancy and depth prediction under various benchmark settings, notably the KITTI-360 dataset. The model achieves substantial improvements in predicting the geometry of occluded regions. Moreover, the use of knowledge distillation demonstrates a clear advantage in reducing model size while maintaining performance, thus addressing practical constraints such as computational resources and inference time.

Implications and Future Directions

The implications of this work are significant for applications requiring comprehensive scene understanding from image data, such as autonomous driving and robotics. By enhancing the capability of single-view models, this research paves the way for more robust and efficient 3D perception systems. Future research might explore the integration of dynamic scene elements, as current assumptions rely on static scenes, potentially limiting applications in environments with moving objects.

In summary, this paper provides a thorough investigation into the use of multi-view knowledge distillation to enhance single-view scene completion, offering effective methodologies and demonstrating substantial performance gains in 3D occupancy prediction.