4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks (1904.08755v4)

Abstract: In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, those 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose the generalized sparse convolution that encompasses all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that convolutional neural networks with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise, outperform 3D convolutional neural networks and are faster than the 3D counterpart in some cases.

Summary of "4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks"

The paper "4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks" introduces a novel framework for high-dimensional convolutional neural networks specifically designed to process 3D videos, termed as 4D spatio-temporal convolutional networks (ConvNets). The central contribution of this work is the development and application of generalized sparse convolutions for high-dimensional perception, accommodating the inherent sparsity in 3D data.

Key Contributions

1. Generalized Sparse Convolution:
   • The paper extends sparse convolution to a generalized form that can handle arbitrary input and output coordinates along with adaptable kernel shapes.
   • The generalized sparse convolution unifies various types of convolutions, making it a superset of conventional dense convolutions and sparse submanifold convolutions.
2. Minkowski Engine:
   • The authors propose and implement an open-source auto-differentiation library for sparse tensors named Minkowski Engine.
   • This library facilitates the development of high-dimensional convolutional neural networks by providing essential functions such as sparse tensor quantization, generalized sparse convolution, max pooling, and global/average pooling in 4D and higher dimensions.
3. 4D Spatio-Temporal ConvNets:
   • The paper introduces 4D ConvNets for spatio-temporal perception leveraging the proposed generalized sparse convolutions.
   • They introduce the "hybrid kernel" to manage the computational complexity and memory overhead associated with high-dimensional convolutions. It uses cross-shaped kernels for temporal and cubic kernels for spatial dimensions to balance between performance and efficiency.
4. Trilateral Stationary-CRF:
   • To enforce spatio-temporal consistency in predictions, the authors propose a 7D trilateral stationary conditional random field (TS-CRF), extending the network's capabilities from the 4D space-time domain into the 7D space-time-chroma domain.
   • They apply variational inference to train the TS-CRF, converting update equations into recurrent neural network layers to achieve end-to-end training with the base network.

Experimental Validation

The effectiveness of the proposed 4D ConvNets and the Minkowski Engine is validated through extensive benchmarking on several datasets:

1. 3D Semantic Segmentation:
   • Experiments on the ScanNet and S3DIS datasets demonstrate that 3D ConvNets using generalized sparse convolutions outperform state-of-the-art 2D and hybrid methods by a significant margin.
   • Specifically, the model achieves 67.9% mIoU on the ScanNet benchmark using a 5cm voxel size, outperforming the best-reviewed work by 19% mIoU.
2. 4D Spatio-Temporal Perception:
   • The Synthia and RueMonge 2014 (Varcity) datasets are used for 4D analysis. These datasets provide temporal sequences to test the robustness and accuracy of the 4D ConvNets.
   • On the Synthia dataset, the 4D MinkNet coupled with TS-CRF achieves 78.67% mIoU, demonstrating robustness against noise and improved performance over conventional 3D approaches.
3. Ablation Studies:
   • The results indicate the hybrid kernel's superiority over tesseract kernels in 4D ConvNets, not only maintaining better performance but also improving computational efficiency.
   • The TS-CRF effectively enforces spatio-temporal consistency, enhancing the network's resilience to noisy inputs while slightly increasing the computational burden.

Implications and Future Work

The proposed 4D ConvNets open avenues for efficient and accurate high-dimensional perception in various applications, including robotics, autonomous driving, and AR/VR systems. The generalized sparse convolution framework presents a flexible approach adaptable to multiple dimensions, potentially benefiting other domains dealing with high-dimensional sparse data.

Future work could explore:

• Further optimization of the Minkowski Engine to handle even larger and more complex datasets.
• Extending the applications of 4D ConvNets to real-time scenarios and integrating them with other perception systems.
• Investigating the use of generalized sparse convolutions in other high-dimensional data types outside of spatio-temporal domains.

By tackling the challenges of high-dimensional data processing and providing a robust, generalized convolution framework, this research significantly advances the capabilities of neural networks in handling complex, sparse data structures.