Review: deep learning on 3D point clouds (2001.06280v1)

Abstract: Point cloud is point sets defined in 3D metric space. Point cloud has become one of the most significant data format for 3D representation. Its gaining increased popularity as a result of increased availability of acquisition devices, such as LiDAR, as well as increased application in areas such as robotics, autonomous driving, augmented and virtual reality. Deep learning is now the most powerful tool for data processing in computer vision, becoming the most preferred technique for tasks such as classification, segmentation, and detection. While deep learning techniques are mainly applied to data with a structured grid, point cloud, on the other hand, is unstructured. The unstructuredness of point clouds makes use of deep learning for its processing directly very challenging. Earlier approaches overcome this challenge by preprocessing the point cloud into a structured grid format at the cost of increased computational cost or lost of depth information. Recently, however, many state-of-the-arts deep learning techniques that directly operate on point cloud are being developed. This paper contains a survey of the recent state-of-the-art deep learning techniques that mainly focused on point cloud data. We first briefly discussed the major challenges faced when using deep learning directly on point cloud, we also briefly discussed earlier approaches which overcome the challenges by preprocessing the point cloud into a structured grid. We then give the review of the various state-of-the-art deep learning approaches that directly process point cloud in its unstructured form. We introduced the popular 3D point cloud benchmark datasets. And we also further discussed the application of deep learning in popular 3D vision tasks including classification, segmentation and detection.

Review: Deep Learning on 3D Point Clouds

The paper "Review: deep learning on 3D point clouds" offers a comprehensive survey of state-of-the-art deep learning techniques that focus on processing 3D point cloud data directly. As the use of 3D point clouds becomes increasingly significant due to advancements in acquisition technologies like LiDAR and applications ranging from autonomous driving to augmented and virtual reality, the capability to process such data efficiently and effectively is crucial.

Point clouds represent a set of points in a three-dimensional space, which inherently lack structure (unlike images that reside on a regular grid). This unstructuredness poses several challenges for the direct application of deep learning techniques, which are typically designed for structured data forms. Earlier methods addressed these challenges by converting point clouds into structured formats such as voxel grids or multiple 2D views, though often at the cost of increased computational demands or loss of spatial resolution.

However, recent approaches have shifted towards leveraging deep learning directly on raw point clouds without requiring this conversion. The paper illustrates the evolution of these techniques, beginning with PointNet, which pioneered the direct processing of point clouds using deep learning by employing symmetric functions to maintain permutation invariance. While PointNet laid foundational work, its limitation was its inability to capture local structures within the point cloud—a capability vital for enhancing the discriminative power of the model.

Subsequent techniques extended PointNet by establishing methods to capture local structures hierarchically, akin to the convolutional layers in grid-based data processing. This extension includes operations such as sampling to reduce data resolution, grouping by identifying neighboring points to each representative point, and applying a mapping function, usually via a multilayer perceptron (MLP), to interpret these local regions.

Beyond PointNet, the paper discusses innovations like PointNet++, which introduced hierarchical feature learning on point clouds, and various other models focusing on exploiting the local geometric correlations to enhance point cloud processing. These improvements come from further innovations in sampling methods, grouping strategies, and advanced mapping functions that substantially improve model accuracy and applicability across 3D vision tasks including classification, segmentation, and detection.

Performance evaluations on standard benchmark datasets, such as ModelNet40 for classification, ShapeNet for part segmentation, and S3DIS for semantic segmentation, provide empirical evidence of these approaches' advances. Moreover, the implications of these methodologies extend to practical applications in critical areas such as autonomous driving, where precise environmental perception through object classification and detection is paramount.

Looking forward, future developments in AI point cloud processing might focus on overcoming challenges associated with scalability and efficiency in handling large-scale data, as well as improving robustness against noise and dynamic environmental conditions. Integration with other sensor modalities and advancing real-time processing capabilities will also be pivotal in broadening the application scope of these deep learning techniques in 3D perception and beyond.