Multi-View Radar Semantic Segmentation (2103.16214v2)

Abstract: Understanding the scene around the ego-vehicle is key to assisted and autonomous driving. Nowadays, this is mostly conducted using cameras and laser scanners, despite their reduced performances in adverse weather conditions. Automotive radars are low-cost active sensors that measure properties of surrounding objects, including their relative speed, and have the key advantage of not being impacted by rain, snow or fog. However, they are seldom used for scene understanding due to the size and complexity of radar raw data and the lack of annotated datasets. Fortunately, recent open-sourced datasets have opened up research on classification, object detection and semantic segmentation with raw radar signals using end-to-end trainable models. In this work, we propose several novel architectures, and their associated losses, which analyse multiple "views" of the range-angle-Doppler radar tensor to segment it semantically. Experiments conducted on the recent CARRADA dataset demonstrate that our best model outperforms alternative models, derived either from the semantic segmentation of natural images or from radar scene understanding, while requiring significantly fewer parameters. Both our code and trained models are available at https://github.com/valeoai/MVRSS.

• The paper introduces three innovative architectures—MV-Net, MVA-Net, and TMVA-Net—that leverage multi-view radar data for improved semantic segmentation.
• The paper enhances segmentation performance using techniques like Atrous Spatial Pyramid Pooling and 3D convolutions, achieving superior mIoU and mDice metrics on the CARRADA dataset.
• The paper demonstrates that integrating radar sensor capabilities can strengthen autonomous vehicle perception in adverse weather, paving the way for robust sensor fusion methods.

An Analysis of Multi-View Radar Semantic Segmentation

The paper "Multi-View Radar Semantic Segmentation" addresses a pertinent challenge in the field of automotive perception, specifically focusing on the use of radar sensors to understand and segment scenes. Despite the ubiquity of optical sensors like cameras and LiDAR in automated driving systems, radar remains underexplored due to several inherent challenges. This work makes significant strides in utilizing radar data for semantic segmentation, proposing novel architectures that leverage the full potential of radar’s multi-view capabilities.

Context and Motivation

Radar sensors possess unique advantages over cameras and LiDAR, particularly under adverse weather conditions where optical sensors may falter. They are less affected by rain, fog, and snow, and provide valuable information such as the relative speed of surrounding objects. Historically, the use of radar in scene perception has been limited by the inadequacy of data processing techniques to handle the high noise levels and the size of radar datasets. With the recent development of annotated radar datasets, such as CARRADA, there exists an opportunity to harness radar data for scene understanding through advanced deep learning techniques.

Proposed Contributions

The authors propose several innovative neural network architectures tailored to the multi-view synthesis of radar data, namely MV-Net, MVA-Net, and TMVA-Net. These architectures process radar data in the form of a Range-Angle-Doppler (RAD) tensor, optimizing for semantic segmentation across different views:

1. MV-Net serves as the baseline, utilizing dual encoders and decoders to derive semantic segmentation results for both range-angle and range-Doppler slices of radar data.
2. MVA-Net enhances this with Atrous Spatial Pyramid Pooling (ASPP) modules, aiming to better capture objects at different scales and improve the coherence between multiple views.
3. TMVA-Net adds a temporal dimension to the analysis, using 3D convolutions to exploit temporal radar signals to improve object recognition, achieving better segmentation continuity across frames.

Key Results and Performance Metrics

The proposed models were evaluated using the CARRADA dataset across metrics like the mean Intersection over Union (mIoU) and the mean Dice coefficient. TMVA-Net consistently outperformed other methods including specialized existing architectures like DeepLabv3+ and RSS-Net, achieving a balance between segmentation accuracy and computational efficiency.

1. TMVA-Net demonstrated strong performance on both range-Doppler and range-angle tasks, with the highest mIoU and mDice scores compared to competing architectures.
2. The use of a coherence loss in TMVA-Net promoted consistent object recognition across different radar views, increasing segmentation accuracy.

Practical and Theoretical Implications

The adoption of these radar-based semantic segmentation architectures could significantly enhance the robustness and reliability of perception systems in autonomous vehicles, especially in challenging environments that degrade the performance of optical sensors. From a theoretical standpoint, this approach demonstrates the potential of exploiting radar’s intrinsic features, such as Doppler information, for rich, multi-dimensional scene understanding. It encourages further research into sensor fusion strategies and adaptive models that can leverage the strengths of radar in concert with other sensor modalities.

Future Directions

The research sets the foundation for future inquiry into radar-based scene analysis, suggesting areas such as:

• Enhanced segmentation techniques that further mitigate the challenges of radar noise and resolution.
• Exploration of data augmentation strategies specific to radar to improve machine learning outcomes.
• Integration with other sensor-based systems for a more comprehensive environmental perception model.

This work represents a promising advance in automotive sensor fusion, highlighting the capabilities of radar sensors beyond their traditional applications and paving the way for more resilient autonomous driving technologies.