CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Abstract: Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality (X-modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for modality fusion. Specifically, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate bi-modal features by leveraging the features from one modality to rectify the features of the other modality. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to perform sufficient exchange of long-range contexts before mixing. To verify CMX, for the first time, we unify five modalities complementary to RGB, i.e., depth, thermal, polarization, event, and LiDAR. Extensive experiments show that CMX generalizes well to diverse multi-modal fusion, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available at https://github.com/huaaaliu/RGBX_Semantic_Segmentation.

• The paper presents a transformer-based framework that fuses RGB and complementary modalities through cross-modal feature rectification and bidirectional feature fusion.
• It demonstrates state-of-the-art performance on RGB-Depth, RGB-Thermal, and RGB-Event benchmarks with significant mIoU improvements.
• The CMX architecture, featuring the CM-FRM and FFM, enables effective multi-modal interaction and reliable segmentation for autonomous vehicle applications.

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Introduction

The complexity of semantic segmentation for Autonomous Vehicles (AVs) has drawn considerable attention, primarily due to its critical role in providing detailed scene understanding for Advanced Driver-Assistance Systems (ADAS). Segmenting RGB images can be significantly enhanced by integrating complementary features from additional sensing modalities such as depth, thermal, polarization, event, and LiDAR data. The CMX framework is proposed as a unified solution for RGB-X semantic segmentation, overcoming the limitations that arise from the varying characteristics of different sensors by leveraging a transformer-based approach. CMX's architecture includes a Cross-Modal Feature Rectification Module (CM-FRM) and a Feature Fusion Module (FFM) to achieve comprehensive cross-modal interaction and effective feature calibration, ultimately aiming for robust segmentation across diverse modality combinations.

Figure 1: RGB-X semantic segmentation unifies diverse sensing modality combinations: RGB-Depth, -Thermal, -Polarization, -Event, and -LiDAR segmentation.

Methodology

Framework Overview

CMX is structured as a two-stream architecture, where separate branches process RGB and X-modal data, facilitating parallel but interactive feature extraction (Figure 2). Key modules within this architecture are the Cross-Modal Feature Rectification Module (CM-FRM) and the Feature Fusion Module (FFM).

Figure 2: a) Overview of CMX for RGB-X semantic segmentation. b) CM-FRM with information flows of two modalities. c) FFM with stages for information exchange and fusion.

Cross-Modal Feature Rectification Module (CM-FRM)

CM-FRM addresses the challenge of noisy measurements intrinsic to different sensor modalities. It enhances RGB-X feature extraction using two sub-modules: channel-wise and spatial-wise feature rectification. The intent is to enable mutually beneficial interactions between modalities:

• Channel-wise Rectification: Introduces global weights by utilizing global max and average pooling to embed features along the spatial axis. This results in bidirectional calibration of features, thus improving multi-modal interaction.
• Spatial-wise Rectification: Embeds bi-modal features into spatial weight maps using consecutive convolutional layers. These maps rectify each other by exploiting spatial and channel-wise correlations.

At the output, rectified modal features are fused into the next stage, allowing for further enhancement of the learned features.

Feature Fusion with FFM

Within the Feature Fusion Module (FFM), feature interactions and fusions occur at multiple levels. The module is split into two primary stages:

• Information Exchange Stage: Utilizes a dual-path structure for bidirectional cross-modal feature rectification, incorporating a cross-attention mechanism. This enables enhanced cross-modal interactions and improved multi-modal feature extraction.
• Fusion Stage: Implements a mixed channel embedding to combine rectified features, resulting in comprehensive multi-modal feature maps (Figure 3).

(Figure 3)

Figure 4: Comparison of different fusion methods. (a) Input fusion merges inputs with modality-specific operations. (b) Feature fusion applies channel attention in a unidirectional manner. (c) Our interactive fusion employs bidirectional cross-modal feature rectification and sequence-to-sequence cross-attention.

Experimental Results and Analysis

Results on RGB-Depth Datasets

Experiments conducted on multiple datasets demonstrated CMX's superiority in RGB-Depth semantic segmentation. In the NYU Depth V2 benchmark, CMX achieves an mIoU of up to 56.9%, surpassing existing methods designed specifically for RGB-D data.

Key Numerical Results:

• NYU Depth V2 Benchmark:
  • CMX with MiT-B5 achieved an mIoU of 56.9% with pixel accuracy of 80.1%.
  • 

  • Figure 1: Performance comparison on different RGB-X semantic segmentation benchmarks, demonstrating the superior versatility of CMX over other methods.

Results on Additional RGB-X Modalities

• RGB-Thermal (MFNet Dataset): CMX outpaces earlier methods by achieving a mean Intersection over Union (mIoU) of 59.7%, clearly outperforming specialized models like ABMDRNet and GMNet. Notably, pedestrian segmentation accuracy improved due to more effective utilization of thermal data characteristics, achieving an improvement of over 11%.

(Table 1)

Table 1: Per-class results on MFNet dataset for RGB-Thermal segmentation.

Results on RGB-Event Dataset

CMX establishes a robust RGB-Event semantic segmentation benchmark based on the EventScape dataset. The model sets a new state-of-the-art with substantial improvements in performance metrics over previous methods, confirming the effectiveness of CMX for RGB-X tasks.

Numerical Results:

• On the EventScape dataset, CMX with MiT-B4 achieved the highest mIoU of 64.28% with pixel accuracy of 92.60%.

  Figure 5: Per-class IoU results of the RGB-only baseline and our RGB-Event model on our RGB-Event benchmark.

Conclusion

The CMX framework represents an advance in the area of RGB-X semantic segmentation by offering a universal solution for robust multi-modal scene understanding applicable across a multitude of sensory combinations. The framework demonstrates significant improvements over modality-specific methods on multiple benchmarks, extraordinarily enhancing performance in scenarios characterized by supplements and uncertainties. Future research should focus on refining cross-modal fusion strategies, potentially leading to further improvement and application across broader sensor combinations and domain variants.

Figure 6: Visualization of failure cases. From top to bottom: RGB-Depth, RGB-Thermal, RGB-Polarization (AoLP), and RGB-Event semantic segmentation.