Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Abstract: In this paper, we address panoramic semantic segmentation which is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of semantic annotations in the 360{\deg} imagery. To tackle these problems, first, we propose the upgraded Transformer for Panoramic Semantic Segmentation, i.e., Trans4PASS+, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) modules for handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels). Second, we enhance the Mutual Prototypical Adaptation (MPA) strategy via pseudo-label rectification for unsupervised domain adaptive panoramic segmentation. Third, aside from Pinhole-to-Panoramic (Pin2Pan) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images, facilitating Synthetic-to-Real (Syn2Real) adaptation scheme in 360{\deg} imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with Pin2Pan and Syn2Real regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

• The paper proposes Trans4PASS+, incorporating deformable patch embedding and MLP modules to effectively handle panoramic image distortions.
• It introduces Mutual Prototypical Adaptation with pseudo-label rectification to enhance unsupervised domain adaptation in 360° imagery.
• The creation of the SynPASS dataset and extensive experiments yield up to 59.43% mIoU on DensePASS, setting new benchmarks for panoramic segmentation.

Adaptive Vision Transformers for Panoramic Semantic Segmentation

The paper "Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation" undertakes an exploration into the domain of panoramic semantic segmentation, an area challenged by inherent image distortions and the scarcity of annotated data for 360-degree imagery. With the increased integration of 360-degree cameras in diverse applications such as autonomous vehicles and AR/VR devices, the demand for more sophisticated and reliable panoramic imagery segmentation has heightened. This paper presents a transformative approach using vision transformers while addressing the fundamental problems posed by panoramic representations.

Core Contributions

The authors propose an upgraded transformer model named Trans4PASS+ alongside several methodologies designed to tackle these challenges:

1. Trans4PASS+ Architecture:
   • They introduce distortion-aware modules like Deformable Patch Embedding (DPE) and Deformable MLP (DMLPv2) to manage object deformation and image distortion across various levels of processing (pre/post-adaptation, shallow/deep layers).
2. Mutual Prototypical Adaptation (MPA):
   • MPA is enhanced via pseudo-label rectification aimed at unsupervised domain adaptive panoramic segmentation. This process allows for the generation of pseudo-labels from mutual prototypes across source and target domains, increasing robustness against inaccurate target domain pseudo-labels.
3. New Dataset - SynPASS:
   • The creation of SynPASS, encompassing 9,080 panoramic images, facilitates Synthetic-to-Real (Syn2Real) adaptation schemes. This dataset allows for comprehensive domain adaptation exploration beyond traditional Pinhole-to-Panoramic (Pin2Pan) paradigms.

Experimental Validation

Through extensive experiments covering both indoor and outdoor scenarios, each scrutinized under Pin2Pan and Syn2Real regimens, Trans4PASS+ demonstrates state-of-the-art performance across four benchmark datasets. The paper introduces compelling results:

• The models achieve up to 59.43% mIoU on DensePASS, outperforming prior benchmarks significantly.
• The SynPASS dataset leads to notable advancements in synthetic to real-world adaptation scenarios typically constrained by large domain shifts.

Comparative Analysis

The results highlight the versatility and performance improvement provided by Trans4PASS+ over previous models like PVT and SegFormer, with the distortion-aware architecture proving pivotal. The SynPASS dataset has introduced challenges accommodating its diverse conditions (weather, nighttime scenes) but facilitates groundbreaking accuracy improvements beyond existing segmentation datasets.

Implications and Future Directions

The developments within this paper pave the path forward towards robust and dynamic domain adaptation strategies for panoramic segmentation. The implications of these methods could extend into other areas of AI requiring similar adaptation methodologies, particularly where distortion acts as a significant hindrance. Future studies may look into integrating multi-modal sensor data, such as combining lidar information with panoramic images for further enhanced scene understanding.

In sum, "Behind Every Domain There is a Shift" marks a significant advancement in tackling panoramic semantic segmentation through adaptive distortion-aware vision transformers. Its novel SynPASS dataset and Mutual Prototypical Adaptation strategy highlight a promising direction in the effective utilization of synthetic data and domain adaptation, facilitating a more seamless deployment in real-world environments.