- The paper presents a novel PanoSAM2 framework that integrates a pano-aware decoder, distortion-guided mask loss, and a long-short memory module to address ERP distortions and seam inconsistencies in 360° video segmentation.
- It achieves significant performance gains, including a 5.6-point improvement in J&F on the 360VOTS benchmark while maintaining low GPU memory overhead.
- The experimental results demonstrate robust mask coherence and segmentation fidelity across challenging panoramic scenarios, enabling reliable applications in VR/AR and autonomous systems.
Lightweight Distortion- and Memory-Aware Adaptation of SAM2 for 360° Video Object Segmentation
Introduction
Conventional video object segmentation (VOS) frameworks are ill-suited for omnidirectional video due to geometric distortions imposed by equirectangular projection (ERP) and the inherent continuity across the 0°/360° seam. The "PanoSAM2" framework addresses the unique challenges posed by 360° video object segmentation (360VOS), such as projection-induced distortion, seam inconsistency, and sparse target objects, by integrating distortion-awareness and refined memory strategies into the segment-anything model (SAM2) backbone. This work introduces a novel, efficient approach that maintains promptable segmentation while explicitly modeling the geometric and temporal complexities of panoramic data.
Motivation and Limitations of Existing Methods
While foundational models like SAM2 demonstrate robust performance in conventional video segmentation, their direct application to 360VOS yields suboptimal results. The projection distortion in ERP images non-uniformly magnifies spatial regions, especially at extreme latitudes, leading to IoU degradation and fragmented mask contours. Additionally, objects crossing the 0°/360° seam often suffer identity breaks due to a lack of explicit seam handling in standard mask decoders. Existing adaptations for panoramic segmentation (e.g., BFoV, PSCFormer) introduce specialized attention or windowing, but often at the expense of efficiency or robustness, particularly under long-term occlusions and extreme appearance changes.
PanoSAM2 Framework
Pano-Aware Decoder
To enforce seamless left-right semantic continuity, PanoSAM2 employs a Pano-Aware (PA) decoder that utilizes left-right wrap padding and concatenation within convolutional layers, ensuring receptive fields bridge the 0°/360° seam. The decoder also incorporates feedback from previous mask predictions for iterative distortion refinement, facilitating robust tracking under abrupt projection-induced shifts and mitigating edge fragmentation. This geometry-aware mask decoding distinctly preserves object integrity across the ERP image domain without incurring heavy computational overhead.
Distortion-Guided Mask Loss
A distortion-guided mask loss is introduced to address the pixel-level class imbalance and spatial bias resulting from ERP. Here, per-pixel weights are computed based on distortion magnitude: foreground weights are proportional to their local density within a geometrically calibrated search region, while background weights decay with distance from the object boundary. This design amplifies the learning signal for high-uncertainty and severely distorted regions, resulting in sharper mask boundaries and enhanced segmentation of small or stretched targets. The loss is architecture-agnostic and is tightly coupled with the physical characteristics of omnidirectional capture.
Long-Short Memory Module (LSMM)
Effective segmentation in panoramic video requires stable temporal reasoning despite object sparsity and frequent occlusions. The LSMM module augments the short-term memory bank of SAM2 with compact, object-level long-term pointers distilled from frames with high predicted object visibility (as determined by an occlusion score). Key historical frames are selected via weighted sampling, promoting diversity in long-term context. LSMM then fuses these long-term pointers with recent short-term representations through FiLM modulation, ensuring that identity remains consistent even during long occlusions and that segmentation rapidly recovers as objects reappear.
Experimental Evaluation
Datasets and Benchmarks
PanoSAM2 is evaluated on major 360VOS benchmarks—360VOTS and PanoVOS—which together encompass broad spatial, temporal, and semantic variation. Both datasets feature dense, frame-level annotations facilitating rigorous comparison with perspective- and panorama-specific VOS baselines.
Quantitative Results
PanoSAM2 consistently achieves state-of-the-art results across all primary metrics. On 360VOTS, it delivers a +5.6 J&F improvement over SAM2 (65.8 vs. 60.2 with the Hiera-S backbone) with minimal GPU memory overhead (increment from 4.1GB to 4.3GB). On PanoVOS validation, it surpasses prior methods with a J&F of 78.1, a +6.7 gain over SAM2. Superior performance is also sustained for unseen object categories, demonstrating not only segmentation fidelity but also cross-domain generalization.
Ablations and Analysis
Ablation studies confirm that:
- The PA decoder contributes the largest single-component performance gain (+3.3 J&F).
- LSMM prevents drift and stabilizes tracking, especially under severe occlusion (+2.2 J&F gain).
- The distortion-guided loss further sharpens object boundaries (+0.7 J&F).
- Combined, these adaptations yield maximized accuracy and temporal coherence.
Moreover, increasing the long-term memory size L in LSMM provides diminishing returns beyond L=2, indicating the importance of balancing long- and short-term context for optimal conditioning.
Qualitative Analysis
Visual comparisons highlight that PanoSAM2 maintains mask connectivity and identity for objects straddling the panoramic seam, outperforms on small/distant targets, and exhibits robust recovery from occlusion, unlike both baseline SAM2 and panorama-adapted PSCFormer. The framework generalizes effectively across open-world 360° content and is less susceptible to identity swaps and mask fragmentation.
Limitations
Failure cases primarily involve (1) targets at extreme distances with minimal discriminative features and (2) fast vertical (latitude) motion resulting in severe projection deformation, where previous-frame masks become unreliable. These cases are typically challenging for all existing 360VOS frameworks.
Implications and Future Directions
PanoSAM2 illustrates that lightweight, geometry-aware architectural modifications and principled loss modeling significantly enhance foundation model expressiveness in omnidirectional scenarios. This work underscores the necessity of integrating spatial priors (e.g., ERP distortion) and robust temporal memory for panoramic video understanding. Practically, these advances have direct implications for VR/AR, autonomous navigation, and embodied AI systems, enabling stable object tracking in naturalistic 360° environments.
Future research directions include extending PanoSAM2 to multi-object segmentation and richer prompt types (points, boxes, free-form scribbles) for finer-grained, adaptive control. Additionally, integrating scene priors and context-aware memory selection could further enhance robustness in densely interactive or open-world settings.
Conclusion
PanoSAM2 establishes a new state-of-the-art in 360° video object segmentation by introducing distortion- and memory-aware adaptations to the SAM2 backbone. Its tightly integrated Pano-Aware decoder, distortion-guided loss, and long-short memory module jointly preserve mask integrity across panoramic seams, mitigate geometric distortion, and stabilize object identity under challenging temporal conditions. These advances demonstrate that panoramic-specific modeling is critical to elevating foundation segmentation models for immersive and generalizable video understanding tasks (2604.07901).