SFR-Net: Learning Scale-Frustum Representations for Ultra-Wide Area Remote Sensing Image Segmentation

Abstract: Pixel count and geographical coverage are two key characteristics of remote sensing images. Existing remote sensing image segmentation methods typically focus on images with either a small pixel count or a large pixel count but limited geographical coverage. In this paper, we introduce a novel segmentation task targeting ultra-wide area (UWA) remote sensing images, characterized by both a large pixel count and extremely wide geographical coverage. The core challenges of UWA segmentation lie in simultaneously handling ground objects with significantly varying scales and maintaining long-range contextual semantic continuity. To address these challenges, we propose the Scale-Frustum Representation Network (SFR-Net). Inspired by the viewing frustums of remote sensing images captured from different altitudes, we construct scale-frustum representations, enabling unified modeling of ground objects and contextual features at different scales. Furthermore, we design a cascaded cross-scale fusion mechanism to effectively integrate these representations, enhancing local semantic understanding while ensuring long-range contextual continuity. Experimental results on GID and FBPS demonstrate that SFR-Net achieves state-of-the-art performance, improving mIoU by 1.72% and 4.29%, respectively, over the strongest competing methods. In addition, the proposed scale-frustum representations can be integrated into generic segmentation networks to improve both segmentation accuracy and convergence speed. The implementation code will be publicly available at https://github.com/ChuyuZhong/SFR-Net.

• The paper proposes scale-frustum representations to effectively capture both fine local details and long-range context in ultra-wide remote sensing images.
• It introduces Cascaded Cross-Scale Fusion (CCSF) for robust multi-scale feature alignment, achieving mIoU improvements up to 4.29% on benchmark datasets.
• The approach integrates efficiently with existing models, advancing segmentation of complex landscapes with enhanced detail and semantic continuity.

SFR-Net for Ultra-Wide Area Remote Sensing Image Segmentation: Technical Summary

Motivation and Problem Formulation

Ultra-wide area (UWA) remote sensing image segmentation is characterized by high spatial resolution ($\geq 5000 \times 5000$ pixels) and broad geographical coverage ($\geq 500$ km$^2$), demanding simultaneous recognition of fine-grained local targets and preservation of long-range contextual semantic continuity. Previous segmentation methods either rely on patch-wise models which lose long-range dependencies, or process full images with lightweight encoders, resulting in loss of local granular detail. The task is further complicated by extreme object scale variability and scene complexity, as visualized in prominent examples which demonstrate both tiny urban features and expansive agricultural regions, alongside structures like roads and rivers requiring long-range context to ensure semantic coherence.

Figure 1: Comparison of generic, UHR, and UWA segmentation task regimes in terms of pixel count and coverage.

These dual challenges—handling highly varying scales and maintaining semantic continuity across vast spatial extents—are foundational, requiring new representation paradigms beyond traditional cropping or simple global fusion.

Figure 2: Illustration of the challenging scale variability and contextual continuity inherent to UWA segmentation, with examples showing small buildings versus extensive cropland, and fragmented versus continuous infrastructure.

SFR-Net Architecture and Methodology

SFR-Net (Scale-Frustum Representation Network) is designed around the observation that sensor frustums (i.e., virtual pyramids of view at different altitudes) naturally encapsulate multi-scale context. For each pixel or region (projection reference point, PRP), SFR-Net constructs a set of observation windows at increasing “distances,” each window covering progressively larger scenes but retaining alignment around the same local region. These windows are resized to a fixed input size and embedded with learnable scale identities. The architecture comprises a main encoder for high-fidelity local features and lightweight sub-encoders for broader-context windows.

Figure 3: SFR-Net pipeline showing the construction of scale-frustum representations, their fusion via CCSF, and whole-image prediction via iterative scanning.

Instead of early fusion, contextual features at different scales are integrated via Cascaded Cross-Scale Fusion (CCSF). In CCSF, feature alignment is achieved by MLP layers, then features are reduced via Feature Dimensionality Reduction (FDR), cross-attention computes semantic dependencies between adjacent scales, and finally Feature Dimensionality Expansion (FDE) restores the merged features for decoding.

Figure 4: A single fusion unit in CCSF, illustrating alignment, dimensionality reduction, cross-attention, and expansion for efficient multi-scale feature interaction.

Iterative scanning with multiple PRPs and overlapping inference mitigates discontinuities between local predictions. Main and auxiliary decoders are supervised by dice loss and cross-entropy, with loss weighting reflecting pixel-wise and overlap quality.

Experimental Results

Theoretical contributions are validated on GID and FBPS datasets, both comprising $7300 \times 6900$ pixel images and covering complex land-cover scenarios. SFR-Net set new benchmarks on both datasets, with mIoU improvements of 1.72% on GID and 4.29% on FBPS over previous best models. Importantly, SFR-Net achieves these results with modest increases in memory and computational overhead compared to REST and Swin-Large baselines.

Qualitative assessment confirms SFR-Net’s superior ability to capture fine local structures and ensure semantic continuity for elongated features (e.g., roads/rivers), contrasted with baseline models which present discontinuities and degraded performance for small or confusable classes.

Figure 5: Visual comparison showing SFR-Net's refined segmentation details and enhanced road continuity.

On FBPS, SFR-Net distinguishes confusing classes such as “river” versus “pond” by leveraging context at multiple scales, outperforming generic models that misclassify such structures due to their local similarity yet global semantic difference.

Figure 6: Comparison on semantically confusable regions, with SFR-Net distinguishing categories through multi-scale contextual structure.

SFR module integration into generic segmentation networks (UperNet, DeepLab, PSPNet) produced consistent accuracy gains and accelerated convergence, underscoring the generality and efficiency of SFR as a standalone module.

Figure 7: SFR-augmented models achieve higher mIoU and faster convergence, evidenced by training curves.

CCSF provided sharper edges and stronger foreground-background separation in encoder feature maps compared to baselines, as demonstrated by qualitative visualizations.

Figure 8: CCSF-enhanced feature maps exhibit improved boundary clarity and semantic continuity.

Ablation studies showed that the three-scale (local, short-range, long-range) configuration offers optimal trade-offs. Overlapping inference strategy had a limited yet positive effect, with SFR-Net achieving minimal dependence on overlap for continuity, unlike competing models.

Limitations and Discussion

Despite robust performance, SFR-Net’s reliance on visual and contextual information is insufficient for resolving ambiguities in highly similar classes (e.g., river vs. lake) when appearance and annotation boundaries converge or lack global topology differentiation.

Figure 9: Failure cases indicating persistent ambiguities between similar water body classes.

Computationally, multi-scale windowing and fusion incur increased overhead, motivating future research into adaptive scale selection and lightweight fusion mechanisms. Dataset annotation inconsistencies and inherent semantic ambiguities present ongoing challenges for UWA segmentation.

Implications and Future Directions

Theoretically, SFR-Net establishes a scalable context modeling paradigm suitable for tasks requiring both granular local discrimination and expansive global continuity, relevant for high-resolution semantic vision across remote sensing, environmental monitoring, and geospatial analytics. Practically, the framework enables integration into generic segmentation pipelines and offers accelerated convergence, facilitating broader deployment in real-world remote sensing workflows.

Future research should target dynamic scale determination, efficient cross-scale interaction, and ambiguity-resilient semantic reasoning. In addition, annotation robustness and domain adaptation across diverse UWA scenes are pressing topics. As foundation models and multimodal approaches mature, integrating scale-frustum representations with semantic priors and temporal context may further enhance UWA segmentation reliability.

Conclusion

SFR-Net effectively addresses the dual challenges of UWA remote sensing image segmentation by constructing aligned scale-frustum representations and fusing them via cascaded mechanisms. The method achieves superior quantitative and qualitative results, demonstrating improved accuracy, convergence, and contextual robustness. The framework promises flexible integration and motivates further advances in scalable vision architectures for geospatial intelligence and other domains requiring joint local-global modeling. Future directions include model efficiency, adaptive context, robust semantic inference, and broader applications in AI-driven remote sensing.