Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels

Abstract: Supervised learning for image segmentation typically requires spatially aligned image and label sets. When images and labels originate from different sources, the pairing may be misaligned, which can significantly deteriorate the performance of the learned models. This is especially common in remote sensing, when aerial or satellite images are co-registered with labels from another source (e.g., OpenStreetMap). In this work, we propose a novel approach for training on misaligned labels, where we simultaneously learn the label alignment. Our align and segment (AnS) approach builds on the spatial transformer module to transform the misaligned labels using an affine transformation to provide a better learning target for a canonical semantic segmentation network. We prevent shortcut learning of misaligned labels in these semantic segmentation networks through a self-supervised regularization loss and show that it is complementary to data augmentation, especially for systematically misaligned training data. A decisive characteristic of our AnS approach is that it learns without requiring any golden labels. We experimentally show on both synthetic and real-world data from different cities that our approach enables high-quality building segmentation and precise label-image alignment at the same time. Code and derived datasets are available at https://github.com/venkanna37/align-and-segment

• The paper proposes the AnS framework that simultaneously learns to align misaligned labels and perform semantic segmentation.
• It integrates a segmentation network with a transformation network using spatial transformer concepts and regularization to correct for affine misalignments.
• Empirical results show significant IoU improvements across synthetic and real-world datasets, demonstrating robust scalability and practical impact.

Unsupervised Label Alignment and Segmentation for Building Footprints

Problem Overview

Semantic segmentation in remote sensing commonly suffers from misaligned labels due to discrepancies in georeferencing between remote imagery (e.g., satellite) and external label sources (e.g., OpenStreetMap, OSM). Models trained on mismatched labels often overfit label noise, resulting in significant geospatial errors and limiting the utility of large, crowd-sourced label collections. "Align and Segment: Unsupervised Learning for Building Segmentation From Misaligned Labels" (2607.10841) directly addresses this by proposing an unsupervised methodology—Align and Segment (AnS)—that jointly learns to spatially align noisy labels and perform semantic segmentation, without any requirement for curated, perfectly aligned "golden" data.

Methodological Contributions

AnS employs a model-agnostic framework, integrating with any semantic segmentation architecture. The core technical innovation is the co-optimization of segmentation and label alignment within a unified training paradigm:

• Segmentation Network (SNet): Canonical semantic segmentation models (e.g., U-Net with ConvNeXt or ViT encoders) produce probabilistic masks from input images.
• Transformation Network (TNet): A lightweight module, inspired by spatial transformer networks, infers affine transformations between the SNet predictions and the (misaligned) label masks. The TNet operates in mask space, not image space, ensuring it is agnostic to input modality.
• Regularization: A consistency loss, computed by applying random transformations to misaligned labels and enforcing invariance under corrective transformations, steers the TNet away from degenerate solutions where the segmentation network adapts to misalignment rather than aligning labels.
• Data Augmentation: Aggressive geometric transformations (rotations, flips) on both images and masks explicitly decorrelate systematic label shifts, removing the incentive for the segmentation network to learn the bias.

This approach is fundamentally unsupervised from the perspective of label-image alignment: it does not presume access to perfectly aligned reference data, and it does not assume misalignment is small or unbiased.

Empirical Analysis

Datasets and Experimental Design

Experiments are conducted on both synthetic datasets—derived from the SpaceNet 2 collection for Las Vegas, Paris, and Khartoum with systematically (Dbias) and randomly (Duni) misaligned building footprints—and two real-world corpora: OpenStreetMap footprints atop WorldView-3 imagery (San Juan) and the ReBO dataset spanning 41 cities with both misaligned and golden labels.

Performance is evaluated using Intersection-over-Union (IoU) for three roles: segmentation accuracy relative to golden reference (IoUseg), alignment accuracy (IoUalign), and alignment between the transformed labels and SNet's prediction (IoUlearn). A broad spectrum of comparative baselines is considered, including MapRepair, Alignment Correction Network (supervised), Map Alignment, Spatial Correction, and AutoCorrect (unsupervised).

Quantitative Results

• Regularization and Augmentation: Simultaneous use of the regularization loss and data augmentation dramatically increases IoUseg and IoUalign, especially for systematically biased datasets. For example, on Las Vegas Dbias, IoUseg/+align increased from 0.39/0.41 (baseline) to 0.78/0.88 (AnS).
• Comparison to Baselines: AnS systematically outperforms unsupervised baselines across all cities and misalignment conditions. For instance, on Paris (Dbias), AnS achieves IoUseg/IoUalign of 0.52/0.65 versus 0.42/0.37 for Map Alignment and 0.54/0.36 for AutoCorrect. While supervised baselines (MR, ACN) reach even higher scores, they utilize golden labels for training or evaluation and cannot be deployed at continental scales in practice.
• Robustness: AnS maintains high IoU performance for misalignments up to 100-pixel translations or ±4.5° rotations. Systematic misalignments are more challenging: performance decays when shifts approach patch size (~70 pixels for Dbias), but well above the operational misalignment commonly observed between OSM and public imagery.

Qualitative Observations

On real-world San Juan OSM data lacking ground truth, qualitative analyses show effective correction of spatial misalignment, with the TNet estimating plausible affine corrections. Predicted segmentations remain slightly blurred around edges—a consequence of persistent annotation noise and decoder limitations.

Practical and Theoretical Implications

The AnS framework demonstrates that large-scale building segmentation can robustly leverage imperfect, misaligned label collections without intensive manual curation. This has direct impact potential for:

• Urban Analytics and Population Mapping: Automating the extraction of aligned building maps across the globe, including in data-deficient regions, without region-specific manual alignment, unlocks applications in urbanization monitoring, disaster assessment, and infrastructure analysis.
• Scalability: By operating without golden labels, the method enables scalable training on global open-data resources (e.g., OSM), previously hampered by spatial misregistration.
• Generalization: The decoupled design (arbitrary backbone, patch-level transformations) suggests broad applicability to other instance segmentation tasks (roads, farmlands) and alignment across multimodal or temporally shifted datasets.

Methodologically, the framework provides empirical clarification that simplistic inclusion of a spatial alignment module is insufficient: only with carefully designed regularization and augmentation can effective disentanglement of alignment and classification be achieved under high-capacity networks prone to shortcut learning.

Limitations and Future Directions

• Restricted Transformation Model: AnS is currently limited to affine transformations at the patch scale. Extension to deformable, high-complexity transforms is required for applications with substantial local distortion or label warping.
• Decoder and Segmentation Granularity: The current implementation employs a generic decoder without post-processing or task-specific optimization. More advanced segmentation heads or the inclusion of robust loss functions (e.g., for missing/corrupted object instances) could further improve spatial accuracy and label completeness.
• Missing Label Robustness: The framework does not explicitly address the problem of incomplete or partially missing annotations, a common feature of "open" label sets like OSM.

Further work could explore hierarchical or multi-resolution alignment, integration of uncertainty quantification for label trustworthiness, and adaptation to instance-level or vectorized outputs.

Conclusion

AnS provides a principled, unsupervised mechanism for disentangling and correcting label misalignment during semantic segmentation in geospatial and remote sensing domains. The empirical evidence demonstrates strong robustness to both random and systematic misalignment, effective scaling to heterogeneous global datasets, and competitive performance with state-of-the-art supervised and unsupervised baselines. These results showcase the feasibility of directly leveraging global-scale, crowd-sourced labels for high-precision building footprint segmentation, with significant implications for practical remote sensing and large-scale machine learning model training.