AlignEarth: Optical-to-SAR Distillation

• AlignEarth is a cross-modal distillation method that transfers semantic structure from a frozen optical CLIP encoder to a trainable SAR encoder for open-vocabulary segmentation.
• The approach leverages paired optical–SAR imagery with global contrastive loss, direct distillation, and region-level local alignment to bridge modal gaps.
• Empirical results show mIoU improvements from 10.6% to 34.0% over direct CLIP application, highlighting effective integration with the SegEarth-OV pipeline despite sensor-specific challenges.

AlignEarth most explicitly denotes a cross-modal knowledge distillation strategy introduced within SegEarth-OV for annotation-free open-vocabulary semantic segmentation of synthetic aperture radar (SAR) imagery. In that formulation, a frozen optical Vision-LLM (VLM) encoder, instantiated as CLIP’s image encoder, serves as teacher, and a trainable SAR encoder serves as student; the objective is to make SAR features semantically compatible with the optical CLIP feature space, so that the same text-driven segmentation pipeline used for optical remote-sensing imagery can operate on SAR without SAR-specific text annotations or SAR VLM pre-training (Li et al., 25 Aug 2025). In adjacent literature, the term also appears in broader “AlignEarth-style” senses, including Earth-referenced scene grounding, satellite–map alignment, generative 3D Earth modeling, and global embedding fields, but those usages describe related alignment goals rather than the same method (Cohen et al., 27 Mar 2026, Dong et al., 28 Apr 2025, Qian et al., 8 Jun 2026, Brown et al., 29 Jul 2025).

1. Conceptual definition and system role

AlignEarth is a bridge between modalities inside SegEarth-OV. The underlying problem is that SegEarth-OV’s core open-vocabulary semantic segmentation capabilities rely on a CLIP-like image encoder, SimFeatUp for recovering high-resolution spatial detail from coarse patch features, Global Bias Alleviation for correcting global-[CLS]-contaminated patch features, and text embeddings from the CLIP text encoder. That stack works naturally for optical imagery because CLIP is pre-trained on massive optical-like image–text data, but it does not transfer directly to SAR, where there is no SAR CLIP or comparable SAR VLM trained on millions of SAR–text pairs and where the imaging physics differ fundamentally through backscatter, speckle, and radar geometry (Li et al., 25 Aug 2025).

Within this design, AlignEarth avoids SAR–text alignment entirely. Instead, it uses paired optical–SAR images to distill the semantic structure of CLIP’s optical feature space into a dedicated SAR encoder. For optical inference, the pipeline is CLIP $\rightarrow$ SimFeatUp $\rightarrow$ Global Bias Alleviation $\rightarrow$ text similarity $\rightarrow$ segmentation. For SAR inference, the AlignEarth-trained SAR encoder replaces CLIP’s image encoder and is followed by the same SimFeatUp, Global Bias Alleviation, and text-similarity stages. The resulting claim is not that SAR has been endowed with an independently trained VLM, but that SAR features have been made plug-compatible with an optical open-vocabulary pipeline (Li et al., 25 Aug 2025).

A common misconception is to treat AlignEarth as a general-purpose SAR foundation model. In the paper’s formulation, it is instead a distillation-based adaptation mechanism whose purpose is to align SAR image features to an already existing optical semantic space. Its contribution is therefore architectural compatibility and semantic transfer, not SAR-native text supervision or SAR foundation-model construction from scratch (Li et al., 25 Aug 2025).

2. Mathematical formulation

Given a paired optical–SAR sample $(I_{opt}, I_{sar})$, AlignEarth defines a frozen optical encoder $\mathcal{E}_{opt}$ and a trainable SAR encoder $\mathcal{E}_{sar}$. The teacher output is

$$\mathcal{O}_{opt}=\mathcal{E}_{opt}(I_{opt})=[o_{opt,cls},o_{opt,1},\dots,o_{opt,hw}]\in\mathbb{R}^{(hw+1)\times c},$$

and the student output is

$$\mathcal{O}_{sar}=\mathcal{E}_{sar}(I_{sar})=[o_{sar,cls},o_{sar,1},\dots,o_{sar,hw}]\in\mathbb{R}^{(hw+1)\times c}.$$

Here, $hw$ is the number of spatial tokens and $\rightarrow$0 is the shared projection dimension. The objective is to align the two token sequences both globally, through the [CLS] token, and locally, through patch regions (Li et al., 25 Aug 2025).

Image-level alignment is imposed with a CLIP-like symmetric contrastive loss over global tokens. Using cosine similarity

$\rightarrow$1

and a learnable temperature $\rightarrow$2, the batch loss is

$\rightarrow$3

This enforces pairwise optical–SAR matching and batchwise discrimination in both directions. AlignEarth supplements this with a direct distillation term on global tokens,

$\rightarrow$4

which acts as teacher–student regression in the shared embedding space (Li et al., 25 Aug 2025).

Dense segmentation requires local semantic consistency, but exact tokenwise correspondence is brittle because optical–SAR pairs are often imperfectly co-registered and SAR-specific effects distort local structure. AlignEarth therefore uses region-level distillation. Patch tokens are divided into a $\rightarrow$5 grid of non-overlapping regions; mean feature vectors $\rightarrow$6 and $\rightarrow$7 are computed for corresponding cells, and the local loss is

$\rightarrow$8

This averages over coarse regions rather than enforcing exact pixelwise matching. The paper describes the training objective as a combination of the three losses,

$\rightarrow$9

with $\rightarrow$0 as balancing coefficients that are not explicitly specified (Li et al., 25 Aug 2025).

3. Architecture, data, and training protocol

The teacher encoder $\rightarrow$1 is CLIP’s ViT-based image encoder. It tokenizes the image into $\rightarrow$2, applies transformer self-attention and feed-forward blocks, and projects the final token sequence into the shared multi-modal embedding space. This encoder remains frozen throughout AlignEarth training. The student encoder $\rightarrow$3 is structurally identical or very similar to the CLIP image encoder, uses the same projection dimension $\rightarrow$4, is initialized from the OpenAI CLIP image encoder weights, and is then tuned on SAR through the AlignEarth losses. No additional adapters or multi-branch modules are introduced; the adaptation is loss-driven rather than architecture-driven (Li et al., 25 Aug 2025).

The required supervision is paired optical–SAR imagery without pixel labels or captions. The training set comprises about 65k pairs assembled from SpaceNet 6, MSAW, QXS-SAROPT, SAR2Opt, DFC2023 Track1, PIE-RGB-SAR, DFC2025 Track1, WHU-OPT-SAR, DDHR-SK, FUSAR-Map, and YESeg-OPT-SAR. The pairs are co-registered or weakly aligned, with some geometric and temporal discrepancies. This is sufficient for global alignment and region-level local distillation, but it does not imply strict pixel correspondence (Li et al., 25 Aug 2025).

The reported protocol uses AdamW, learning rate $\rightarrow$5, 20 training epochs, a single NVIDIA A100 GPU, and batch size 512 paired samples per iteration. The contrastive temperature is $\rightarrow$6; the local distillation grid size is $\rightarrow$7, giving 49 regions per feature map; and the later inference-time Global Bias Alleviation uses $\rightarrow$8. Cosine similarity implies L2-normalization during similarity computation, and no special feature normalization beyond what CLIP already uses is introduced. After training, the SAR encoder is frozen and deployed within SegEarth-OV (Li et al., 25 Aug 2025).

4. Integration with SegEarth-OV and inference semantics

For optical images, SegEarth-OV uses CLIP patch tokens, SimFeatUp for high-resolution upsampling, Global Bias Alleviation to subtract a scaled [CLS] contribution, and cosine similarity against text embeddings from the CLIP text encoder. The bias-alleviated patch features are

$\rightarrow$9

with $\rightarrow$0 in experiments. Text prompts are class names in CLIP prompt templates, and per-pixel labels are produced by argmax over cosine similarities between visual features and class text embeddings (Li et al., 25 Aug 2025).

For SAR images, AlignEarth modifies only the image encoder. The inference stack becomes AlignEarth-trained $\rightarrow$1 $\rightarrow$2 SimFeatUp $\rightarrow$3 Global Bias Alleviation $\rightarrow$4 CLIP text similarity. SimFeatUp is unchanged and reused without retraining; specifically, the same SimFeatUp module trained on optical remote-sensing images from Million-AID is applied to SAR features. Global Bias Alleviation is also unchanged:

$\rightarrow$5

Text embeddings continue to come from the optical CLIP text encoder, with no SAR-specific text branch added (Li et al., 25 Aug 2025).

The annotation-free character of the method is precise rather than absolute. No SAR pixel-level labels are used to train $\rightarrow$6; no SAR captions or SAR-specific text annotations are required; and AlignEarth uses only paired optical–SAR images plus CLIP’s existing optical–text alignment. At the same time, the approach still depends on optical CLIP pre-training and on an unannotated paired optical–SAR corpus. This distinction matters because “annotation-free” in the paper refers to the absence of SAR segmentation labels and SAR text supervision, not to the absence of all external data (Li et al., 25 Aug 2025).

5. Empirical performance and comparative evaluation

The paper evaluates SAR open-vocabulary semantic segmentation on eight datasets: OpenEarthMap-SAR, DDHR-Korea, DDHR-SD, DDHR-XA, FUSAR-Map, WHU-SAR, YESeg-SAR, and PIE-SAR. For each baseline, two variants are reported: “OpenAI,” which applies optical CLIP directly to SAR, and “AlignEarth,” which replaces the image encoder with the AlignEarth-trained SAR encoder. The central quantitative result is that SegEarth-OV improves from 10.6% average mIoU in the OpenAI configuration to 34.0% average mIoU with AlignEarth (Li et al., 25 Aug 2025).

Setting	Average mIoU	Note
SegEarth-OV (OpenAI)	10.6%	Optical CLIP used directly on SAR
SegEarth-OV (AlignEarth)	34.0%	AlignEarth-trained SAR encoder
SkyCLIP	15.4%	Best RS CLIP in Table 13
AlignEarth	30.2%	ClearCLIP-style OVSS comparison

Dataset-level gains are similarly large. On DDHR-Korea, SegEarth-OV rises from 16.5% mIoU to 48.6%; on PIE-SAR, from 12.1% to 51.1%; and on OpenEarthMap-SAR, from 6.3% to 19.3%. On PIE-SAR, the fully supervised Oracle based on SegFormer trained with labels reaches 63.2% mIoU, while SegEarth-OV with AlignEarth reaches 51.1%, which the paper characterizes as about 80% of oracle performance in a purely annotation-free regime (Li et al., 25 Aug 2025).

The comparison against generic and remote-sensing-specific CLIP variants is also notable. In ClearCLIP-style OVSS evaluation, CLIP, OpenCLIP, MetaCLIP, ALIP, GeoRSCLIP, RemoteCLIP, SkyCLIP, and CLIPSelf are all outperformed by AlignEarth; the best remote-sensing CLIP, SkyCLIP, reports 15.4% average mIoU, GeoRSCLIP 15.1%, RemoteCLIP 11.1%, CLIPSelf 4.4%, and AlignEarth 30.2%. The reported interpretation is that training CLIP on millions of optical remote-sensing image–text pairs does not by itself solve SAR OVSS, whereas cross-modal optical-to-SAR distillation does (Li et al., 25 Aug 2025).

6. Limitations, scope, and broader research context

The method has several explicit constraints. It depends on a substantial amount of paired optical–SAR imagery, about 65k pairs, and its robustness to weak registration relies on region-level distillation rather than exact correspondence. It assumes the availability of a strong optical VLM such as CLIP and therefore does not solve the foundation-model problem for SAR from scratch. It also inherits sensor-domain gap issues across SAR frequency, polarization, resolution, and incidence angle, and it aligns SAR to CLIP’s optical feature space rather than directly to text. A plausible implication is that performance on entirely novel SAR sensors may require further distillation or domain adaptation (Li et al., 25 Aug 2025).

The name “AlignEarth” also appears in a wider alignment literature, but with different referents. “Scene Grounding in the Wild” describes its framework as effectively a scene-scale AlignEarth module: partial 3D reconstructions from in-the-wild imagery are grounded to a geospatially anchored reference model derived from Google Earth Studio, represented with semantic 3D Gaussian Splatting, and aligned through inverse feature-based optimization of global pose and scale (Cohen et al., 27 Mar 2026). “EarthMapper” and CNSatMap are described as a proto-AlignEarth system for controllable bidirectional satellite–map translation, centered on geographic coordinate embeddings, Geo-conditioned Joint Scale Autoregression, Semantic Infusion, and Key Point Adaptive Guidance (Dong et al., 28 Apr 2025).

Related work extends the idea further toward planetary-scale infrastructure. ABot-Earth 0.5 is presented as the core physical layer of an AlignEarth system: a generative 3D Earth model that turns georeferenced satellite imagery into globally indexed 3D Gaussian Splatting tiles with hierarchical level-of-detail and web-map streaming (Qian et al., 8 Jun 2026). AlphaEarth Foundations is presented as a natural backbone for an AlignEarth-style system because it supplies a 64-dimensional global embedding field, annually from 2017 through 2024, that supports accurate downstream mapping from sparse labels without retraining the backbone (Brown et al., 29 Jul 2025). This suggests that the term now spans several levels of abstraction: cross-modal encoder alignment for SAR segmentation, Earth-referenced scene grounding, cross-modal cartographic translation, generative 3D base layers, and foundation representations for global mapping. Within that spectrum, however, AlignEarth in the strict technical sense remains the optical-to-SAR distillation strategy introduced as part of SegEarth-OV (Li et al., 25 Aug 2025).