---
title: 'AlignEarth: Optical-to-SAR Distillation'
url: https://www.emergentmind.com/topics/alignearth
type: topic
---

# AlignEarth: Optical-to-SAR Distillation

AlignEarth most explicitly denotes a cross-modal knowledge distillation strategy introduced within SegEarth-OV for annotation-free open-vocabulary semantic segmentation of synthetic aperture radar (SAR) imagery. In that formulation, a frozen optical Vision-Language Model (VLM) encoder, instantiated as CLIP’s image encoder, serves as teacher, and a trainable SAR encoder serves as student; the objective is to make SAR features semantically compatible with the optical CLIP feature space, so that the same text-driven segmentation pipeline used for optical remote-sensing imagery can operate on SAR without SAR-specific text annotations or SAR VLM pre-training [2508.18067]. In adjacent literature, the term also appears in broader “AlignEarth-style” senses, including Earth-referenced scene grounding, satellite–map alignment, generative 3D Earth modeling, and global embedding fields, but those usages describe related alignment goals rather than the same method [2603.26584][2504.19432][2606.09967][2507.22291].

## 1. Conceptual definition and system role

AlignEarth is a bridge between modalities inside SegEarth-OV. The underlying problem is that SegEarth-OV’s core open-vocabulary semantic segmentation capabilities rely on a CLIP-like image encoder, SimFeatUp for recovering high-resolution spatial detail from coarse patch features, Global Bias Alleviation for correcting global-[CLS]-contaminated patch features, and text embeddings from the CLIP text encoder. That stack works naturally for optical imagery because CLIP is pre-trained on massive optical-like image–text data, but it does not transfer directly to SAR, where there is no SAR CLIP or comparable SAR VLM trained on millions of SAR–text pairs and where the imaging physics differ fundamentally through backscatter, speckle, and radar geometry [2508.18067].

Within this design, AlignEarth avoids SAR–text alignment entirely. Instead, it uses paired optical–SAR images to distill the semantic structure of CLIP’s optical feature space into a dedicated SAR encoder. For optical inference, the pipeline is CLIP $\rightarrow$ SimFeatUp $\rightarrow$ Global Bias Alleviation $\rightarrow$ text similarity $\rightarrow$ segmentation. For SAR inference, the AlignEarth-trained SAR encoder replaces CLIP’s image encoder and is followed by the same SimFeatUp, Global Bias Alleviation, and text-similarity stages. The resulting claim is not that SAR has been endowed with an independently trained VLM, but that SAR features have been made plug-compatible with an optical open-vocabulary pipeline [2508.18067].

A common misconception is to treat AlignEarth as a general-purpose SAR foundation model. In the paper’s formulation, it is instead a distillation-based adaptation mechanism whose purpose is to align SAR image features to an already existing optical semantic space. Its contribution is therefore architectural compatibility and semantic transfer, not SAR-native text supervision or SAR foundation-model construction from scratch [2508.18067].

## 2. Mathematical formulation

Given a paired optical–SAR sample $(I_{opt}, I_{sar})$, AlignEarth defines a frozen optical encoder $\mathcal{E}_{opt}$ and a trainable SAR encoder $\mathcal{E}_{sar}$. The teacher output is
$$
\mathcal{O}_{opt}=\mathcal{E}_{opt}(I_{opt})=[o_{opt,cls},o_{opt,1},\dots,o_{opt,hw}]\in\mathbb{R}^{(hw+1)\times c},
$$
and the student output is
$$
\mathcal{O}_{sar}=\mathcal{E}_{sar}(I_{sar})=[o_{sar,cls},o_{sar,1},\dots,o_{sar,hw}]\in\mathbb{R}^{(hw+1)\times c}.
$$
Here, $hw$ is the number of spatial tokens and $c$ is the shared projection dimension. The objective is to align the two token sequences both globally, through the [CLS] token, and locally, through patch regions [2508.18067].

Image-level alignment is imposed with a CLIP-like symmetric contrastive loss over global tokens. Using cosine similarity
$$
\text{sim}(a,b)=\frac{a^\top b}{\|a\|_2\|b\|_2},
$$
and a learnable temperature $\tau$, the batch loss is
$$
\begin{split}
\mathcal{L}_{cls\_contrast}=-\frac{1}{N}\sum_{i=1}^{N}&\Bigg[
\log\frac{\exp\left(\text{sim}\left(\mathcal{O}_{opt\_cls}^{(i)},\mathcal{O}_{sar\_cls}^{(i)}\right)/\tau\right)}
{\sum_{j=1}^{N}\exp\left(\text{sim}\left(\mathcal{O}_{opt\_cls}^{(i)},\mathcal{O}_{sar\_cls}^{(j)}\right)/\tau\right)} \\
&+
\log\frac{\exp\left(\text{sim}\left(\mathcal{O}_{sar\_cls}^{(i)},\mathcal{O}_{opt\_cls}^{(i)}\right)/\tau\right)}
{\sum_{j=1}^{N}\exp\left(\text{sim}\left(\mathcal{O}_{sar\_cls}^{(i)},\mathcal{O}_{opt\_cls}^{(j)}\right)/\tau\right)}
\Bigg].
\end{split}
$$
This enforces pairwise optical–SAR matching and batchwise discrimination in both directions. AlignEarth supplements this with a direct distillation term on global tokens,
$$
\mathcal{L}_{cls\_distill}=1-\text{sim}(\mathcal{O}_{opt\_cls},\mathcal{O}_{sar\_cls}),
$$
which acts as teacher–student regression in the shared embedding space [2508.18067].

Dense segmentation requires local semantic consistency, but exact tokenwise correspondence is brittle because optical–SAR pairs are often imperfectly co-registered and SAR-specific effects distort local structure. AlignEarth therefore uses region-level distillation. Patch tokens are divided into a $K\times K$ grid of non-overlapping regions; mean feature vectors $\bar f_{opt\_patch}^{(i,j)}$ and $\bar f_{sar\_patch}^{(i,j)}$ are computed for corresponding cells, and the local loss is
$$
\mathcal{L}_{local\_distill}
=
\frac{1}{K^2}\sum_{i=1}^{K}\sum_{j=1}^{K}
\left(
1-\text{sim}\left(\bar f_{opt\_patch}^{(i,j)},\bar f_{sar\_patch}^{(i,j)}\right)
\right).
$$
This averages over coarse regions rather than enforcing exact pixelwise matching. The paper describes the training objective as a combination of the three losses,
$$
\mathcal{L}_{AlignEarth}=\alpha\mathcal{L}_{cls\_contrast}+\beta\mathcal{L}_{cls\_distill}+\gamma\mathcal{L}_{local\_distill},
$$
with $\alpha,\beta,\gamma$ as balancing coefficients that are not explicitly specified [2508.18067].

## 3. Architecture, data, and training protocol

The teacher encoder $\mathcal{E}_{opt}$ is CLIP’s ViT-based image encoder. It tokenizes the image into $[x_{cls},x_1,\dots,x_{hw}]$, applies transformer self-attention and feed-forward blocks, and projects the final token sequence into the shared multi-modal embedding space. This encoder remains frozen throughout AlignEarth training. The student encoder $\mathcal{E}_{sar}$ is structurally identical or very similar to the CLIP image encoder, uses the same projection dimension $c$, is initialized from the OpenAI CLIP image encoder weights, and is then tuned on SAR through the AlignEarth losses. No additional adapters or multi-branch modules are introduced; the adaptation is loss-driven rather than architecture-driven [2508.18067].

The required supervision is paired optical–SAR imagery without pixel labels or captions. The training set comprises about 65k pairs assembled from SpaceNet 6, MSAW, QXS-SAROPT, SAR2Opt, DFC2023 Track1, PIE-RGB-SAR, DFC2025 Track1, WHU-OPT-SAR, DDHR-SK, FUSAR-Map, and YESeg-OPT-SAR. The pairs are co-registered or weakly aligned, with some geometric and temporal discrepancies. This is sufficient for global alignment and region-level local distillation, but it does not imply strict pixel correspondence [2508.18067].

The reported protocol uses AdamW, learning rate $1\times10^{-4}$, 20 training epochs, a single NVIDIA A100 GPU, and batch size 512 paired samples per iteration. The contrastive temperature is $\tau=0.07$; the local distillation grid size is $K=7$, giving 49 regions per feature map; and the later inference-time Global Bias Alleviation uses $\lambda=0.3$. Cosine similarity implies L2-normalization during similarity computation, and no special feature normalization beyond what CLIP already uses is introduced. After training, the SAR encoder is frozen and deployed within SegEarth-OV [2508.18067].

## 4. Integration with SegEarth-OV and inference semantics

For optical images, SegEarth-OV uses CLIP patch tokens, SimFeatUp for high-resolution upsampling, Global Bias Alleviation to subtract a scaled [CLS] contribution, and cosine similarity against text embeddings from the CLIP text encoder. The bias-alleviated patch features are
$$
\hat{\mathcal{O}}=\mathcal{O}[1:hw+1]-\lambda\mathcal{O}[0],
$$
with $\lambda=0.3$ in experiments. Text prompts are class names in CLIP prompt templates, and per-pixel labels are produced by argmax over cosine similarities between visual features and class text embeddings [2508.18067].

For SAR images, AlignEarth modifies only the image encoder. The inference stack becomes AlignEarth-trained $\mathcal{E}_{sar}$ $\rightarrow$ SimFeatUp $\rightarrow$ Global Bias Alleviation $\rightarrow$ CLIP text similarity. SimFeatUp is unchanged and reused without retraining; specifically, the same SimFeatUp module trained on optical remote-sensing images from Million-AID is applied to SAR features. Global Bias Alleviation is also unchanged:
$$
\hat{\mathcal{O}}_{sar}=\mathcal{O}_{sar}[1:hw+1]-\lambda\mathcal{O}_{sar}[0].
$$
Text embeddings continue to come from the optical CLIP text encoder, with no SAR-specific text branch added [2508.18067].

The annotation-free character of the method is precise rather than absolute. No SAR pixel-level labels are used to train $\mathcal{E}_{sar}$; no SAR captions or SAR-specific text annotations are required; and AlignEarth uses only paired optical–SAR images plus CLIP’s existing optical–text alignment. At the same time, the approach still depends on optical CLIP pre-training and on an unannotated paired optical–SAR corpus. This distinction matters because “annotation-free” in the paper refers to the absence of SAR segmentation labels and SAR text supervision, not to the absence of all external data [2508.18067].

## 5. Empirical performance and comparative evaluation

The paper evaluates SAR open-vocabulary semantic segmentation on eight datasets: OpenEarthMap-SAR, DDHR-Korea, DDHR-SD, DDHR-XA, FUSAR-Map, WHU-SAR, YESeg-SAR, and PIE-SAR. For each baseline, two variants are reported: “OpenAI,” which applies optical CLIP directly to SAR, and “AlignEarth,” which replaces the image encoder with the AlignEarth-trained SAR encoder. The central quantitative result is that SegEarth-OV improves from 10.6% average mIoU in the OpenAI configuration to 34.0% average mIoU with AlignEarth [2508.18067].

| Setting | Average mIoU | Note |
|---|---:|---|
| SegEarth-OV (OpenAI) | 10.6% | Optical CLIP used directly on SAR |
| SegEarth-OV (AlignEarth) | 34.0% | AlignEarth-trained SAR encoder |
| SkyCLIP | 15.4% | Best RS CLIP in Table 13 |
| AlignEarth | 30.2% | ClearCLIP-style OVSS comparison |

Dataset-level gains are similarly large. On DDHR-Korea, SegEarth-OV rises from 16.5% mIoU to 48.6%; on PIE-SAR, from 12.1% to 51.1%; and on OpenEarthMap-SAR, from 6.3% to 19.3%. On PIE-SAR, the fully supervised Oracle based on SegFormer trained with labels reaches 63.2% mIoU, while SegEarth-OV with AlignEarth reaches 51.1%, which the paper characterizes as about 80% of oracle performance in a purely annotation-free regime [2508.18067].

The comparison against generic and remote-sensing-specific CLIP variants is also notable. In ClearCLIP-style OVSS evaluation, CLIP, OpenCLIP, MetaCLIP, ALIP, GeoRSCLIP, RemoteCLIP, SkyCLIP, and CLIPSelf are all outperformed by AlignEarth; the best remote-sensing CLIP, SkyCLIP, reports 15.4% average mIoU, GeoRSCLIP 15.1%, RemoteCLIP 11.1%, CLIPSelf 4.4%, and AlignEarth 30.2%. The reported interpretation is that training CLIP on millions of optical remote-sensing image–text pairs does not by itself solve SAR OVSS, whereas cross-modal optical-to-SAR distillation does [2508.18067].

## 6. Limitations, scope, and broader research context

The method has several explicit constraints. It depends on a substantial amount of paired optical–SAR imagery, about 65k pairs, and its robustness to weak registration relies on region-level distillation rather than exact correspondence. It assumes the availability of a strong optical VLM such as CLIP and therefore does not solve the foundation-model problem for SAR from scratch. It also inherits sensor-domain gap issues across SAR frequency, polarization, resolution, and incidence angle, and it aligns SAR to CLIP’s optical feature space rather than directly to text. A plausible implication is that performance on entirely novel SAR sensors may require further distillation or domain adaptation [2508.18067].

The name “AlignEarth” also appears in a wider alignment literature, but with different referents. “Scene Grounding in the Wild” describes its framework as effectively a scene-scale AlignEarth module: partial 3D reconstructions from in-the-wild imagery are grounded to a geospatially anchored reference model derived from Google Earth Studio, represented with semantic 3D Gaussian Splatting, and aligned through inverse feature-based optimization of global pose and scale [2603.26584]. “EarthMapper” and CNSatMap are described as a proto-AlignEarth system for controllable bidirectional satellite–map translation, centered on geographic coordinate embeddings, Geo-conditioned Joint Scale Autoregression, Semantic Infusion, and Key Point Adaptive Guidance [2504.19432].

Related work extends the idea further toward planetary-scale infrastructure. ABot-Earth 0.5 is presented as the core physical layer of an AlignEarth system: a generative 3D Earth model that turns georeferenced satellite imagery into globally indexed 3D Gaussian Splatting tiles with hierarchical level-of-detail and web-map streaming [2606.09967]. AlphaEarth Foundations is presented as a natural backbone for an AlignEarth-style system because it supplies a 64-dimensional global embedding field, annually from 2017 through 2024, that supports accurate downstream mapping from sparse labels without retraining the backbone [2507.22291]. This suggests that the term now spans several levels of abstraction: cross-modal encoder alignment for SAR segmentation, Earth-referenced scene grounding, cross-modal cartographic translation, generative 3D base layers, and foundation representations for global mapping. Within that spectrum, however, AlignEarth in the strict technical sense remains the optical-to-SAR distillation strategy introduced as part of SegEarth-OV [2508.18067].

Source: https://www.emergentmind.com/topics/alignearth