---
title: 'GRAD-Former: Robust Change Detection'
url: https://www.emergentmind.com/topics/grad-former
type: topic
---

# GRAD-Former: Robust Change Detection

Searching arXiv for the GRAD-Former paper and closely related change-detection context.
GRAD-Former, short for **Gated Robust Attention-based Differential Transformer**, is a **Siamese encoder-decoder framework** for **remote sensing change detection (CD)** that operates on **bitemporal satellite images** and predicts a **binary map** of changed versus unchanged pixels. It was introduced to address the difficulty of precisely delineating change regions in **very high-resolution (VHR)** imagery, where background clutter, structurally irregular changes, and **pseudo-changes** caused by shadows, illumination, seasonality, vehicle movement, and roof color variations can obscure semantic differences. The method combines a shared encoder, an **Adaptive Feature Relevance and Refinement (AFRAR)** module, a **Differential Amalgamation (DA)** fusion block, and a lightweight decoder, with the stated goal of enhancing contextual understanding while maintaining efficiency through reduced model size [2603.01161].

## 1. Problem setting and design motivation

Remote sensing CD seeks to identify semantic differences between satellite images captured at different times. In the formulation used by GRAD-Former, the task is to produce a binary change map from a pre-change image and a post-change image. The problem is especially difficult in VHR scenes because relevant changes may be small, sparse, and boundary-sensitive, while irrelevant appearance variations can dominate the visual signal [2603.01161].

The paper situates GRAD-Former against three established model families. **CNN-based methods** are described as effective for local spatial feature extraction but limited in modeling **long-range dependencies** and global contextual reasoning, which can impair sensitivity to subtle or spatially distributed changes. **Transformer-based methods** improve global modeling through self-attention, but in VHR remote sensing they incur **quadratic computational complexity**, high memory usage, large parameter counts, weak local inductive bias, and a **diffused attention** problem in which attention spreads too broadly and captures irrelevant background. **Selective State Space Models (SSMs)** such as Mamba are presented as linearly scaling alternatives that are strong at long-range sequence modeling but still challenged by **fine boundary delineation** and local-global balancing [2603.01161].

GRAD-Former is designed around a specific response to these limitations: it aims to **filter out irrelevant information early**, **amplify meaningful embeddings**, and **refine features using sparse differential attention**, while avoiding heavy full-width self-attention and not relying on a pretrained backbone. This positioning is central to understanding why the architecture combines gating, channel splitting, reduced-channel attention, and multi-scale differential fusion rather than adopting a conventional transformer encoder wholesale [2603.01161].

## 2. Overall architecture

At the highest level, GRAD-Former uses a **shared Siamese encoder** for the two input images, followed by stage-wise fusion and a decoder. The encoder produces multi-scale features at **4 stages**,
\[
\hat{\mathcal{F}}_{\text{pre}}^i,\ \hat{\mathcal{F}}_{\text{post}}^i,\quad i \in \{1,2,3,4\},
\]
and these paired features are fused at each stage by the **DA module** before being passed to the decoder [2603.01161].

The encoder stages use channel widths of **64, 96, 128, 256**. Each stage contains **patch embedding/downsampling**, an **AFRAR module**, and a **convolutional MLP layer**. Before AFRAR and the MLP, the input undergoes **instance normalization across channels**. The encoder block counts are reported as **3, 3, 4, 3**, with an input image size of **256 × 256 × 3** during training and evaluation [2603.01161].

AFRAR is the central architectural component. For a normalized feature tensor at stage \(i\),
\[
\mathcal{F}_{\text{norm}}^i \in \mathbb{R}^{H^i \times W^i \times C^i},
\]
the tensor is split channel-wise into two equal parts,
\[
\mathcal{F}_{\text{GLFR}}^i,\ \mathcal{F}_{\text{SEA}}^i = \text{Split}(\mathcal{F}_{\text{norm}}^i),
\]
which are processed by two complementary branches: **Selective Embedding Amplification (SEA)** and **Global-Local Feature Refinement (GLFR)**. Their outputs are then concatenated,
\[
\bar{\mathcal{F}}_{\text{norm}}^i = \text{Concat}(\bar{\mathcal{F}}_{\text{SEA}}^i,\ \bar{\mathcal{F}}_{\text{GLFR}}^i).
\]
This split-branch structure is the mechanism by which GRAD-Former distributes local/channel relevance processing and global-local contextual refinement across separate but coordinated operators [2603.01161].

## 3. AFRAR: selective amplification and differential attention

The **SEA** branch is a **gating-based channel reweighting module** intended to amplify channels containing useful change information while suppressing noisy or irrelevant channels. Given \(\mathcal{F}_{\text{SEA}}^i\), SEA first applies \(L_2\) normalization with a stabilizer \(\epsilon\), scaled by a learnable scalar \(\alpha\),
\[
E = \alpha \cdot \sqrt{\sum_{H,W}\left(\mathcal{F}_{\text{SEA}}^i\right)^2 + \epsilon}.
\]
It then computes an RMS-based normalization factor using a learnable scalar \(\gamma\),
\[
N = \frac{\gamma}{\sqrt{\mathbb{E}_c[E^2] + \epsilon}},
\]
followed by a gated activation with learnable bias \(\beta\),
\[
G = 1 + \tanh(E \cdot N + \beta).
\]
The output is
\[
\bar{\mathcal{F}}_{\text{SEA}}^i = \mathcal{F}_{\text{SEA}}^i \cdot G.
\]
The paper interprets this as selectively boosting relevant channel responses while attenuating irrelevant ones, with the \(1+\tanh(\cdot)\) gate used to stabilize scaling behavior [2603.01161].

The **GLFR** branch provides the attention mechanism in AFRAR and is explicitly described as being inspired by **differential attention**. From the input \(\mathcal{F}_{\text{GLFR}}^i\), the branch forms
\[
Q = W_Q \mathcal{F}_{\text{GLFR}}^i,\quad K = W_K \mathcal{F}_{\text{GLFR}}^i,\quad V = W_V \mathcal{F}_{\text{GLFR}}^i,
\]
with reduced channel dimensions:
\(Q, K \in \mathbb{R}^{B \times C/2 \times H \times W}\) and
\(V \in \mathbb{R}^{B \times C/4 \times H \times W}\).
The query and key tensors are then split,
\[
Q_1, Q_2 = \text{split}(Q),\quad K_1, K_2 = \text{split}(K),
\]
so that
\[
Q_1, Q_2, K_1, K_2 \in \mathbb{R}^{B \times C/4 \times H \times W}.
\]
These tensors are reshaped for multi-head attention using **\(h=4\) heads** and
\[
h_{\text{dim}} = \left\lfloor \frac{\text{input dimension}}{4 \times \text{head}} \right\rfloor.
\]
Two softmax attention maps are computed,
\[
A_1 = \sigma\left(\frac{Q_1^T K_1}{\sqrt{h_{\text{dim}}}}\right),\quad
A_2 = \sigma\left(\frac{Q_2^T K_2}{\sqrt{h_{\text{dim}}}}\right),
\]
and combined through subtraction,
\[
A = A_1 - \lambda \cdot A_2,
\]
where \(\lambda\) is learnable and defined by
\[
\lambda = \lambda_1 - \lambda_2 + \lambda_{\text{init}},
\]
with
\[
\lambda_1 = e^{(\lambda_{q_1} \cdot \lambda_{k_1})},\qquad
\lambda_2 = e^{(\lambda_{q_2} \cdot \lambda_{k_2})},
\]
and \(\lambda_{\text{init}} = 0.8\). The attention output is then
\[
\text{Attn} = A @ V^T.
\]
After reshaping, the refined attention features are concatenated with local features,
\[
l_{\text{feat}} = \text{reshape}(\mathcal{F}_{\text{GLFR}}^i),\qquad
\bar{\mathcal{F}}_{\text{GLFR}}^i = \text{concat}(l_{\text{feat}}, \text{Attn}).
\]
The paper’s interpretation is that the subtraction \(A_1 - \lambda A_2\) produces **sparser attention**, canceling common or redundant patterns and suppressing distractors so that meaningful change-related responses remain [2603.01161].

A common misunderstanding is to read GRAD-Former as a standard transformer variant with full self-attention over all channels. The architecture described in the paper is narrower and more selective: channels are split, attention is applied only to a **reduced channel subset**, and local features are explicitly retained through concatenation. In that sense, the “Transformer” component is structurally constrained by the model’s efficiency objective rather than used in its canonical dense form [2603.01161].

## 4. Differential fusion, decoder, and optimization

After encoding, GRAD-Former uses **Differential Amalgamation (DA)** to fuse the pre-change and post-change features at each stage. For stage \(i\), the fusion is
\[
\hat{\mathcal{F}}_{\text{fuse}}^i =
\sigma\left(
\text{Conv}_{1\times1}\left(
\text{Concat}\left(
\hat{\mathcal{F}}_{\text{pre}}^i,\
\hat{\mathcal{F}}_{\text{post}}^i,\
\hat{\mathcal{F}}_{\text{post}}^i - \hat{\mathcal{F}}_{\text{pre}}^i
\right)\right)\right),
\]
where \(\sigma(\cdot)\) is **GELU** and the \(1\times1\) convolution reduces channels. This formulation explicitly injects both semantic content from the two temporal states and their direct difference, while keeping the fusion lightweight [2603.01161].

The decoder consumes the four fused feature maps \(\hat{\mathcal{F}}_{\text{fuse}}^i\), concatenates them along channels, applies a \(1\times1\) convolution, upsamples twice using transposed convolutions, refines twice with residual blocks, and produces **2-channel prediction scores**. An **argmax over the channel dimension** yields the final binary change map. The decoder is described as preserving full image resolution [2603.01161].

Training uses **pixel-wise cross-entropy loss**,
\[
\text{Loss} =
-\frac{1}{N} \sum_{i=1}^{N}
\left[
y_i \log(\hat{y}_i) + (1-y_i)\log(1-\hat{y}_i)
\right],
\]
where \(y_i\) is the ground-truth label, \(\hat{y}_i\) is the predicted probability, and \(N\) is the number of pixels. The paper states that cross-entropy outperforms **focal loss** and **mIoU loss** for this model [2603.01161].

The implementation is in **PyTorch** and training is performed on a **single NVIDIA RTX A4000 GPU with 16 GB memory**. Optimization uses **AdamW** with an initial learning rate of **0.0001**, a linear decay schedule with decay steps at **100 and 200 epochs**, **400 epochs** of total training, and **batch size 2**. Data augmentation consists of **random flipping**, **scaling**, **cropping**, **color jittering**, and **Gaussian blur** [2603.01161].

## 5. Datasets, evaluation protocol, and reported performance

GRAD-Former is evaluated on three CD benchmarks: **LEVIR-CD**, **DSIFN-CD**, and **CDD**. LEVIR-CD contains **637 high-resolution image pairs**, with **0.5 m/pixel** resolution and original image size **1024 × 1024**, cropped to **256 × 256**; the split is **7120 train**, **1024 val**, and **2048 test**. DSIFN-CD contains **3988 bitemporal image pairs** from **Xi’an, Chongqing, Beijing, Chengdu, Wuhan, Shenzhen**, cropped to **256 × 256**, with split **14,400 train**, **1,360 val**, and **192 test**. CDD contains **11 raw image pairs**, at resolutions from **0.03 to 1 m/pixel**, including seasonal and disaster-related changes, cropped to **256 × 256**, with split **10,000 train**, **3,000 val**, and **3,000 test** [2603.01161].

Evaluation uses **F1 score**, **IoU**, and **OA** (**overall accuracy**), with IoU and F1 computed on the **change class**. The paper reports that GRAD-Former outperforms the compared methods on all three datasets [2603.01161].

| Dataset | Reported performance | Comparison statement |
|---|---|---|
| CDD | **F1 = 97.57%**, **IoU = 95.26%**, **OA = 99.43%** | Best among all compared methods |
| DSIFN-CD | **F1 = 93.14%**, **IoU = 87.16%**, **OA = 97.65%** | Best among all compared methods |
| LEVIR-CD | **F1 = 91.52%**, **IoU = 84.36%**, **OA = 99.14%** | Best among all compared methods |

The comparison set includes **CNN-based** methods such as FC-EF, FC-Siam-diff, FC-Siam-conc, DTCDSCN, SNUNet, ISNet, and SARAS-Net; **Transformer-based** methods such as BIT, ChangeFormer, ScratchFormer, ELGC-Net, Meta-SGNet, RAHFF-Net, AMDANet, CICD, and AdvCP; and **Mamba-based** methods such as ChangeMamba, RS-Mamba, CDMamba, and MF-VMamba. The paper highlights three margins in particular: on **CDD**, GRAD-Former surpasses **ScratchFormer** by **+1.45 F1** and **+2.72 IoU**; on **DSIFN-CD**, it outperforms **ChangeMamba** by **+2.93 F1** and **about +5.0 IoU**; and on **LEVIR-CD**, it exceeds **CICD** by **+0.49 IoU** and **+0.10 OA** [2603.01161].

The paper also includes a parameter and complexity comparison on DSIFN-CD. **GRAD-Former** is reported with **10.90M parameters**, **129.50 GFLOPs**, and **F1 93.14**, compared with **ChangeFormer** at **41.03M parameters**, **211.15 GFLOPs**, **F1 87.34**; **ChangeMamba** at **85.53M parameters**, **179.32 GFLOPs**, **F1 90.21**; and **MF-VMamba** at **57.84M parameters**, **94.06 GFLOPs**, **F1 88.05**. The authors use these results to argue that GRAD-Former achieves strong accuracy with a **moderate model size** [2603.01161].

## 6. Ablations, qualitative behavior, and limitations

The ablation studies attribute measurable gains to each major module. On **LEVIR-CD**, using **SEA only** yields **IoU 79.69** and **F1 87.93**; **GLFR only** yields **IoU 82.92** and **F1 90.63**; **DA only** yields **IoU 80.30** and **F1 88.90**; and the full combination **SEA + GLFR + DA** yields **IoU 84.36**, **F1 91.52**, **OA 99.14**, and **10.90M parameters**. The paper states that these results confirm that all components contribute positively and that the full model is best [2603.01161].

A second ablation compares attention mechanisms inside GLFR on LEVIR-CD. **Simple Attention** gives **F1 84.32** and **IoU 72.89**; **Pooled-Transpose Attention** gives **F1 91.08** and **IoU 83.62**; and **Differential Attention** gives **F1 91.52** and **IoU 84.36**. The paper concludes that differential attention is best in both accuracy and efficiency. A third study reports that **Cross-Entropy Loss** performs better than **Focal Loss** and that **mIoU Loss** is **much worse and unstable** [2603.01161].

The qualitative findings reported by the authors are that GRAD-Former reduces false positives and false negatives, produces sharper and cleaner boundaries, detects small changes more reliably, ignores seasonal and lighting-induced pseudo-changes, and preserves semantic consistency across objects. The method is described as performing particularly well on **small objects like vehicles**, **narrow structures such as streams**, and **mixed scenes with both large and tiny change regions** [2603.01161].

The paper does not discuss major explicit limitations, but it does indicate future directions including **further improving efficiency**, adaptation for **real-time change detection**, and deployment on **edge devices**. A plausible implication is that, although GRAD-Former is more efficient than many transformer-based baselines, it remains a model with a transformer-style attention component and is **not extremely lightweight for edge deployment yet**. That implication aligns with the paper’s own emphasis on reduced model size rather than minimum possible deployment cost [2603.01161].

Source: https://www.emergentmind.com/topics/grad-former