DeltaVLM: Interactive RS Change Analysis

• DeltaVLM is an integrated architecture for interactive remote sensing image change analysis that merges change detection, captioning, and visual Q&A into a unified workflow.
• It leverages a selectively fine-tuned vision backbone, a semantic difference filtering module (CSRM), and an instruction-guided Q-former to generate detailed, context-aware change responses.
• Empirical results show state-of-the-art performance in change captioning, classification, quantification, and localization tasks on the large-scale ChangeChat-105k dataset.

DeltaVLM is an end-to-end architecture designed for interactive remote sensing image change analysis (RSICA), a paradigm that unifies classical change detection and captioning with visual question answering to enable instruction-guided, multi-turn exploration of changes in bi-temporal satellite imagery. DeltaVLM conditions on user instructions and generates text responses about changes between co-registered satellite images, supporting a wide range of query types. The system integrates a selectively fine-tuned vision backbone, a difference perception and semantic filtering module, a cross-modal Q-former for instruction alignment, and a frozen LLM (Vicuna-7B) as the decoder. The approach achieves state-of-the-art results on tasks spanning structured change description, classification, quantification, localization, open-ended QA, and multi-turn dialogue (Deng et al., 30 Jul 2025).

1. Remote Sensing Image Change Analysis (RSICA): Paradigm and Tasks

RSICA is formulated as a multimodal task in which a model processes a pair of co-registered satellite images $I_{t_1}$ and $I_{t_2}$, responding interactively to natural-language user queries regarding the changes between the two timepoints. This paradigm enables flexible, instruction-driven analysis, with the model adapting its output modality (e.g., caption, count, location, free-form text) to the query.

DeltaVLM is trained on six major interaction types:

1. Change Captioning: Free-form natural-language description of observed changes.
2. Binary Change Classification: Yes/no decision on whether any change occurred.
3. Category-Specific Quantification: Object count differences (e.g., number of new buildings).
4. Change Localization: Identifying which cells in a $3\times3$ spatial grid contain changes.
5. Open-Ended QA: Answers to arbitrary, high-level questions about the scene.
6. Multi-Turn Dialogue: Sequences of interdependent queries and answers, supporting multi-step reasoning and exploration.

This design facilitates analysis workflows not possible in one-shot change detection or static captioning frameworks.

2. The ChangeChat-105k Dataset

ChangeChat-105k is a large-scale instruction-following dataset constructed to comprehensively train and evaluate interactive RSICA. The dataset is based on core imagery from LEVIR-CC (10,077 bi-temporal image pairs with captions) and LEVIR-MCI (paired change masks). Its diversity of tasks is ensured by a two-stage generation process:

• Rule-Based Extraction: For structured tasks (interaction types 1–4), extraction leverages pixel-level change masks, OpenCV for object contours, and spatial tiling for localization.
• GPT-Assisted Generation: For open-ended QA and multi-turn dialogue, prompts and seed examples direct ChatGPT to create diverse question–answer pairs, with integration of contour/count data.

The dataset's task and subset breakdown is:

Type	Training Pairs	Test Pairs
Change captioning	34,075	1,929
Binary classification	6,815	1,929
Category quantification	6,815	1,929
Localization	6,815	1,929
Open-ended QA	26,600	7,527
Multi-turn conversation	6,815	1,929

Overall, the dataset contains 87,935 training and 17,172 test instruction–response pairs (total 105,107). Each pair provides $(I_{t_1}, I_{t_2}, P, T)$ where $P$ is the instruction and $T$ the output text (Deng et al., 30 Jul 2025).

3. DeltaVLM Architecture

DeltaVLM embodies a modular hybrid of vision, difference perception, cross-modal alignment, and language-generation components:

a. Bi-Temporal Vision Encoder (Bi-VE)

The vision backbone is EVA-ViT-g/14. Only the final two transformer blocks are fine-tuned for RSICA; the first 37 are frozen, preserving generic vision priors. Each input image, $I_{t_i} \in \mathbb{R}^{H \times W \times 3}$, is split into $16 \times 16$ patches before independent processing:

$$F_{t_1} = \Phi_{\mathrm{ViT}}(I_{t_1}; \Theta_{\mathrm{fine-tuned}}) \in \mathbb{R}^{N\times D}$$

$$F_{t_2} = \Phi_{\mathrm{ViT}}(I_{t_2}; \Theta_{\mathrm{fine-tuned}})$$

with $N = (H/16)(W/16)$, $D$ the hidden dim. A pixelwise difference is computed:

$$F_{\mathrm{diff}} = F_{t_2} - F_{t_1} \in \mathbb{R}^{N \times D}$$

b. Visual Difference Perception Module with Cross-Semantic Relation Measuring (CSRM)

Direct decoding of $F_{\mathrm{diff}}$ is confounded by remote sensing-specific artifacts. CSRM refines differences by:

1. Contextualizing:

$$C_{t_1} = \tanh(W_c [F_{\mathrm{diff}} ; F_{t_1}] + b_c)$$

$$C_{t_2} = \tanh(W'_c [F_{\mathrm{diff}} ; F_{t_2}] + b'_c)$$

2. Gating:

$$G_{t_1} = \sigma(W_g [F_{\mathrm{diff}} ; F_{t_1}] + b_g)$$

$$G_{t_2} = \sigma(W'_g [F_{\mathrm{diff}} ; F_{t_2}] + b'_g)$$

3. Filtering:

$$F'_{t_1} = G_{t_1} \odot C_{t_1}, \qquad F'_{t_2} = G_{t_2} \odot C_{t_2}$$

Filtered difference features $F'_{t_1}, F'_{t_2}$ selectively emphasize semantically relevant changes.

c. Instruction-Guided Q-Former

DeltaVLM utilizes $L=32$ learnable queries $Q \in \mathbb{R}^{L \times d}$ ($d$ is LLM’s embedding size). Processing proceeds as:

1. Self-attention:

$Q_{\mathrm{SA}} = \mathrm{SelfAttention}(Q)$

2. Cross-attention:

$$Q_{\mathrm{CA}} = \mathrm{CrossAttention}(Q_{\mathrm{SA}}, [F'_{t_1}; F'_{t_2}], P)$$

where $P$ is the tokenized instruction.

3. Feed-forward bottleneck:

$$\hat F_{\mathrm{diff}} = \mathrm{FFN}(Q_{\mathrm{CA}}) \in \mathbb{R}^{L\times d}$$

The resulting $\hat F_{\mathrm{diff}}$ is a compact, instruction-aware representation for downstream decoding.

d. Decoder

The concatenated features are input to a frozen Vicuna-7B LLM, which autoregressively generates token sequences as the output.

4. Training Objective and Optimization

Training optimizes only the final two Bi-VE blocks, the full Difference Perception module (including all CSRM weights), and all Q-former layers. The Vicuna-7B decoder and 37 vision backbone layers are held fixed. The training objective for each example $(I_{t_1}, I_{t_2}, P, T)$ is standard cross-entropy loss over the label sequence:

$$\mathcal{L}_{\mathrm{CE}} = - \frac{1}{K} \sum_{i=1}^{K} w_i \cdot \log(\hat w_i)$$

where $w_i$ is the one-hot vector for ground-truth token at position $i$; $K$ is output length. No auxiliary losses are introduced.

5. Empirical Results and Ablation Studies

DeltaVLM demonstrates leading performance across all RSICA sub-tasks relative to domain-specific and general vision-language baselines:

Task	DeltaVLM Metric(s)	Best General VLM/RS Baseline
Change Captioning	BLEU-1=85.78, BLEU-4=62.51, CIDEr=136.72	BLEU-4≈62.87 (SFT), lower on others
Binary Classification	Acc=93.99%, Prec=96.29%, Rec=91.49%, F1=93.83%	F1=85.07% (GPT-4o)
Category Quantification	Roads: MAE=0.24, RMSE=0.70; Buildings: MAE=1.32, RMSE=2.89	Roads MAE=0.49, Buildings MAE=1.86 (GPT-4o)
Localization	Buildings F1=78.99%, Roads F1=67.94%	Up to 26pp F1 lower
Open-Ended QA	BLEU-4=16.21, CIDEr=127.38	BLEU-4=9.68, CIDEr=72.58 (GPT-4o)

Ablation studies reveal that removing CSRM leads to substantial degradation (caption BLEU-1 from 85.78→64.42, CIDEr from 136.72→101.92, classification F1=0.62%), confirming the essential role of semantic filtering. Omitting Bi-VE fine-tuning reduces both captioning and classification scores, but to a lesser extent, indicating that selective adaptation of vision features aids generalization to RSICA without catastrophic loss of prior knowledge (Deng et al., 30 Jul 2025).

6. Contributions, Limitations, and Prospects

DeltaVLM’s principal contributions include:

• Formalization of RSICA as an interactive, language-conditioned paradigm for bi-temporal remote sensing analysis.
• Release of the large-scale, multi-interaction ChangeChat-105k dataset to promote advances in multimodal, dialogue-based geospatial analysis.
• Development of an architecture that (1) selectively fine-tunes a large ViT for specialization, (2) introduces CSRM for semantic difference filtering in remote sensing contexts, and (3) aligns visual and linguistic modalities by means of an instruction-guided Q-former bottleneck and a frozen LLM decoder.

While DeltaVLM consistently outperforms both specialized remote sensing models and general-purpose vision-LLMs, several limitations remain. Outputs are currently textual only; the integration of dense change segmentation or bounding-box prediction into hybrid outputs is an open research area. Further efficiency gains may be possible via LLM distillation. Extending to deeper chain-of-thought dialogues and leveraging self-supervised pretraining for unlabeled multi-temporal RS images are identified as promising directions (Deng et al., 30 Jul 2025).