---
title: 'SnSRIC: Adaptive Remote Sensing Captioning'
url: https://www.emergentmind.com/topics/snsric
type: topic
---

# SnSRIC: Adaptive Remote Sensing Captioning

Searching arXiv for the specified paper and closely related remote sensing image captioning work.
SnSRIC, identified in the cited source as SD-RSIC, is a remote sensing image captioning framework that addresses redundancy in multi-caption training data by combining standard visual-language caption generation with caption summarization and adaptive fusion [2006.08432]. In remote sensing image captioning, the objective is to generate a descriptive sentence $y$ for an image $x$ that faithfully reflects high-level scene semantics in very high-resolution remote sensing imagery. The central motivation for SnSRIC is that each remote sensing image is typically annotated with multiple captions, many of which are repetitive or semantically near-duplicates; this redundancy creates information deficiency during learning and increases the risk that a model memorizes prevalent $n$-grams and sentence forms rather than generalizing to unseen images [2006.08432]. SnSRIC addresses this by using three steps: standard CNN+LSTM captioning, summarization of all ground-truth captions per image into one condensed caption via seq2seq pointer-generator networks, and adaptive combination of standard and summarized captions through image-conditioned time-varying weights learned by an LSTM [2006.08432].

## 1. Conceptual basis and motivation

SnSRIC is designed for remote sensing image captioning under a supervision regime in which each training image has multiple human-written captions [2006.08432]. The framework assumes that these captions often differ only through small lexical variations or alternative word orders while conveying nearly identical scene semantics. According to the source, such redundancy harms learning in two specific ways: it yields information deficiency because repeated semantics provide little new signal for mapping visual concepts to language, and it increases over-fitting risk because the model can memorize dominant sentence patterns instead of learning transferable image-language correspondences [2006.08432].

The method therefore reformulates caption learning as a joint problem of visual caption generation and textual condensation. Rather than treating all references as equally informative, SnSRIC summarizes the full caption set for each image into a single condensed caption and then blends this summarized signal with the standard caption distribution. This suggests a hybrid supervision strategy in which the original captions preserve direct image-grounded expression, while the summarized caption functions as a redundancy-reduced semantic prior.

At the probabilistic level, caption generation follows the standard autoregressive factorization
$$
p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).
$$
Training over image-caption pairs maximizes the log-likelihood
$$
\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),
$$
which corresponds to Eq. (1) in the paper [2006.08432].

## 2. Three-stage architecture

SnSRIC consists of three main steps [2006.08432]. First, it performs standard caption generation with a CNN encoder and an LSTM decoder. Second, it summarizes all ground-truth captions of each training image into a single caption using a seq2seq pointer-generator model. Third, it integrates the standard and summarized caption distributions through adaptive time-varying weights produced by a dedicated LSTM [2006.08432].

In the first step, image features are extracted as $v=\phi(x)$ by a pre-trained CNN $\phi$ using ImageNet pretraining and no fine-tuning. These features are projected by a fully connected layer to an image embedding $e \in \mathbb{R}^W$ so that the image representation matches the word embedding space. An LSTM decoder then generates tokens sequentially, conditioned on the image embedding and previous words, producing $p_{\mathrm{std}}(y_t \mid y_{1:t-1}, x)$ [2006.08432].

In the second step, all $K$ ground-truth captions $\{y^{(k)}\}_{k=1..K}$ of an image are stacked into a single document and passed to a pre-trained pointer-generator summarizer $\psi$. This produces a summarized caption $y_{\mathrm{sum}}$ with token probabilities $p_{\mathrm{sum}}(y_t \mid \{y^{(k)}\})$ [2006.08432]. The pointer-generator architecture mixes generating from a fixed vocabulary with copying source tokens, allowing faithful condensation of the training captions while also supporting novel words learned from a large external text corpus.

The third step is the defining integration mechanism of SnSRIC. A separate LSTM with scalar hidden size repeatedly consumes the image embedding $e$ and outputs a gate $\alpha_t \in (0,1)$ through a sigmoid transformation. The final token distribution is a convex combination of the standard caption distribution and the summarized caption distribution:
$$
p(y_t \mid x) = \alpha_t \cdot p_{\mathrm{std}}(y_t \mid x) + (1-\alpha_t)\cdot p_{\mathrm{sum}}(y_t \mid \{y^{(k)}\}).
$$
In the notation of the paper,
$$
P_t(V) = \alpha_t \cdot P_t^c(V \mid y_{1:t-1}, x) + (1-\alpha_t)\cdot P_t^s(V \mid C_i).
$$
This adaptive weighting is learned end-to-end and is explicitly image-adaptive [2006.08432].

## 3. Mathematical formulation

The standard LSTM decoder in the first stage uses word embeddings $u_t = Ew_t$, with $E \in \mathbb{R}^{W\times |V|}$, previous hidden state $h_{t-1}$, previous cell state $c_{t-1}$, and initial image input $e$ at $t=0$. Its recurrence is
$$
i_t = \sigma(W_i u_t + U_i h_{t-1} + b_i)
$$
$$
f_t = \sigma(W_f u_t + U_f h_{t-1} + b_f)
$$
$$
o_t = \sigma(W_o u_t + U_o h_{t-1} + b_o)
$$
$$
\tilde{c}_t = \tanh(W_c u_t + U_c h_{t-1} + b_c)
$$
$$
c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t
$$
$$
h_t = o_t \odot \tanh(c_t).
$$
The corresponding vocabulary distribution is
$$
P_t^c(V \mid y_{1:t-1}, x; \theta) = \mathrm{softmax}(W_p h_t + b_p)
$$
[2006.08432].

The summarization component is a seq2seq pointer-generator trained on Annotated Gigaword. The encoder produces states $h_i$, the decoder state is $s_t$, and attention is computed as
$$
e_{t,i} = v_a^T \tanh(W_a s_t + U_a h_i), \quad a_t = \mathrm{softmax}(e_t), \quad c_t = \sum_i a_{t,i} h_i.
$$
The pointer-generator switch is
$$
p_{\mathrm{gen}} = \sigma(w_c^T c_t + w_s^T s_t + w_x^T x_t + b_g),
$$
and the output distribution is
$$
P_t^s(y) = p_{\mathrm{gen}} \cdot P_{\mathrm{vocab}}(y \mid s_t, c_t) + (1-p_{\mathrm{gen}})\cdot \sum_{i:x_i=y} a_{t,i}.
$$
Its training objective is the standard cross-entropy loss
$$
L_{\mathrm{sum}} = - \sum_t \log P_t^s(y_t^{\mathrm{sum}})
$$
[2006.08432].

The adaptive weighting LSTM produces a scalar hidden score $s_t^a$, which is transformed into
$$
\alpha_t = \sigma(s_t^a) \in (0,1).
$$
If the two contributing distributions have different lengths, zero-padding is applied to align time steps before mixing [2006.08432]. The captioning loss for the full SnSRIC model is
$$
L_{\mathrm{cap}} = - \sum_{\text{images }i} \sum_{\text{captions }j\in C_i} \sum_{t=1}^{L_{i,j}} \log P_t(w_t^{(i,j)}).
$$
The summarizer $\psi$ is pre-trained separately, and its outputs are used during SnSRIC training [2006.08432].

## 4. Data, implementation, and evaluation protocol

SnSRIC was evaluated on three benchmark datasets: Sydney-Captions, UCM-Captions, and RSICD [2006.08432]. Sydney-Captions contains 613 images of size $500\times500$ pixels at 0.5 m resolution across 7 classes, with 5 captions per image for a total of 3065 captions. UCM-Captions contains 2100 aerial images of size $256\times256$ pixels at approximately 1 ft resolution across 21 land-use classes, with 5 captions per image for a total of 10500 captions. RSICD contains 10921 images of size $224\times224$ pixels with varying resolutions; captions per image originally vary between 1 and 5 and were augmented to 5 by duplication, resulting in 54605 captions [2006.08432]. The train, validation, and test splits follow an 80% / 10% / 10% partition.

The summarization model is pre-trained on Annotated Gigaword, with approximately 4 million article-headline pairs after preprocessing following Rush et al. 2015 and an initial vocabulary of approximately 110k words [2006.08432]. SnSRIC uses a shared vocabulary of 50,000 words that covers all words from Sydney-Captions, UCM-Captions, and RSICD together with the most frequent Annotated Gigaword words. This common vocabulary is used across all stages to avoid out-of-vocabulary problems when the summarizer contributes tokens [2006.08432].

The evaluated CNN backbones are VGG16/19, GoogleNet, InceptionV3, ResNet34/50/101/152, and DenseNet121/169/201, all ImageNet-pretrained and not fine-tuned [2006.08432]. The best backbone differs by dataset: ResNet101 for Sydney-Captions, VGG16 for UCM-Captions, and ResNet152 for RSICD. The image embedding size $W$ was tested in $\{128,256,512,1024\}$, with $W=512$ selected as the best tradeoff [2006.08432]. The step-1 LSTM hidden size equals $W$, whereas the step-3 weighting LSTM has hidden size 1, reflecting its role as a scalar gate generator.

Optimization uses stochastic gradient descent with initial learning rate $10^{-3}$; the learning rate decays by 20% after 8 consecutive epochs without validation improvement [2006.08432]. Beam search with beam size 4 is used at inference. Training uses five captions per image and replicates images to form image-caption pairs. The implementation was trained on NVIDIA Tesla V100 GPUs [2006.08432].

Evaluation reports BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, ROUGE-L, and CIDEr [2006.08432]. BLEU-$n$ is defined as
$$
\mathrm{BLEU}\text{-}n = BP \cdot \exp\left(\sum_{m=1}^{n} w_m^B \cdot \log P_m^B\right),
$$
with brevity penalty
$$
BP = 1 \text{ if } l_c > l_r; \text{ else } BP = \exp(1-l_r/l_c).
$$
CIDEr is defined by
$$
\mathrm{CIDEr}_n = (1/m)\sum_j \frac{g^n(c^*) \cdot g^n(c_j)}{||g^n(c^*)||\cdot ||g^n(c_j)||},
$$
and
$$
\mathrm{CIDEr} = \sum_{n=1}^{N} w_n^B \cdot \mathrm{CIDEr}_n
$$
[2006.08432].

## 5. Empirical performance and ablations

SnSRIC outperforms the compared baselines on Sydney-Captions, UCM-Captions, and RSICD [2006.08432]. The baselines include BoW+CNN, DeViSE+CNN, CSMLF, and NIC. The paper characterizes these alternatives as follows: BoW+CNN relies on cosine distance between bag-of-words caption vectors and CNN features; DeViSE+CNN uses a joint visual-semantic embedding; CSMLF maps GloVe caption embeddings and CNN features to a common metric space; NIC is the standard CNN+LSTM encoder-decoder and does not address redundancy or vocabulary enrichment [2006.08432].

| Dataset | SnSRIC results | Reported comparison |
|---|---|---|
| Sydney-Captions | BLEU-4 0.517, METEOR 0.366, ROUGE-L 0.657, CIDEr 1.690 | vs NIC: +21.6% BLEU-4, +14.4% METEOR, +8.4% ROUGE-L, +32.4% CIDEr |
| UCM-Captions | BLEU-4 0.538, METEOR 0.390, ROUGE-L 0.695, CIDEr 2.132 | vs NIC: +4.1% BLEU-4, +4.3% METEOR, +3.3% ROUGE-L, +6.3% CIDEr |
| RSICD | BLEU-4 0.300, METEOR 0.249, ROUGE-L 0.523, CIDEr 0.794 | vs NIC: +3.1% BLEU-4, +2.5% METEOR, +1.6% ROUGE-L, +4.5% CIDEr |

On Sydney-Captions, SnSRIC achieves BLEU-1 0.761, BLEU-2 0.666, BLEU-3 0.586, BLEU-4 0.517, METEOR 0.366, ROUGE-L 0.657, and CIDEr 1.690 [2006.08432]. The NIC baseline reaches BLEU-1 0.707, BLEU-4 0.425, METEOR 0.320, ROUGE-L 0.606, and CIDEr 1.277, while BoW+CNN, DeViSE+CNN, and CSMLF are weaker still [2006.08432]. On UCM-Captions, SnSRIC reaches BLEU-1 0.748, BLEU-2 0.664, BLEU-3 0.598, BLEU-4 0.538, METEOR 0.390, ROUGE-L 0.695, and CIDEr 2.132 [2006.08432]. On RSICD, it achieves BLEU-1 0.644, BLEU-2 0.474, BLEU-3 0.369, BLEU-4 0.300, METEOR 0.249, ROUGE-L 0.523, and CIDEr 0.794 [2006.08432].

A central ablation compares the full model with “Step 1 (Single Caption).” The full model improves BLEU-4 from 0.399 to 0.517 and CIDEr from 1.099 to 1.690 on Sydney; from 0.460 to 0.538 and 1.774 to 2.132 on UCM; and from 0.285 to 0.300 and 0.740 to 0.794 on RSICD [2006.08432]. The isolated contributions of summarization alone and adaptive weighting alone were not separately reported. Statistical significance was also not reported [2006.08432].

Sensitivity analysis shows that increasing the embedding size improves performance up to $W=512$, while increasing to $W=1024$ yields limited gains or small regressions with higher complexity [2006.08432]. Performance is described as relatively robust across CNN backbones, though InceptionV3 is notably weaker on UCM-Captions. Zero-padding in the fusion stage is rare, with average application rates of 0.4245% for Sydney, 0.3248% for UCM, and 0.3385% for RSICD [2006.08432].

## 6. Qualitative behavior, complexity, and limitations

Qualitative examples indicate that SnSRIC produces more semantically complete and grammatically correct captions than NIC [2006.08432]. The source specifically notes examples such as capturing “Green trees near the bridge,” which NIC misses, and correctly describing the type of residential area when NIC fails or is ungrammatical. It also states that vocabulary enrichment through the summarizer introduces informative words absent from the remote sensing datasets, such as “matt” and “nature,” enabling finer-grained descriptions of complex scenes [2006.08432].

A potential failure mode arises when summarized and standard captions are semantically misaligned, either in length or in semantic focus [2006.08432]. The adaptive weighting sequence $\alpha_t$ is intended to mitigate this by learning which source to trust at each time step. When summarized captions end earlier than standard captions, zero-padding is applied; the rarity of this event suggests that final outputs are not dominated by standard captions solely because of length mismatch.

In computational terms, the dominant cost comes from the CNN backbone in the first stage [2006.08432]. Parameter count and FLOPs vary mainly with the selected backbone. For the ResNet101 setting used on Sydney-Captions, the paper reports approximately 43.6M–77.9M parameters and 7.83–8.63G FLOPs. For the VGG16 setting on UCM-Captions, it reports approximately 119.6M–154.0M parameters and 15.46–16.27G FLOPs. For the ResNet152 setting on RSICD, it reports approximately 59.2M–93.6M parameters and 11.55–12.35G FLOPs [2006.08432]. Including the adaptive weighting stage increases parameters by less than 1%, and the summarization stage raises FLOPs only modestly. This suggests that the method’s additional modeling capacity is inexpensive relative to the visual encoder.

The principal limitations named in the source concern domain mismatch, attention, and evaluation [2006.08432]. The summarizer is trained on news headline data from Annotated Gigaword, so its domain gap relative to remote sensing caption text may limit the quality of condensation for remote-sensing-specific semantics. The first stage does not use explicit visual attention, though the paper notes that attention mechanisms such as those in “Show, Attend and Tell” could be integrated into future versions. The reported metrics emphasize lexical overlap rather than semantic equivalence, and the authors explicitly suggest that semantics-aware metrics would be desirable [2006.08432].

Future directions mentioned in the source include using different summarization models, such as pre-trained transformer summarizers, alternative summarization datasets such as DUC 2004, and attention-enhanced SnSRIC variants [2006.08432]. A plausible implication is that the framework is modular: the summarization component, the image encoder, and the adaptive integration mechanism can each be upgraded independently while preserving the overall redundancy-reduction principle.

Source: https://www.emergentmind.com/topics/snsric