SnSRIC: Adaptive Remote Sensing Captioning

• SnSRIC is a remote sensing image captioning framework that reduces redundancy by condensing multiple similar captions into a single, semantically rich descriptor.
• It integrates CNN+LSTM based standard caption generation with a seq2seq pointer-generator summarizer to create a more informative caption from diverse annotations.
• Adaptive weighting via an LSTM dynamically fuses standard and summarized caption distributions, mitigating overfitting and improving scene understanding.

Searching arXiv for the specified paper and closely related remote sensing image captioning work. SnSRIC, identified in the cited source as SD-RSIC, is a remote sensing image captioning framework that addresses redundancy in multi-caption training data by combining standard visual-language caption generation with caption summarization and adaptive fusion (Sumbul et al., 2020). In remote sensing image captioning, the objective is to generate a descriptive sentence $y$ for an image $x$ that faithfully reflects high-level scene semantics in very high-resolution remote sensing imagery. The central motivation for SnSRIC is that each remote sensing image is typically annotated with multiple captions, many of which are repetitive or semantically near-duplicates; this redundancy creates information deficiency during learning and increases the risk that a model memorizes prevalent $n$-grams and sentence forms rather than generalizing to unseen images (Sumbul et al., 2020). SnSRIC addresses this by using three steps: standard CNN+LSTM captioning, summarization of all ground-truth captions per image into one condensed caption via seq2seq pointer-generator networks, and adaptive combination of standard and summarized captions through image-conditioned time-varying weights learned by an LSTM (Sumbul et al., 2020).

1. Conceptual basis and motivation

SnSRIC is designed for remote sensing image captioning under a supervision regime in which each training image has multiple human-written captions (Sumbul et al., 2020). The framework assumes that these captions often differ only through small lexical variations or alternative word orders while conveying nearly identical scene semantics. According to the source, such redundancy harms learning in two specific ways: it yields information deficiency because repeated semantics provide little new signal for mapping visual concepts to language, and it increases over-fitting risk because the model can memorize dominant sentence patterns instead of learning transferable image-language correspondences (Sumbul et al., 2020).

The method therefore reformulates caption learning as a joint problem of visual caption generation and textual condensation. Rather than treating all references as equally informative, SnSRIC summarizes the full caption set for each image into a single condensed caption and then blends this summarized signal with the standard caption distribution. This suggests a hybrid supervision strategy in which the original captions preserve direct image-grounded expression, while the summarized caption functions as a redundancy-reduced semantic prior.

At the probabilistic level, caption generation follows the standard autoregressive factorization

$$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$

Training over image-caption pairs maximizes the log-likelihood

$$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$

which corresponds to Eq. (1) in the paper (Sumbul et al., 2020).

2. Three-stage architecture

SnSRIC consists of three main steps (Sumbul et al., 2020). First, it performs standard caption generation with a CNN encoder and an LSTM decoder. Second, it summarizes all ground-truth captions of each training image into a single caption using a seq2seq pointer-generator model. Third, it integrates the standard and summarized caption distributions through adaptive time-varying weights produced by a dedicated LSTM (Sumbul et al., 2020).

In the first step, image features are extracted as $v=\phi(x)$ by a pre-trained CNN $\phi$ using ImageNet pretraining and no fine-tuning. These features are projected by a fully connected layer to an image embedding $e \in \mathbb{R}^W$ so that the image representation matches the word embedding space. An LSTM decoder then generates tokens sequentially, conditioned on the image embedding and previous words, producing $p_{\mathrm{std}}(y_t \mid y_{1:t-1}, x)$ (Sumbul et al., 2020).

In the second step, all $K$ ground-truth captions $x$0 of an image are stacked into a single document and passed to a pre-trained pointer-generator summarizer $x$1. This produces a summarized caption $x$2 with token probabilities $x$3 (Sumbul et al., 2020). The pointer-generator architecture mixes generating from a fixed vocabulary with copying source tokens, allowing faithful condensation of the training captions while also supporting novel words learned from a large external text corpus.

The third step is the defining integration mechanism of SnSRIC. A separate LSTM with scalar hidden size repeatedly consumes the image embedding $x$4 and outputs a gate $x$5 through a sigmoid transformation. The final token distribution is a convex combination of the standard caption distribution and the summarized caption distribution:

$x$6

In the notation of the paper,

$x$7

This adaptive weighting is learned end-to-end and is explicitly image-adaptive (Sumbul et al., 2020).

3. Mathematical formulation

The standard LSTM decoder in the first stage uses word embeddings $x$8, with $x$9, previous hidden state $n$0, previous cell state $n$1, and initial image input $n$2 at $n$3. Its recurrence is

$n$4

$n$5

$n$6

$n$7

$n$8

$n$9

The corresponding vocabulary distribution is

$$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$0

(Sumbul et al., 2020).

The summarization component is a seq2seq pointer-generator trained on Annotated Gigaword. The encoder produces states $$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$1, the decoder state is $$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$2, and attention is computed as

$$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$3

The pointer-generator switch is

$$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$4

and the output distribution is

$$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$5

Its training objective is the standard cross-entropy loss

$$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$6

(Sumbul et al., 2020).

The adaptive weighting LSTM produces a scalar hidden score $$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$7, which is transformed into

$$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$8

If the two contributing distributions have different lengths, zero-padding is applied to align time steps before mixing (Sumbul et al., 2020). The captioning loss for the full SnSRIC model is

$$p(y \mid x) = \prod_{t=1}^{T} p(y_t \mid y_{1:t-1}, x).$$9

The summarizer $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$0 is pre-trained separately, and its outputs are used during SnSRIC training (Sumbul et al., 2020).

4. Data, implementation, and evaluation protocol

SnSRIC was evaluated on three benchmark datasets: Sydney-Captions, UCM-Captions, and RSICD (Sumbul et al., 2020). Sydney-Captions contains 613 images of size $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$1 pixels at 0.5 m resolution across 7 classes, with 5 captions per image for a total of 3065 captions. UCM-Captions contains 2100 aerial images of size $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$2 pixels at approximately 1 ft resolution across 21 land-use classes, with 5 captions per image for a total of 10500 captions. RSICD contains 10921 images of size $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$3 pixels with varying resolutions; captions per image originally vary between 1 and 5 and were augmented to 5 by duplication, resulting in 54605 captions (Sumbul et al., 2020). The train, validation, and test splits follow an 80% / 10% / 10% partition.

The summarization model is pre-trained on Annotated Gigaword, with approximately 4 million article-headline pairs after preprocessing following Rush et al. 2015 and an initial vocabulary of approximately 110k words (Sumbul et al., 2020). SnSRIC uses a shared vocabulary of 50,000 words that covers all words from Sydney-Captions, UCM-Captions, and RSICD together with the most frequent Annotated Gigaword words. This common vocabulary is used across all stages to avoid out-of-vocabulary problems when the summarizer contributes tokens (Sumbul et al., 2020).

The evaluated CNN backbones are VGG16/19, GoogleNet, InceptionV3, ResNet34/50/101/152, and DenseNet121/169/201, all ImageNet-pretrained and not fine-tuned (Sumbul et al., 2020). The best backbone differs by dataset: ResNet101 for Sydney-Captions, VGG16 for UCM-Captions, and ResNet152 for RSICD. The image embedding size $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$4 was tested in $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$5, with $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$6 selected as the best tradeoff (Sumbul et al., 2020). The step-1 LSTM hidden size equals $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$7, whereas the step-3 weighting LSTM has hidden size 1, reflecting its role as a scalar gate generator.

Optimization uses stochastic gradient descent with initial learning rate $$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$8; the learning rate decays by 20% after 8 consecutive epochs without validation improvement (Sumbul et al., 2020). Beam search with beam size 4 is used at inference. Training uses five captions per image and replicates images to form image-caption pairs. The implementation was trained on NVIDIA Tesla V100 GPUs (Sumbul et al., 2020).

Evaluation reports BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, ROUGE-L, and CIDEr (Sumbul et al., 2020). BLEU-$$\theta^* = \arg\max_\theta \sum_{i=1}^{|T|} \sum_{j=1}^{N_i} \sum_{t=1}^{L_{i,j}} \log P(w_t \mid w_{1:t-1}, I_i; \theta),$$9 is defined as

$v=\phi(x)$0

with brevity penalty

$v=\phi(x)$1

CIDEr is defined by

$v=\phi(x)$2

and

$v=\phi(x)$3

(Sumbul et al., 2020).

5. Empirical performance and ablations

SnSRIC outperforms the compared baselines on Sydney-Captions, UCM-Captions, and RSICD (Sumbul et al., 2020). The baselines include BoW+CNN, DeViSE+CNN, CSMLF, and NIC. The paper characterizes these alternatives as follows: BoW+CNN relies on cosine distance between bag-of-words caption vectors and CNN features; DeViSE+CNN uses a joint visual-semantic embedding; CSMLF maps GloVe caption embeddings and CNN features to a common metric space; NIC is the standard CNN+LSTM encoder-decoder and does not address redundancy or vocabulary enrichment (Sumbul et al., 2020).

Dataset	SnSRIC results	Reported comparison
Sydney-Captions	BLEU-4 0.517, METEOR 0.366, ROUGE-L 0.657, CIDEr 1.690	vs NIC: +21.6% BLEU-4, +14.4% METEOR, +8.4% ROUGE-L, +32.4% CIDEr
UCM-Captions	BLEU-4 0.538, METEOR 0.390, ROUGE-L 0.695, CIDEr 2.132	vs NIC: +4.1% BLEU-4, +4.3% METEOR, +3.3% ROUGE-L, +6.3% CIDEr
RSICD	BLEU-4 0.300, METEOR 0.249, ROUGE-L 0.523, CIDEr 0.794	vs NIC: +3.1% BLEU-4, +2.5% METEOR, +1.6% ROUGE-L, +4.5% CIDEr

On Sydney-Captions, SnSRIC achieves BLEU-1 0.761, BLEU-2 0.666, BLEU-3 0.586, BLEU-4 0.517, METEOR 0.366, ROUGE-L 0.657, and CIDEr 1.690 (Sumbul et al., 2020). The NIC baseline reaches BLEU-1 0.707, BLEU-4 0.425, METEOR 0.320, ROUGE-L 0.606, and CIDEr 1.277, while BoW+CNN, DeViSE+CNN, and CSMLF are weaker still (Sumbul et al., 2020). On UCM-Captions, SnSRIC reaches BLEU-1 0.748, BLEU-2 0.664, BLEU-3 0.598, BLEU-4 0.538, METEOR 0.390, ROUGE-L 0.695, and CIDEr 2.132 (Sumbul et al., 2020). On RSICD, it achieves BLEU-1 0.644, BLEU-2 0.474, BLEU-3 0.369, BLEU-4 0.300, METEOR 0.249, ROUGE-L 0.523, and CIDEr 0.794 (Sumbul et al., 2020).

A central ablation compares the full model with “Step 1 (Single Caption).” The full model improves BLEU-4 from 0.399 to 0.517 and CIDEr from 1.099 to 1.690 on Sydney; from 0.460 to 0.538 and 1.774 to 2.132 on UCM; and from 0.285 to 0.300 and 0.740 to 0.794 on RSICD (Sumbul et al., 2020). The isolated contributions of summarization alone and adaptive weighting alone were not separately reported. Statistical significance was also not reported (Sumbul et al., 2020).

Sensitivity analysis shows that increasing the embedding size improves performance up to $v=\phi(x)$4, while increasing to $v=\phi(x)$5 yields limited gains or small regressions with higher complexity (Sumbul et al., 2020). Performance is described as relatively robust across CNN backbones, though InceptionV3 is notably weaker on UCM-Captions. Zero-padding in the fusion stage is rare, with average application rates of 0.4245% for Sydney, 0.3248% for UCM, and 0.3385% for RSICD (Sumbul et al., 2020).

6. Qualitative behavior, complexity, and limitations

Qualitative examples indicate that SnSRIC produces more semantically complete and grammatically correct captions than NIC (Sumbul et al., 2020). The source specifically notes examples such as capturing “Green trees near the bridge,” which NIC misses, and correctly describing the type of residential area when NIC fails or is ungrammatical. It also states that vocabulary enrichment through the summarizer introduces informative words absent from the remote sensing datasets, such as “matt” and “nature,” enabling finer-grained descriptions of complex scenes (Sumbul et al., 2020).

A potential failure mode arises when summarized and standard captions are semantically misaligned, either in length or in semantic focus (Sumbul et al., 2020). The adaptive weighting sequence $v=\phi(x)$6 is intended to mitigate this by learning which source to trust at each time step. When summarized captions end earlier than standard captions, zero-padding is applied; the rarity of this event suggests that final outputs are not dominated by standard captions solely because of length mismatch.

In computational terms, the dominant cost comes from the CNN backbone in the first stage (Sumbul et al., 2020). Parameter count and FLOPs vary mainly with the selected backbone. For the ResNet101 setting used on Sydney-Captions, the paper reports approximately 43.6M–77.9M parameters and 7.83–8.63G FLOPs. For the VGG16 setting on UCM-Captions, it reports approximately 119.6M–154.0M parameters and 15.46–16.27G FLOPs. For the ResNet152 setting on RSICD, it reports approximately 59.2M–93.6M parameters and 11.55–12.35G FLOPs (Sumbul et al., 2020). Including the adaptive weighting stage increases parameters by less than 1%, and the summarization stage raises FLOPs only modestly. This suggests that the method’s additional modeling capacity is inexpensive relative to the visual encoder.

The principal limitations named in the source concern domain mismatch, attention, and evaluation (Sumbul et al., 2020). The summarizer is trained on news headline data from Annotated Gigaword, so its domain gap relative to remote sensing caption text may limit the quality of condensation for remote-sensing-specific semantics. The first stage does not use explicit visual attention, though the paper notes that attention mechanisms such as those in “Show, Attend and Tell” could be integrated into future versions. The reported metrics emphasize lexical overlap rather than semantic equivalence, and the authors explicitly suggest that semantics-aware metrics would be desirable (Sumbul et al., 2020).

Future directions mentioned in the source include using different summarization models, such as pre-trained transformer summarizers, alternative summarization datasets such as DUC 2004, and attention-enhanced SnSRIC variants (Sumbul et al., 2020). A plausible implication is that the framework is modular: the summarization component, the image encoder, and the adaptive integration mechanism can each be upgraded independently while preserving the overall redundancy-reduction principle.