RSRT Benchmark for Remote Sensing Retrieval

• RSRT Benchmark is a large-scale, semantically dense dataset that uses multiple GPT-4.1 generated caption variants per image to bridge the gap between low-level features and high-level semantics.
• The benchmark supports both text-to-image and image-to-text retrieval by embedding structured captions into a shared semantic space using state-of-the-art vision-language models.
• Empirical evaluations show that the TRSLLaVA retrieval pipeline achieves superior mean recall compared to conventional zero-shot methods, highlighting its potential for fine-grained remote sensing tasks.

The RSRT Benchmark refers to the "Remote Sensing Rich Text" benchmark, a large-scale, structured, and semantically dense corpus and evaluation suite for fine-grained, zero-shot remote sensing image retrieval. RSRT departs from conventional image-centric retrieval paradigms by leveraging state-of-the-art vision-LLMs (VLMs) to generate multiple, detailed textual captions per image, recasting the retrieval problem as text-to-text matching. This approach enables training-free, semantic-level retrieval workflows and establishes new evaluation standards for bridging the semantic gap in remote sensing applications (Xiao et al., 11 Dec 2025).

1. Purpose and Conceptual Foundations

RSRT was developed to address a critical need in remote sensing information retrieval: the persistence of the semantic gap—the disconnect between low-level visual features and high-level human-understandable semantics. Traditional cross-modal image retrieval systems require extensive domain-specific training and often collapse each image into a single embedding vector, limiting semantic expressivity. RSRT, in contrast, formulates both text-to-image (T2I) and image-to-text (I2T) retrieval as matching among richly structured text descriptions. Each image is represented not by a single vector, but by multiple diverse captions, increasing the granularity and verifiability of retrieval and evaluation in a fully zero-shot setting.

2. Dataset Construction and Annotation Pipeline

2.1 Image Source and Corpus Composition

The RSRT corpus consists of 17,764 images sourced from two established benchmarks: RSITMD and RSICD, spanning a broad range of geographic settings, resolutions from 0.3 m to 5 m per pixel, and 8-bit RGB encodings in JPEG and TIFF formats. The image set encompasses diverse land cover types, including urban, agricultural, and aquatic regions, with annotations ensuring global representativity.

2.2 Structured Caption Generation

For each image $I_i$, five structured caption variants are generated using GPT-4.1 under a rigorous prompting protocol:

• One-sentence summary
• Bullet list detailing directional/relational object interactions
• Three distinct paragraphs, each emphasizing land use, man-made structures, and natural features, respectively

The formal mapping is

$$\mathcal D_i = \{ D_{i,1}, \dots, D_{i,5} \} = f_{\mathrm{GPT4.1}}(I_i, P_{\mathrm{structured}})$$

with automated post-processing and human spot-audits to validate factual consistency, de-duplication, and format conformance. The final RSRT corpus consists of 88,820 distinct caption sets.

Semantic Statistics

Statistic	Value
Total Images	17,764
Caption Sets	88,820
Avg. Entities	4.58 per image
Avg. Relations	10.16 per image
Vocab Size	5,829 unique tokens
Avg. Caption Words	42.99

Stratified splits derived from the RSITMD and RSICD class taxonomy maintain semantic balance over train, validation, and test sets.

3. Retrieval Tasks and Evaluation Protocols

3.1 Retrieval Formulation

RSRT supports:

• Text-to-Image (T2I): Given a textual query $T_q$, retrieve relevant images by textual similarity to their captions.
• Image-to-Text (I2T): Given an image query $I_q$, generate a text description via visual-LLM, then retrieve matching textual entries.

Both retrieval tasks are performed exclusively in the text domain, with all queries and captions embedded into a shared $d$-dimensional space using a frozen encoder $f_{\mathrm{encoder}}$. For each candidate image, the maximum cosine similarity among its five captions is taken as the relevance score: $$S(T_q, I_c) = \max_{v \in \{1,\ldots,5\}} \text{sim}(f_\mathrm{encoder}(T_q), f_\mathrm{encoder}(D_{c,v}))$$ with

$$\text{sim}(\mathbf a, \mathbf b) = \frac{\mathbf a \cdot \mathbf b}{\|\mathbf a\| \| \mathbf b \|}$$

3.2 Evaluation Metrics

Official evaluation is based on:

• Recall@K: Proportion of relevant items within the top K results.
• Mean Recall (mR): Mean of Recall@K over K ∈ {1, 5, 10} for both T2I and I2T directions.
• (Optional) Mean Average Precision (mAP): Area under the precision-recall curve, averaged across all queries.

$$R@K = \frac{1}{N}\sum_{i=1}^N \mathbb I\left\{ \text{relevant items for query } i \text{ in top } K \right\}$$

4. TRSLLaVA: Zero-Shot Text-to-Text Retrieval Baseline

TRSLLaVA is the reference training-free retrieval pipeline implemented over RSRT. It operates as follows:

• Offline: All caption variants are embedded using $f_{\mathrm{encoder}}$.
• Query-time: Both text or image queries are mapped to text, embedded, and scored against the database using nearest neighbor search (e.g., FAISS with IVF + PQ).
• Ranking: For each image, the highest scoring caption is used for retrieval ranking.

No supervised training or domain adaptation is needed; all retrieval and evaluation are performed in a frozen semantic embedding space.

5. Empirical Results and Benchmark Comparisons

The performance of TRSLLaVA is evaluated on both RSITMD (55 classes) and RSICD (35 classes) benchmarks.

RSITMD Results

Method	Zero-Shot	R@1 (I2T/T2I)	R@5	R@10	mR
CLIP-Baseline	✔	8.8/9.5	27.9/21.0	43.2/32.7	23.86
Multilanguage	✔	17.6/19.7	49.7/40.3	66.6/54.4	41.38
TRSLLaVA	✔	18.3/19.5	43.3/44.9	68.4/61.3	42.62

RSICD Results

Method	Zero-Shot	R@1	R@5	R@10	mR
CLIP-Baseline	✔	5.8/5.3	17.7/14.2	27.2/23.7	15.64
MTGFE	✔	8.7/15.3	27.6/37.1	43.9/51.6	30.68
TRSLLaVA	✔	9.4/15.2	28.4/38.4	44.0/52.7	31.33

TRSLLaVA achieves mean Recall scores that are not only superior to the conventional zero-shot CLIP baseline but also slightly surpass the best fully supervised methods. Particularly, improvements are pronounced in the higher K (R@5, R@10) metrics, reflecting the robustness imparted by multiple semantically rich captions per image.

6. Semantic Gap Mitigation and Technical Significance

RSRT and its accompanying methods partially resolve the semantic gap in remote sensing imagery retrieval by explicit decomposition of image content into entities, relations, and attributes spanning multiple perspectives. The abundance of caption variants introduces redundancy and diversity in retrieval signals, supporting successful matching at varying semantic granularity: from broad scene types to fine-grained object-level distinctions. This paradigm demonstrates that high-quality, structured textual representation—absent any domain-specific network training—yields retrieval performance comparable to state-of-the-art supervised cross-modal frameworks (Xiao et al., 11 Dec 2025).

7. Usage, Access, and Future Extensions

The RSRT benchmark is publicly available via a dedicated GitHub repository. Reproduction of results requires Python 3.8+, an OpenAI API key (for embedding replication), and standard libraries including FAISS and PyTorch. Scripts for embedding computation, retrieval, and evaluation are provided. Key usage steps are:

1. Acquire RSRT corpus and source images
2. Configure environment and dependencies
3. Compute embeddings: embed_corpus.py
4. Query (text: t2i, image: i2t): retrieve.py
5. Evaluate performance: evaluate.py

Planned forward extensions include multi-lingual caption generation, linking of textual and spatial object annotations (bounding boxes, masks), integration of fully open-source LLM pipelines (e.g., LLaMA-2 with visual adapters), and broadening to new remote sensing modalities such as polarimetric, hyperspectral, and SAR imagery.

RSRT sets a new standard for zero-shot, fine-grained, and semantically transparent benchmarking in remote sensing image retrieval, supporting robust, reproducible, and cost-effective research evaluation (Xiao et al., 11 Dec 2025).