HQRS-CLIP: Remote Sensing Vision-Language Model

• The paper introduces HQRS-CLIP, which leverages a high-quality image–text dataset and multi-perspective captioning to enhance remote sensing task performance using a dual-encoder CLIP architecture.
• It employs a two-stage captioning pipeline that fuses rule-based, MLLM, and LLM-generated descriptions to produce detailed captions optimized for retrieval and semantic localization.
• The model achieves state-of-the-art accuracy and efficiency in zero-shot and fine-tuned settings across remote sensing benchmarks, outperforming prior approaches with only 210K pairs.

HQRS-CLIP is a remote sensing–specialized adaptation of the CLIP vision-LLM, distinguished by its use of a meticulously constructed, high-quality image–text paired dataset (HQRS-IT-210K) generated via multi-perspective MLLM prompting and LLM-based fusion. The HQRS-CLIP framework demonstrates that careful curation and advanced captioning strategies can yield state-of-the-art performance for remote sensing vision-language tasks with orders of magnitude less data than previous approaches, while retaining the standard CLIP two-tower architecture and contrastive training paradigm (He et al., 22 Jul 2025).

1. Architecture and Training Objective

HQRS-CLIP inherits the CLIP (Contrastive Language-Image Pretraining) dual-encoder architecture: a Vision Transformer (ViT, both ViT-B-32 and ViT-L-14 variants) operates as the image encoder with standard ViT patch embedding, while a Transformer-based text encoder processes tokenized captions using the original CLIP 49K BPE vocabulary with a 77-token maximum. Final layer [CLS] embeddings from both towers are projected to a joint 512-dimensional, L₂-normalized latent space. Cosine similarity between embeddings is modulated by a learned temperature parameter $τ$.

The model is trained with a symmetric InfoNCE loss. For a mini-batch with $N$ image–text pairs $(x_i, t_i)$, define

$$s_{ij} = \frac{ \text{cos}( \text{Proj}_v(f_v(x_i)), \text{Proj}_t(f_t(t_j)) ) }{ τ }$$

The per-pair loss is:

$$L_i = -\log \frac{ \exp(s_{ii}) }{ \sum_{j=1}^N \exp(s_{ij}) } - \log \frac{ \exp(s_{ii}) }{ \sum_{j=1}^N \exp(s_{ji}) }$$

and the total loss is averaged over the batch.

Both the image and text towers are initialized from pretrained CLIP weights and fully fine-tuned on HQRS-IT-210K, emphasizing data and caption quality over pretraining scale.

2. Construction of the HQRS-IT-210K Dataset and Captioning Pipeline

HQRS-IT-210K comprises 210,556 remote sensing images collected from 23 public aerial/satellite datasets (classification, detection, segmentation, UAV). Each image is paired with six captions (≈1.26M pairs total), created via a two-stage Multi-Perspective Generation and Integration (MpGI) pipeline:

Stage 1 (Multi-Perspective Generation):

• Rule-MLLM Relay (R-M Relay) converts object detection/segmentation annotations into structured English using the Annotation-to-Description (A2D) algorithm. For classification data, class names are expanded into natural language with ChatGPT-4V.
• Instruction-Guided MLLM Generation employs Kosmos-2 and Llava-1.6 with prompts conditioned on class and bounding box context to yield diverse, detailed descriptions; several distinct prompts are used per model to maximize descriptive coverage.

Stage 2 (LLM-Based Summarization and Fusion):

• Three Stage 1 descriptions per image enter LLaMA-3-8B-Instruct, using two prompt variants (derived from CO-STAR) to produce concise, content-rich summaries (<77 tokens). For each image, either Prompt1 or one of five Prompt2 summaries is randomly selected, with α ≈ 0.5 probability for Prompt2 maximizing downstream retrieval and semantic localization performance.

Key dataset statistics:

• Images: 210,556
• Captions per image: 6
• Average caption length: 35.6 words (substantially exceeding standard human-annotated RS sets)
• Rich vocabulary and tailored length distribution for CLIP compatibility.

3. Fine-Tuning Protocol

Fine-tuning uses only 1/6 of the full HQRS-IT-210K (one caption per image, 210,556 pairs), with 15% reserved for validation, highlighting the model’s efficiency with respect to high-quality data density.

Key hyperparameters:

• Optimizer: AdamW (weight decay 0.05)
• Learning rate: ViT-B-32, 2 × 10⁻⁵; ViT-L-14, 1 × 10⁻⁶
• Batch size: ViT-B-32, 256; ViT-L-14, 32
• Mixed precision enabled
• No additional image augmentations or prompt-tuning is applied

This empirically prioritizes caption informativeness and data curation over brute-force scaling.

4. Benchmark Performance Across Remote Sensing Tasks

HQRS-CLIP achieves SOTA results across remote sensing benchmarks, with particularly strong data-efficiency:

• Zero-Shot Cross-Modal Retrieval (RSCTIR): With only 210K pairs, HQRS-CLIP ViT-B-32 surpasses previous SOTA GeoRSCLIP (trained on 5M pairs) by +3.5% mean recall (RSITMD/RSICD/UCM), ViT-L-14 yields an additional +1.06%.
• Fine-Tuned Retrieval: Outperforms RemoteCLIP and GeoRSCLIP on RET-3, raising mean recall by +2.9% and +2.3% relative to prior bests.
• Long-Text Retrieval: Excels on long, LLM-rewritten queries (mean recall 33.87% zero-shot, 39.03% after fine-tuning), exceeding previous methods by 6.9–8.0%.
• Semantic Localization (AIR-SLT): Highest Rsu=0.7635, lowest Ras=0.2488, highest Rda=0.7340 and Rmi=0.7518, outperforming all prior RS-specialized CLIPs.
• Zero-Shot and Few-Shot Classification: Top-1 accuracy of 70.94% across AID/RESISC45/EuroSAT (+1.91% over GeoRSCLIP, +4.7% over RemoteCLIP), using only 4.2% as much data as RS5M. HQRS-CLIP also leads or ties in 15 of 16 N-way K-shot classification setups.

Ablations identify that:

• Multi-perspective, complete captions raise retrieval recall @1 from 32.49% to 37.40%.
• Retrieval scales linearly up to 210K images, while semantic localization saturates around 50K.
• Caption length up to ~50 words improves performance before exceeding CLIP’s token budget.

5. Ablation Results and Empirical Insights

Factor	Effect on Performance (Summary)
Multi-perspective captions	Improves retrieval/semantic localization metrics
Dataset scale	Linear performance gains in retrieval up to 210K
Caption length	Peak at 50 words; degradation beyond 77 tokens
Captions/image	Marginal returns after 2–3 for ViT-B; up to 5 for ViT-L
Sub-dataset composition	CLS and DET dominate gains; UAV and MSRGB boost diversity
Captioning pipeline ablation	Progressive improvements: rule-based < MLLM < LLM fusion

Each major stage of the captioning pipeline (rule-based, MLLM, LLM summaries, fusion) adds quantitative value, with final fusion yielding optimal overall recall.

6. Limitations and Prospective Directions

Identified limitations include:

• Occasional hallucinations in object/location details remain, despite manual and regex filters, stemming from MLLM/LLM caption generation. More robust hallucination mitigation via prompting or verification is needed.
• Caption style bias can persist due to underrepresentation of certain scene types, even with prompt and style sampling.
• CLIP’s 77-token truncation introduces some information loss; longer-context models such as Long-CLIP represent a plausible extension.
• The framework is not yet scaled to training large multimodal LLMs for end-to-end remote sensing understanding; adaptation to models like LLaVA or Qwen-VL is suggested as a next step.

7. Significance for Remote Sensing Vision-Language Research

HQRS-CLIP empirically demonstrates that domain-adapted, high-informativeness image–text pairs—curated with advanced MLLM/LLM pipelines—can yield state-of-the-art accuracy for retrieval, classification, and localization tasks in remote sensing, while remaining highly parameter- and data-efficient. This paradigm stands in contrast to prior approaches that rely on noisy, massive datasets and highlights an emerging preference for data quality and descriptive richness over scale alone. Resources associated with HQRS-CLIP, including the HQRS-IT-210K dataset and pretrained models, are available for further research and reproducibility (He et al., 22 Jul 2025).