BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation

Abstract: Vision-langugage models (VLMs) have shown strong performance in computer vision (CV), yet their performance on remote sensing (RS) data remains limited due to the lack of large-scale, multi-sensor RS image-text datasets with diverse textual annotations. Existing datasets predominantly include aerial Red-Green-Blue imagery, with short or weakly grounded captions, and provide limited diversity in annotation types. To address this limitation, we introduce BigEarthNet$.$txt, a large-scale, multi-sensor image-text dataset designed to advance instruction-driven image-text learning in Earth observation across multiple tasks. BigEarthNet$.$txt contains 464044 co-registered Sentinel-1 synthetic aperture radar and Sentinel-2 multispectral images with 9.6M text annotations, including: i) geographically anchored captions describing land-use/land-cover (LULC) classes, their spatial relations, and environmental context; ii) visual question answering pairs relevant for different tasks; and iii) referring expression detection instructions for bounding box prediction. Through a comparative statistical analysis, we demonstrate that BigEarthNet$.$txt surpasses existing RS image-text datasets in textual richness and annotation type variety. We further establish a manually-verified benchmark split to evaluate VLMs in RS and CV. The results show the limitations of these models on tasks that involve complex LULC classes, whereas fine-tuning using BigEarthNet$.$txt results in consistent performance gains across all considered tasks.

• The paper introduces BigEarthNet.txt, a large-scale multi-sensor dataset with 464,044 Sentinel-1/Sentinel-2 image pairs and 9.6 million detailed text annotations for 15 EO tasks.
• The paper details a novel annotation pipeline combining template extraction, metadata infusion, and LLM-based self-refinement that achieves 93.76% correctness in semantic grounding.
• The paper demonstrates significant performance gains when fine-tuning vision-language models on BigEarthNet.txt, with improvements like a BLEU-4 score of 34.04 and binary VQA accuracy of 73.29%.

BigEarthNet.txt: A Comprehensive Multi-Sensor Image-Text Benchmark for Earth Observation

Motivation and Context

Vision-LLMs (VLMs), prominent in computer vision (CV), have demonstrated limited efficacy for remote sensing (RS) due to a scarcity of large-scale, multi-sensor image-text datasets featuring rich, semantically diverse textual annotations. Existing RS datasets predominantly comprise RGB aerial imagery with concise and often weakly grounded captions, lacking annotation diversity essential for complex land-use/land-cover (LULC) discrimination and multimodal understanding. This restricts VLMs from fully exploiting RS data's spatial and spectral intricacies, impeding generalized task performance, especially on non-trivial LULC queries.

Dataset Construction and Annotation Pipeline

BigEarthNet.txt is introduced to fill these ecological and annotation gaps. It consists of 464,044 co-registered Sentinel-1 SAR and Sentinel-2 multispectral images, matched with approximately 9.6 million high-quality textual annotations supporting 15 downstream tasks across four categories: captioning, binary VQA, multiple-choice questions, and referring expression detection.

Figure 1: BigEarthNet.txt comprises 464,044 co-registered S1 and S2 images with diverse text annotations, totaling $\sim$9.6 million S1-S2-text triplets across 15 RS tasks.

The annotation pipeline combines template-driven attribute extraction from pixel-level LULC reference maps, contextual metadata infusion (season, country, climate zone), and LLM augmentation for lexical and syntactic diversity. This process includes paraphrasing and iterative self-refinement via Llama-4-Scout-17B, achieving an average correctness rate of 93.76% and comprehensive semantic grounding. VQA pairs are constructed for binary presence queries, count puzzles, spatial adjacency, and area estimation; referring expressions instruct bounding box or centroid localization for highly granular LULC instances.

Figure 2: Caption generation process extracts spatial, contextual attributes from reference maps, concatenates templates, and applies LLM-based self-refinement for lexical diversity and factual correctness.

Statistical Analysis and Semantic Diversity

BigEarthNet.txt captions contain roughly 50 million words and 2.1 million sentences, averaging 107 words and 4.5 sentences per sample. The vocabulary comprises 12,394 unique terms. Caption lengths exhibit bimodal distributions linked to scene complexity; sentence counts primarily peak between four and six. The dataset achieves a measure of textual lexical diversity (MTLD) score of 64.69, exceeding the most semantically rich prior multi-sensor RS dataset (MS-CLIP) by a factor of 1.7, with 25% more words in half as many samples.

Figure 3: PDF of caption lengths and sentence counts; BigEarthNet.txt outperforms prior RS datasets in size and lexical diversity metrics.

Benchmarking Vision-LLMs

The BigEarthNet.txt benchmark split encompasses manually verified, balanced samples for robust evaluation. Systematic experiments on this split involve general-purpose CV VLMs and RS-specialized VLMs, incorporating models such as GPT-5.2, Qwen3-VL, GLM-4.6v, LLaVa-OneVision, InternVL3-1B, GeoChat, LHRS-Bot, SkyEyeGPT, EarthDial, and EarthMind. Inputs are restricted appropriately by sensor modality, isolating the impact of RGB versus multispectral/multisensor information.

Quantitative evaluation reveals that state-of-the-art VLMs, irrespective of domain specialization or scale, exhibit limited accuracy. For instance, binary VQA performance tops out at 61.96% (Qwen3-VL), MCQ at 37.55%, and referring expression detection mIoU at 31.73% (GPT-5.2). RS-specialized models yield no substantial advantage, and inclusion of additional spectral bands without suitable pre-training does not improve results. Only tasks closely linked to pre-training objectives (e.g., binary presence) manifest moderate success (EarthMind: 69.34% accuracy).

Model Adaptation and Fine-Tuning Results

To demonstrate the impact of high-quality, multimodal training data, InternVL3-1B is adapted for multi-sensor input (RS-InternVL). The architecture incorporates frozen Vision Transformer (ViT) encoders for each sensor, aligned via modality-specific projections and LoRA adapters; fine-tuning is performed separately for each task using the full BigEarthNet.txt training and validation splits.

Fine-tuned RS-InternVL yields substantial performance gains: BLEU-4 of 34.04 (captioning), accuracy of 73.29% (binary VQA), 51.49% (MCQ), and mIoU of 65.84% (referring expression detection). This improvement, averaging 31.52% over the best baseline, indicates that suboptimal performance of other VLMs is predominantly attributable to insufficient task-aligned pre-training data rather than inherent architectural deficiencies.

Practical and Theoretical Implications

BigEarthNet.txt offers a paradigm shift in RS VLM research, establishing an unprecedented standard for multimodal RS datasets. Its breadth, annotation diversity, and semantic granularity enable development, evaluation, and fine-tuning of multimodal foundation models capable of nuanced LULC reasoning, spatial relations, and complex query resolution. The demonstrated improvements from multi-sensor fine-tuning imply that model architectures can leverage added spectral data when trained on adequate resources, leading to generalizable, instruction-following models suitable for EO applications, including environmental monitoring, land use classification, and interactive geospatial querying.

Implications for future AI developments include the necessity for large-scale, richly annotated, sensor-diverse datasets in other remote domains, further exploration of instruction tuning, and increased democratization of EO via natural language interfaces. The flexibility to extend to temporal sequences or higher-level domain adaptation (climate, agricultural, disaster assessment) is also evident.

Conclusion

BigEarthNet.txt substantially advances the state of RS image-text datasets by unifying co-registered multi-sensor imagery with exhaustive textual annotations and benchmark splits. The dataset enables rigorous evaluation and improvement of vision-LLMs tailored for complex EO tasks, establishing that data scarcity, not architectural limitations, restricts model performance. The successful adaptation of InternVL for RS validates the utility of multi-sensor datasets and lays the groundwork for more accurate, interactive EO modeling via natural language queries.