BigEarthNet-MM: Multi-Modal Remote Sensing Dataset

• BigEarthNet-MM is a large-scale benchmark comprising co-registered Sentinel-1 SAR and Sentinel-2 MSI image patches annotated with multi-labels from CORINE Land Cover maps.
• It supports both supervised and self-supervised learning, enabling evaluations of CNNs, vision transformers, and multi-modal fusion strategies in remote sensing.
• Optimized processing pipelines including quality control and efficient data formatting boost training speed by 2–3 times, enhancing robust geospatial analysis.

BigEarthNet-MM is a large-scale, multi-modal benchmark archive designed to facilitate deep learning research in remote sensing image classification and retrieval. It consists of co-registered Sentinel-1 Synthetic Aperture Radar (SAR) and Sentinel-2 MultiSpectral Instrument (MSI) image patches, each annotated with multi-labels derived from CORINE Land Cover (CLC) maps. BigEarthNet-MM and its extensions offer a unique testbed for evaluating convolutional neural networks (CNNs), vision transformers (ViTs), and multi-modal fusion strategies in a geospatial context, supporting both supervised and self-supervised learning paradigms.

1. Dataset Structure and Modalities

BigEarthNet-MM comprises 590,326 paired Sentinel-1 and Sentinel-2 image patches, sampled across ten European countries, with patches spatially aligned at 10 m ground sampling distance (GSD) over a 120 × 120 pixel grid per patch. Sentinel-2 contributes 12 spectral bands (excluding those at 60 m resolution), while Sentinel-1 offers dual-polarized VV and VH backscatter at 10 m. The resulting multi-modal stack typically forms a tensor of $[12, 120, 120]$ (Sentinel-2 MSI + 2 SAR channels upsampled as needed) (Sumbul et al., 2021). Patch footprints are defined by subdividing higher-level Sentinel tiles.

Each patch is annotated with at least one land cover/use label. Labeling utilizes the CLC 2018 map at Level-3 nomenclature, later refined to a 19-class compact scheme by merging or omitting classes requiring multi-temporal context or ambiguous in single-date imaging (Sumbul et al., 2021). A substantial number of patches originally covered by clouds or snow are excluded from experimental splits to maintain label and quality integrity.

Table: Patch Modalities and Shapes

Sensor/Layer	Channels	Array Shape
Sentinel-2 MSI	10	$10\times120\times120$ (upsampled)
Sentinel-1 SAR	2	$2\times120\times120$
Total	12	$12\times120\times120$

2. Labeling, Nomenclature, and Extensions

Initial labeling relies on CLC Level-3 for fine-grained land cover classes (44 original), but empirical findings showed many classes were unreliably captured in single-date, single-patch imagery. To address this, a 19-class alternative nomenclature groups semantically and visually similar classes, excludes classes needing time series, and merges those with ambiguous boundaries (Sumbul et al., 2021). Each sample is assigned a binary vector $\mathbf{y} \in \{0,1\}^{19}$ indicating the presence of each class.

Extensions such as reBEN (Refined BigEarthNet) further improve label quality by incorporating the CLC2018 V2020_20u1 version and filtering out patches with insufficiently labeled pixels (less than 75%). This refinement corrects known label errors, such as misclassifications between marine waters and urban fabric, and makes labels more robust for scene and pixel-level learning (Clasen et al., 2024).

3. Preprocessing, Quality Control, and Data Formatting

Patches in BigEarthNet-MM undergo quality screening to eliminate samples with extensive cloud or snow cover. Sentinel-2 L1C data are atmospherically corrected to L2A using Sen2Cor, providing top-of-atmosphere or, with reBEN, bottom-of-atmosphere reflectances. Sentinel-1 undergoes radiometric calibration, terrain correction, and co-registration.

The reBEN pipeline filters tiles using L2A metadata quality-control flags, removes defective tiles entirely, and separates patches with total cloud/snow coverage for flexible benchmarking.

For computational efficiency, data formatting has evolved: original GeoTIFFs are converted using a dedicated tool to an LMDB key-value store with blobs serialized in the .safetensors format. This transformation reduces input/output bottlenecks and accelerates end-to-end training by a factor of 2–3 (Clasen et al., 2024).

4. Experimental Tasks, Model Architectures, and Protocols

BigEarthNet-MM is optimized for two primary machine learning paradigms:

• Multi-modal, multi-label land-use/land-cover classification: The goal is to assign a subset of the 19 labels to each patch. State-of-the-art CNNs (ResNet-50, ConvMixer, ConvNeXt-v2), transformers (ViT-Base), and MLP architectures have been evaluated. Training protocols typically use AdamW optimizer, cosine learning rate decay, dropout and drop-path regularization, and batch sizes of up to 512 (Clasen et al., 2024).
• Multi-label retrieval: Feature representations extracted from trained models are compared (e.g., χ² metric) to rank patches by label overlap with a query.

The standard split used in early BigEarthNet-MM is training/validation/test = 269,695/123,723/125,866 pairs, respectively. The reBEN split employs a geographical assignment algorithm, minimizing spatial autocorrelation to yield more reliable generalization estimates.

Evaluation employs macro and micro average precision (AP), F₁-score, precision, one-error, Hamming loss, and the F₂-score which weights recall twice as strongly as precision. For pixel-level segmentation tasks, U-Net architectures are used; loss functions include binary cross-entropy for multi-label tasks and categorical cross-entropy for segmentation (Mommert et al., 2023).

5. Multi-Modal Fusion and Performance Insights

Systematic ablation studies demonstrate that multi-modal fusion consistently outperforms uni-modal baselines. Stacking Sentinel-1 SAR channels with Sentinel-2 MSI bands improves classification metrics, particularly for challenging land-cover classes where spectral and texture cues are complementary.

On reBEN, S1-only classification achieves AP$^m\approx$62–63%, S2-only $\approx$68–70%, while S1+S2 fusion improves AP$^m$ by 0.5–1.0 percentage points; best results with ResNet-50 S1+S2 yield AP$^m$ = 71.36% and F$_1^u\approx$76.44% (Clasen et al., 2024). On the original BigEarthNet-MM, a ResNet-50 trained from scratch outperforms ImageNet transfer learning, with average F₂ increasing from 39.81% (transfer) to 67.23% (scratch). Adding Sentinel-1 to Sentinel-2 boosts F1-scores by 5.7 percentage points, and further improvements accrue with additional modalities from ben-ge (Mommert et al., 2023).

Table: Representative Classification Results

Model	S1-only AP$^m$	S2-only AP$^m$	S1+S2 AP$^m$
ResNet-50	62–63%	68–70%	71.36%
ViT-Base	~4 pp lower	~4 pp lower	~4 pp lower

A plausible implication is that learned representations from multi-modal remote sensing data are more robust to occlusions, seasonal artefacts, and class imbalance than uni-modal or traditional transfer learning approaches.

6. Self-Supervised and Geographical/Environmental Extensions

Self-supervised approaches such as DINO-MM demonstrate that vision transformers pre-trained with channel-masking and RandomSensorDrop on BigEarthNet-MM can match or surpass supervised performance, especially under label-scarce regimes. DINO-MM achieves mean average precision (mAP) gains of up to 11.6% over fully supervised counterparts when only 1% of the labels are used, and the resulting encoders generalize across SAR-only, optical-only, and fused inputs without architectural adjustment (Wang et al., 2022).

The ben-ge extension further augments each BigEarthNet-MM patch with six additional modalities: Köppen–Geiger climate zones, digital elevation model (DEM), ESA WorldCover 2021 LULC map, season of acquisition, temperature at 2 m, and wind vectors at 10 m (Mommert et al., 2023). Fusion strategies exploit these heterogeneous sources using late-fusion architectures (concatenating backbone feature vectors) to push classification F1-scores up to 85.1% when multiple modalities are combined.

7. Limitations and Future Research Directions

Despite its scale, BigEarthNet-MM presents limitations. Native resolution mismatches (e.g., DEM at 30 m upsampled to 10 m), variable temporal alignment between environmental layers and image acquisition, and the use of patch-level scalars for features such as climate and season represent sources of label noise or reduced discriminative power.

Future directions, as outlined in the ben-ge and reBEN frameworks, include attention-based or cross-modal fusion, spatially resolved meteorological inputs, and self-supervised or semi-supervised learning exploiting unlabelled data. Expanding the BigEarthNet-MM paradigm to time-series and global scales, as well as integrating more granular environmental and geographical data, is expected to further advance generalization and representational capacities in remote sensing deep learning (Mommert et al., 2023, Clasen et al., 2024).

All data, code, and pre-trained models for BigEarthNet-MM, reBEN, and their extensions are openly available for academic use at https://bigearth.net.