---
title: 'BigEarthNet-MM: Multi-Modal Remote Sensing Dataset'
url: https://www.emergentmind.com/topics/bigearthnet-mm
type: topic
---

# BigEarthNet-MM: Multi-Modal Remote Sensing Dataset

BigEarthNet-MM is a large-scale, multi-modal benchmark archive designed to facilitate deep learning research in remote sensing image classification and retrieval. It consists of co-registered Sentinel-1 Synthetic Aperture Radar (SAR) and Sentinel-2 MultiSpectral Instrument (MSI) image patches, each annotated with multi-labels derived from CORINE Land Cover (CLC) maps. BigEarthNet-MM and its extensions offer a unique testbed for evaluating convolutional neural networks (CNNs), vision transformers (ViTs), and multi-modal fusion strategies in a geospatial context, supporting both supervised and self-supervised learning paradigms.

## 1. Dataset Structure and Modalities

BigEarthNet-MM comprises 590,326 paired Sentinel-1 and Sentinel-2 image patches, sampled across ten European countries, with patches spatially aligned at 10 m ground sampling distance (GSD) over a 120 × 120 pixel grid per patch. Sentinel-2 contributes 12 spectral bands (excluding those at 60 m resolution), while Sentinel-1 offers dual-polarized VV and VH backscatter at 10 m. The resulting multi-modal stack typically forms a tensor of $[12, 120, 120]$ (Sentinel-2 MSI + 2 SAR channels upsampled as needed) [2105.07921]. Patch footprints are defined by subdividing higher-level Sentinel tiles.

Each patch is annotated with at least one land cover/use label. Labeling utilizes the CLC 2018 map at Level-3 nomenclature, later refined to a 19-class compact scheme by merging or omitting classes requiring multi-temporal context or ambiguous in single-date imaging [2105.07921]. A substantial number of patches originally covered by clouds or snow are excluded from experimental splits to maintain label and quality integrity.

Table: Patch Modalities and Shapes

| Sensor/Layer        | Channels | Array Shape      |
|---------------------|----------|------------------|
| Sentinel-2 MSI      | 10       | $10\times120\times120$ (upsampled) |
| Sentinel-1 SAR      | 2        | $2\times120\times120$              |
| Total               | 12       | $12\times120\times120$             |


## 2. Labeling, Nomenclature, and Extensions

Initial labeling relies on CLC Level-3 for fine-grained land cover classes (44 original), but empirical findings showed many classes were unreliably captured in single-date, single-patch imagery. To address this, a 19-class alternative nomenclature groups semantically and visually similar classes, excludes classes needing time series, and merges those with ambiguous boundaries [2105.07921]. Each sample is assigned a binary vector $\mathbf{y} \in \{0,1\}^{19}$ indicating the presence of each class.

Extensions such as reBEN (Refined BigEarthNet) further improve label quality by incorporating the CLC2018 V2020_20u1 version and filtering out patches with insufficiently labeled pixels (less than 75%). This refinement corrects known label errors, such as misclassifications between marine waters and urban fabric, and makes labels more robust for scene and pixel-level learning [2407.03653].

## 3. Preprocessing, Quality Control, and Data Formatting

Patches in BigEarthNet-MM undergo quality screening to eliminate samples with extensive cloud or snow cover. Sentinel-2 L1C data are atmospherically corrected to L2A using Sen2Cor, providing top-of-atmosphere or, with reBEN, bottom-of-atmosphere reflectances. Sentinel-1 undergoes radiometric calibration, terrain correction, and co-registration.

The reBEN pipeline filters tiles using L2A metadata quality-control flags, removes defective tiles entirely, and separates patches with total cloud/snow coverage for flexible benchmarking.

For computational efficiency, data formatting has evolved: original GeoTIFFs are converted using a dedicated tool to an LMDB key-value store with blobs serialized in the .safetensors format. This transformation reduces input/output bottlenecks and accelerates end-to-end training by a factor of 2–3 [2407.03653].

## 4. Experimental Tasks, Model Architectures, and Protocols

BigEarthNet-MM is optimized for two primary machine learning paradigms:

- **Multi-modal, multi-label land-use/land-cover classification**: The goal is to assign a subset of the 19 labels to each patch. State-of-the-art CNNs (ResNet-50, ConvMixer, ConvNeXt-v2), transformers (ViT-Base), and MLP architectures have been evaluated. Training protocols typically use AdamW optimizer, cosine learning rate decay, dropout and drop-path regularization, and batch sizes of up to 512 [2407.03653].

- **Multi-label retrieval**: Feature representations extracted from trained models are compared (e.g., χ² metric) to rank patches by label overlap with a query.

The standard split used in early BigEarthNet-MM is training/validation/test = 269,695/123,723/125,866 pairs, respectively. The reBEN split employs a geographical assignment algorithm, minimizing spatial autocorrelation to yield more reliable generalization estimates.

Evaluation employs macro and micro average precision (AP), F₁-score, precision, one-error, Hamming loss, and the F₂-score which weights recall twice as strongly as precision. For pixel-level segmentation tasks, U-Net architectures are used; loss functions include binary cross-entropy for multi-label tasks and categorical cross-entropy for segmentation [2307.01741].

## 5. Multi-Modal Fusion and Performance Insights

Systematic ablation studies demonstrate that multi-modal fusion consistently outperforms uni-modal baselines. Stacking Sentinel-1 SAR channels with Sentinel-2 MSI bands improves classification metrics, particularly for challenging land-cover classes where spectral and texture cues are complementary.

On reBEN, S1-only classification achieves AP$^m\approx$62–63%, S2-only $\approx$68–70%, while S1+S2 fusion improves AP$^m$ by 0.5–1.0 percentage points; best results with ResNet-50 S1+S2 yield AP$^m$ = 71.36% and F$_1^u\approx$76.44% [2407.03653]. On the original BigEarthNet-MM, a ResNet-50 trained from scratch outperforms ImageNet transfer learning, with average F₂ increasing from 39.81% (transfer) to 67.23% (scratch). Adding Sentinel-1 to Sentinel-2 boosts F1-scores by 5.7 percentage points, and further improvements accrue with additional modalities from ben-ge [2307.01741].

Table: Representative Classification Results

| Model       | S1-only AP$^m$ | S2-only AP$^m$ | S1+S2 AP$^m$ |
|-------------|----------------|----------------|--------------|
| ResNet-50   | 62–63%         | 68–70%         | 71.36%       |
| ViT-Base    | ~4 pp lower    | ~4 pp lower    | ~4 pp lower  |

A plausible implication is that learned representations from multi-modal remote sensing data are more robust to occlusions, seasonal artefacts, and class imbalance than uni-modal or traditional transfer learning approaches.

## 6. Self-Supervised and Geographical/Environmental Extensions

Self-supervised approaches such as DINO-MM demonstrate that vision transformers pre-trained with channel-masking and RandomSensorDrop on BigEarthNet-MM can match or surpass supervised performance, especially under label-scarce regimes. DINO-MM achieves mean average precision (mAP) gains of up to 11.6% over fully supervised counterparts when only 1% of the labels are used, and the resulting encoders generalize across SAR-only, optical-only, and fused inputs without architectural adjustment [2204.05381].

The ben-ge extension further augments each BigEarthNet-MM patch with six additional modalities: Köppen–Geiger climate zones, digital elevation model (DEM), ESA WorldCover 2021 LULC map, season of acquisition, temperature at 2 m, and wind vectors at 10 m [2307.01741]. Fusion strategies exploit these heterogeneous sources using late-fusion architectures (concatenating backbone feature vectors) to push classification F1-scores up to 85.1% when multiple modalities are combined.

## 7. Limitations and Future Research Directions

Despite its scale, BigEarthNet-MM presents limitations. Native resolution mismatches (e.g., DEM at 30 m upsampled to 10 m), variable temporal alignment between environmental layers and image acquisition, and the use of patch-level scalars for features such as climate and season represent sources of label noise or reduced discriminative power.

Future directions, as outlined in the ben-ge and reBEN frameworks, include attention-based or cross-modal fusion, spatially resolved meteorological inputs, and self-supervised or semi-supervised learning exploiting unlabelled data. Expanding the BigEarthNet-MM paradigm to time-series and global scales, as well as integrating more granular environmental and geographical data, is expected to further advance generalization and representational capacities in remote sensing deep learning [2307.01741][2407.03653].

All data, code, and pre-trained models for BigEarthNet-MM, reBEN, and their extensions are openly available for academic use at https://bigearth.net.

Source: https://www.emergentmind.com/topics/bigearthnet-mm