RS3DBench: 3D Remote Sensing Benchmark

• RS3DBench is a remote-sensing benchmark for 3D spatial perception with aligned pixel-level RGB–depth and text annotations.
• It provides 54,951 image-depth-text pairs from diverse terrains, supporting robust depth estimation and terrain modeling.
• It pioneers text-conditioned diffusion modeling, achieving improved accuracy and cross-terrain generalization in geographic AI.

Searching arXiv for the specified paper to ground the article and citation. {"query":"(Wang et al., 23 Sep 2025) RS3DBench: A Comprehensive Benchmark for 3D Spatial Perception in Remote Sensing","max_results":5,"sort_by":"submittedDate","sort_order":"descending"} RS3DBench is a remote-sensing benchmark for 3D spatial perception built around pixel-aligned RGB–depth pairs plus text descriptions, introduced to support training and evaluation of models that require geometric understanding of Earth surface scenes from overhead imagery (Wang et al., 23 Sep 2025). The benchmark contains 54,951 pairs of remote sensing images and pixel-level aligned depth maps, each paired with a textual description, and is intended to address a recurrent deficiency in prior remote-sensing resources: many datasets provide either only 2D labels or captions, or depth and elevation data without precise alignment to imagery. Within this formulation, RS3DBench is positioned as a foundation for remote-sensing depth estimation, terrain modeling, broader 3D geospatial intelligence, and the development of general-purpose, large-scale 3D vision models in geographic artificial intelligence.

1. Conceptual motivation and problem setting

The benchmark is motivated by a shift in remote sensing from predominantly 2D classification and detection toward 3D understanding. The paper frames this transition in terms of estimating DEM-like depth from RGB imagery, modeling terrain shape and elevation changes, supporting digital cities, disaster assessment, and geographic AI, and enabling future outdoor visual-language or navigation systems that require top-down spatial reasoning (Wang et al., 23 Sep 2025).

A central premise is that remote-sensing imagery imposes constraints that differ markedly from indoor and driving-scene depth benchmarks. The paper identifies four specific difficulties: objects can be extremely small in pixel space; scale varies massively across regions; terrain can change abruptly; and depth or elevation data are expensive to acquire and are often not aligned to RGB. RS3DBench is therefore designed not simply as another image collection, but as a data substrate for supervised 3D learning under geospatial conditions where both geometric and semantic ambiguity are substantial.

The paper repeatedly emphasizes that the benchmark is the first remote-sensing dataset with aligned spatial information and with both text, depth, and RGB in a unified benchmark. This suggests that its novelty lies not only in scale, but in the joint availability of multimodal supervision targeted at spatial understanding rather than conventional scene recognition alone.

2. Dataset composition, geographic scope, and terrain structure

RS3DBench contains 54,951 pairs of remote sensing RGB images and pixel-level aligned depth maps, and each pair is associated with a text description (Wang et al., 23 Sep 2025). The benchmark spans multiple countries across four continents and is intended to provide diversity across large-scale geographic contexts. The reported geographic coverage includes coastal lowlands, highlands, mountainous regions, and plains.

The terrain taxonomy is organized into six terrain types: Ocean, Plain, Hill, Low undulating mountains, High undulating mountains, and Highland. Their reported proportions are Ocean: 3.3%, Plain: 38.0%, Hill: 6.4%, Low undulating mountains: 9.1%, High undulating mountains: 29.8%, and Highland: 13.5%. The paper also reports an elevation range from $-149 \, \text{m}$ to $4813 \, \text{m}$, with ocean depth values near 0. This distribution indicates that the benchmark is not restricted to a narrow class of landforms; instead, it includes both low-variance and high-variance elevation regimes, which is consequential for evaluating cross-terrain robustness.

The dataset includes four spatial resolutions: 30 m, 5 m, 2 m, and 0.5 m. The paper notes a systematic relationship between resolution and terrain composition: lower-resolution data at 30 m and 5 m is more dominated by plains and highlands, whereas higher-resolution data at 2 m and 0.5 m contains more mountainous terrain. A plausible implication is that the benchmark can probe both scale sensitivity and the interaction between terrain complexity and sensor granularity.

The depth maps are assembled from multiple DEM sources. ALOS AW3D30 provides 30 m resolution data for Japan, South Korea, Southeast Asia, and the Mediterranean, based on PRISM stereo imaging, with stated vertical accuracy of RMSE 5 m in flat areas and RMSE 4 m in undulating terrain. SwissALTI3D provides 2 m and 0.5 m resolution data for Switzerland, using LiDAR and aerial photogrammetry fusion, stored in EPSG:2056 (LV95), with stated accuracy of 0.3–0.6 m RMSE in forest-covered areas and 0.8–1.2 m RMSE in exposed rock walls. The Australian DEM provides 5 m resolution data derived from Sentinel-2 stereo pairs via photogrammetry, synchronized RGB from Google Earth Engine, and covers parts of Australia including coastal lowlands and mountain ranges.

3. Alignment, annotation, and post-processing pipeline

The dataset construction pipeline consists of four stages: data crawling, alignment, annotations, and post-processing (Wang et al., 23 Sep 2025). Among these, pixel-level alignment is presented as a principal contribution because many existing DEM datasets or remote-sensing datasets include depth or elevation information without pixel-level registration to RGB imagery.

The alignment procedure begins by obtaining the geographic coordinates and coordinate-system IDs of the DEM data. These are converted into coordinate systems such as EPSG:4326 and EPSG:2056. Corresponding RGB imagery is then retrieved from sources including Sentinel-1, Sentinel-2, swissALTI3D, and Google Earth. To improve alignment and data quality, the RGB images undergo cloud removal, haze removal, atmospheric correction, radiometric correction, and normalization. The paper further states that some RGB and DEM samples were not perfectly aligned in time or space, and that roughly 20% to 30% of misaligned images were manually screened and filtered. Sensor-induced outliers were identified and corrected via interpolation to avoid misleading training signals.

Text annotation is generated using GLM-v4 through a remote-sensing-specific prompting and validation pipeline. The prompt instructs the LLM to describe terrain categories and landform elements such as mountains, oceans, rivers, plains, islands, ridges, and farmland. A constraint mechanism is used to connect the text to terrain classification and landform description, and the prompt was iteratively refined to improve output quality. Annotation quality control combines manual and automated review to detect API timeouts, network errors, and unreasonable annotations; bad annotations are manually re-annotated. As a result, all image pairs receive semantically meaningful text descriptions that reflect spatial features and terrain details.

Post-processing includes resizing samples to $512 \times 512$ pixels. Because Sentinel-2 imagery can be dark or exhibit atmospheric color issues, the paper applies a three-stage enhancement process: normalizing each RGB channel separately to 0–255, clipping pixel values using percentile bounds, and linearly remapping brightness back to 0–255. The final chosen percentile stretch is 1%, which the paper states improves contrast while reducing haze and color cast. The appendix gives a linear stretching formula, although the printed expression is described as garbled; the intended operation is standard linear stretching after percentile-based clipping.

4. Supported tasks and evaluation protocol

RS3DBench is framed around 3D spatial perception in remote sensing, with remote-sensing depth estimation as the principal task in the experiments (Wang et al., 23 Sep 2025). In that setting, the input is an RGB remote sensing image and the output is a depth or DEM-style map. The benchmark is used to measure how accurately models recover terrain elevation and geometric structure from aerial or satellite imagery.

The paper explicitly supports cross-terrain generalization. Models are tested on terrain regimes such as plains versus mountains, and more generally on low-variance versus high-variance elevation regions. Because each image also has an associated text description, the benchmark supports text-conditioned depth estimation, allowing evaluation of whether geographic semantics improve depth prediction. Beyond the reported experiments, the benchmark is positioned as a foundation for 3D terrain modeling, dynamic environment simulation, digital city and disaster-warning applications, and future spatiotemporal modeling.

Evaluation uses three standard depth-estimation metrics. Threshold accuracy is defined for $k \in \{1,2,3\}$ as

$$\delta^{k} = \frac{1}{M}\sum_{i=1}^M \mathbb{I}\left(\max\left(\frac{d_i}{a_i}, \frac{a_i}{d_i}\right) < 1.25^k\right),$$

where $d_i$ is the predicted depth, $a_i$ is the ground-truth depth, and $M$ is the number of valid pixels. Higher $\delta^k$ indicates better accuracy. The other two metrics are

$$\text{RMSE} = \sqrt{\frac{1}{M}\sum_{i=1}^M (d_i - a_i)^2}$$

and

$4813 \, \text{m}$0

for which lower values are better.

The main depth-estimation split contains 38,875 pairs at 30 m resolution and is divided into 80% train, 10% validation, and 10% test. Two terrain-specific subsets are also constructed: D1, consisting mostly of plain terrain, and D2, consisting mostly of mountainous terrain. Each subset contains 2,400 pairs with an 80/10/10 split. To test the effect of text conditioning, the paper further defines a balanced subset with six terrain types and 400 samples per type, for a total of 2,400 samples; 200 samples per category are selected for test and validation, with the remainder used for training. Experiments ran on NVIDIA A6000 GPUs, training time ranged from a few hours to two weeks, and the best checkpoint was selected using validation performance.

5. Marigold-RS and text-conditioned diffusion modeling

The paper proposes a remotely sensed depth estimation model derived from stable diffusion, effectively called Marigold-RS, as a benchmark-leading model on RS3DBench (Wang et al., 23 Sep 2025). It extends Marigold by incorporating text conditioning from geographic descriptions. The model treats depth estimation as a conditional denoising diffusion problem with conditions given by the RGB image $4813 \, \text{m}$1 and text annotation $4813 \, \text{m}$2, and target depth map $4813 \, \text{m}$3, modeled as $4813 \, \text{m}$4.

The forward diffusion process adds noise to the ground-truth depth:

$4813 \, \text{m}$5

with

$4813 \, \text{m}$6

Training minimizes

$4813 \, \text{m}$7

so that the U-Net predicts the noise residual using the noisy depth latent $4813 \, \text{m}$8, the image latent $4813 \, \text{m}$9, the text embedding $512 \times 512$0, and timestep $512 \times 512$1.

The architecture is described through three components. First, text preprocessing and encoding: the text $512 \times 512$2 is encoded by a pretrained CLIP text encoder,

$512 \times 512$3

Second, cross-modal fusion: RGB images and depth maps are encoded by a pretrained VAE into latent spaces $512 \times 512$4 and $512 \times 512$5; because depth is single-channel, it is triplicated to 3 channels to fit the pretrained encoder; text embeddings are fused using cross-attention,

$512 \times 512$6

Third, noise residual prediction is performed by the U-Net,

$512 \times 512$7

The key adaptation is the injection of geographic semantics into the diffusion pipeline through text-conditioned cross-attention. The model initializes from stable diffusion v2, freezes the VAE, and fine-tunes only the U-Net and the CLIP encoder or text adapter. Reported training hyperparameters are learning rate $512 \times 512$8, batch size 24 for Marigold and 28 for Marigold-RS, and an IterExponential scheduler with the first 100 steps warmed up linearly and then exponentially decayed to 1% of the initial learning rate at 25,000 steps. Training runs for 100K iterations on NVIDIA A6000 hardware.

6. Empirical results, comparative performance, and broader significance

On the 30 m depth-estimation split, the paper compares pix2pix, Marigold, AdaBins, Omnidata, HDN, DPT, and the proposed model (Wang et al., 23 Sep 2025). The reported results are as follows.

Model	MAE ↓	RMSE ↓
pix2pix	34.8	42.4
Marigold	24.8	30.9
AdaBins	25.5	31.6
Omnidata	29.5	34.7
HDN	25.0	31.0
DPT	28.1	34.0
Ours	23.4	29.7

Model	$512 \times 512$9 ↑	$k \in \{1,2,3\}$0 ↑	$k \in \{1,2,3\}$1 ↑
pix2pix	32.7	56.5	71.8
Marigold	45.5	70.7	82.2
AdaBins	43.6	70.5	83.0
Omnidata	42.9	67.3	79.7
HDN	45.1	68.9	80.2
DPT	44.8	69.6	81.8
Ours	46.6	70.9	84.9

The paper’s main claim is that the proposed text-conditioned diffusion model achieves the best overall performance on RS3DBench. It specifically reports that MAE improves by 5.6% versus Marigold, together with better global geometry consistency and stronger semantic guidance in complex terrain.

Cross-terrain generalization results further differentiate the methods. On D1, the mostly plain subset, the proposed model achieves MAE 12.7, RMSE 16.8, $k \in \{1,2,3\}$2 59.5, $k \in \{1,2,3\}$3 77.9, and $k \in \{1,2,3\}$4 87.5. On D2, the mostly mountainous subset, it achieves MAE 20.5, RMSE 28.0, $k \in \{1,2,3\}$5 42.3, $k \in \{1,2,3\}$6 66.7, and $k \in \{1,2,3\}$7 79.0. Relative to Marigold, the paper reports MAE improvements of 8.6% on D1 and 15.3% on D2. The authors interpret this as evidence that textual geographic semantics help especially in difficult mountainous terrain. This suggests that semantic conditioning is not merely auxiliary metadata, but may function as a prior that reduces ambiguity when local appearance alone is insufficient to recover elevation structure.

On the balanced six-terrain text-driven subset, the text-guided model again yields the best reported results, with MAE 21.3, RMSE 29.5, $k \in \{1,2,3\}$8 46.2, $k \in \{1,2,3\}$9 71.5, and $$\delta^{k} = \frac{1}{M}\sum_{i=1}^M \mathbb{I}\left(\max\left(\frac{d_i}{a_i}, \frac{a_i}{d_i}\right) < 1.25^k\right),$$0 85.5, compared with Marigold’s MAE 25.0, RMSE 31.4, and $$\delta^{k} = \frac{1}{M}\sum_{i=1}^M \mathbb{I}\left(\max\left(\frac{d_i}{a_i}, \frac{a_i}{d_i}\right) < 1.25^k\right),$$1 80.5. The paper uses this comparison to argue that geographic text improves depth estimation beyond visual cues alone.

The benchmark is also used to expose an efficiency tradeoff among model families. Pix2Pix is reported as the fastest but least accurate; Marigold is much slower, reportedly about 15× slower than Pix2Pix; HDN is presented as a better compromise among some baselines. In broader methodological terms, RS3DBench is introduced to address five foundational problems in remote-sensing 3D vision: lack of depth supervision, absence of pixel-level RGB-depth alignment, extreme scale variation and tiny objects in aerial imagery, terrain complexity with abrupt elevation transitions, and the need for multimodal or geosemantic reasoning. The paper’s broader conclusion is that remote-sensing 3D understanding cannot simply be transferred from indoor or driving-scene depth estimation, but instead requires geospatially aligned depth data, multi-resolution coverage, terrain diversity, and semantic priors specific to geography.