Atmos-Bench: Standardized Atmospheric Science Benchmarks

• Atmos-Bench is a comprehensive framework of standardized benchmarks and datasets for evaluating machine learning algorithms in atmospheric science and climate modeling.
• It utilizes physics-based simulation, high-resolution reanalysis, and lidar radiative transfer to construct large 3D volumetric datasets for atmospheric structure recovery.
• The framework rigorously compares retrieval models using key metrics such as PSNR, SSIM, and MAE, enabling robust evaluations across different wavelengths.

Atmos-Bench encompasses standardized benchmarks and datasets for atmospheric science and climate modeling, with the term referencing several distinct, rigorously designed resources. These benchmarks support evaluation, comparison, and development of machine learning algorithms and climate models, particularly for remote sensing and exoplanetary studies. The most prominent usage now refers to the large-scale 3D benchmark for recovering atmospheric structure from satellite lidar, but earlier and domain-specific usages exist for global circulation model intercomparison and synthetic atmosphere grids.

1. Foundational Definition and Variants

The term "Atmos-Bench" can refer to:

• A 3D benchmark of atmospheric volume structure recovery from satellite lidar, supporting machine learning for remote sensing and climate applications (Xu, 15 Jul 2025).
• Intercomparison protocols for global climate models applied to exoplanet or shallow-atmosphere scenarios (Bending et al., 2012).
• Grids of synthetic, 1D planetary atmospheres for exoplanet habitability screening and machine learning (Chopra et al., 2023).

While further domains (e.g., atmospheric emission modeling (Errard et al., 2015)) recommend the development of "Atmos-Bench" modules, the canonical modern reference is the dataset and pipeline for 3D atmospheric structure from simulated lidar, as introduced by Zhang et al. (Xu, 15 Jul 2025).

2. 3D Atmospheric Structure Benchmark: Dataset Construction

Atmos-Bench (Xu, 15 Jul 2025) is constructed via physics-based simulation, with a focus on volumetric atmospheric backscatter coefficient (BC) recovery from synthetic satellite lidar attenuated backscatter (ATB):

1. Reanalysis and Mesoscale Modeling:
   • NCEP FNL reanalysis supplies initial and boundary conditions (1° spatial, 6 h temporal, 33 vertical levels to 20 km).
   • High-resolution WRF regional simulations (18 km grid, 44 vertical levels, 0–20 km, domain: 104.5–135.5°E, 4.9–34.9°N) with advanced physical parameterizations (double-moment microphysics, RRTM longwave, cumulus, PBL).
2. Lidar Radiative Transfer Simulation:
   • Enhanced COSP v2 simulator generates both ATB at 355 nm and 532 nm (scatter+absorption physics) and intrinsic, extinction-free BC fields (via COSP-ZT branch).
3. Orbital Re-gridding and Slice Extraction:
   • Volumes are realigned to an orbital-aligned grid: 600 (zonal) × 200 (vertical, 100 m resolution) voxels, approximating the EarthCARE/CALIOP profile.
   • 921,600 paired ATB/BC slices (200×600 pixels), for each wavelength and time, are extracted—establishing a volumetric dataset of 46,080,000 voxels per wavelength.

The following radiative transfer equation underpins the coupling of ATB and BC fields:

$$\mathrm{BC}(x, \lambda) = \frac{Pr(x, \lambda)}{C\,\beta(x, \lambda)} \cdot \exp\left(-2 \int_0^x \alpha(s, \lambda) ds\right)$$

where $Pr$ is received lidar power, $\beta$ is the intrinsic backscatter coefficient, $\alpha$ the extinction, and $C$ an instrument constant.

3. Model Architecture and Recovery Methodology

The benchmark supports evaluation of methods that map observed ATB to true, physically consistent BC. The leading baseline is FourCastX—a frequency-enhanced Spatio-Temporal Mixture-of-Experts (MoE) architecture, integrating:

• Encoder-decoder U-net backbone with Fast Fourier Convolution (FFC) layers to capture global and local patterns.
• Three-stage gated MoE modules: encode spatial/temporal hybridization, with dynamic gating between FFC and ConvLSTM or cross-attention experts.
• Physics-informed loss function: energy-consistency regularization between ATB and predicted BC, using a differentiable forward lidar operator with extinction profile $\alpha(x, \lambda)$.
• Composite objective: combines evidential regression ($\mathcal{L}_{ev}$), adversarial ($\mathcal{L}_{adv}$), high-resolution perceptual ($\mathcal{L}_{HRF}$), feature-matching ($\mathcal{L}_{FM}$), and $Pr$0 reconstruction losses.

This methodology strictly enforces energy conservation and radiative transfer constraints throughout training. Baselines include CAT, DDS2M, EchoIR, TSFormer, UIR-LoRA, and VmambaIR architectures.

4. Quantitative Evaluation and Benchmarking Results

Benchmarked models are evaluated on standard image reconstruction and perceptual metrics. Table 1 summarizes key quantitative results for 532 nm and 355 nm ATB/BC slice recovery.

Method	532 nm PSNR↑	532 nm SSIM↑	532 nm MAE↓	355 nm PSNR↑	355 nm SSIM↑	355 nm MAE↓
FourCastX	23.38	0.969	0.0080	23.94	0.970	0.0060
CAT	17.04	0.807	0.0468	12.72	0.492	0.0893
DDS2M	12.47	0.808	0.0749	13.94	0.847	0.0569
EchoIR	18.27	0.902	0.0348	19.13	0.924	0.0290
TSFormer	18.78	0.877	0.0287	20.81	0.927	0.0217
UIR-LoRA	18.94	0.890	0.0325	20.01	0.924	0.0250
VmambaIR	16.75	0.864	0.0468	17.92	0.899	0.0344

FourCastX exceeds all baselines by >6 dB in PSNR and 70–92% MAE reduction. Qualitative assessments confirm that FourCastX best preserves sharp volumetric structure and vertical cloud filaments under varying signal masking, while others are susceptible to speckle, smoothing, or artifact generation. All methods use common ATB inputs to ensure comparability.

5. Protocols, Reproducibility, and Dataset Utilization

• Data Split: 384 volumes are divided as 90% training, 10% test per wavelength.
• Access: All code and data are publicly available, with standardized evaluation scripts, facilitating replication.
• Protocol: Common input preprocessing (ATB normalization, voxel alignment), a unified evaluation suite, and defined metrics provide robust, fair comparison.
• Extensibility: Future plans include semi-supervised fine-tuning on real CALIOP/EarthCARE observations, multi-sensor (e.g., SAR, radiometer) fusion, and additional wavelengths.

Atmos-Bench is explicitly designed for ML practitioners and geoscientists seeking volumetric ground truth for end-to-end benchmarking and retrieval algorithm calibration.

6. Exoplanet and Classical Model Intercomparison Benchmarks

Exoplanetary Atmos-Bench (Chopra et al., 2023) refers to the PyATMOS dataset: a scalable grid (124,314 steady-state, 1-D vertical column models) of Earth-like planetary atmospheres spanning a 6D grid of major gas mixing ratios (O₂, CO₂, H₂O, CH₄, H₂, N₂). These atmospheres provide T(z), P(z), and chemical profiles for ML and theoretical habitability studies, with outputs in CSV, HDF5, and netCDF, accessible via the NASA Exoplanet Archive.

Shallow-Atmosphere GCM Bench (Bending et al., 2012) documents reproducible primitive-equation hot-Jupiter model intercomparison protocols. It specifies all dynamical and thermodynamical parameters, grid configurations, hyperdiffusion, forcing, and diagnostics (e.g., superrotation index, eddy kinetic spectra), providing a reference for shallow-atmosphere GCM benchmarking.

7. Scientific Impact, Limitations, and Future Directions

Atmos-Bench establishes a new standard for the physically consistent, scalable, and reproducible evaluation of atmospheric structure recovery algorithms. Key applications include:

• Satellite remote sensing: benchmarking retrieval of 3D cloud/aerosol structure from lidar.
• Climate modeling: radiative flux quantification, cloud–radiative feedback studies, and initialization of extreme weather model states.
• Machine learning: training, calibration, and uncertainty quantification for generative, invertible, and hybrid retrieval models in atmospheric science.

Principal limitations are related to the sim-to-real gap (entirely simulated data), lack of multi-sensor fusion (as of the original version), and fixed physical parameterizations. A plausible implication is that extension to semi-supervised and domain adaptation methods will be crucial for operational deployment, as will expansion to real-world orbiter footprints and atmospheric regimes.

Atmos-Bench thus provides a physically grounded foundation and benchmark protocol for state-of-the-art volumetric recovery, retrieval, and broader atmospheric model intercomparison (Xu, 15 Jul 2025, Chopra et al., 2023, Bending et al., 2012).