ThousandWorlds: A benchmark for climate emulation of potentially habitable exoplanets

Abstract: The search for life beyond Earth will depend on detecting faint signatures in the atmospheres of potentially habitable exoplanets. Interpreting those signatures requires understanding the host planet's climate: the same molecule may signal life on one planet and abiotic chemistry on another. Global climate models (GCMs) provide this understanding, but individual runs can require up to millions of core-hours and substantial domain expert time. Machine-learning emulators could remove this bottleneck, but progress has been limited by the absence of a curated, multi-model exoclimate dataset. We introduce ThousandWorlds, an ML-ready benchmark for exoclimate emulation and for the broader regime of low-data, multi-simulator, parameter-to-field regression. The dataset contains approximately 1800 simulations from five GCMs, mapping eight planet parameters to 3D atmospheric fields including temperature, humidity, winds, clouds, and radiation. Three nested subsets define progressively harder challenges: single-simulator regression, multi-simulator regression with complete observations, and multi-simulator regression with structured missingness. We propose two evaluation protocols: one for ranking methods, and one that measures performance relative to the disagreement between GCMs themselves. We evaluate seven baselines spanning simple methods, deep learning, and Gaussian processes. GP-based methods perform best, suggesting that ThousandWorlds exposes a regime where off-the-shelf deep learning does not yet succeed. Data: https://doi.org/10.57967/hf/8695. Code: https://github.com/edstevenson/ThousandWorlds.

• The paper presents a large-scale, multi-GCM exoclimate dataset that enables regression of high-dimensional climate fields across varied planetary parameters.
• It benchmarks seven methods and shows that Gaussian process approaches achieve the lowest RMSE, with relative errors below the inherent GCM disagreement.
• The study highlights the importance of joint representation learning and structured GP regression for improving exoplanet atmospheric characterization.

ThousandWorlds: A Benchmark for Climate Emulation of Habitable Exoplanets

Introduction and Motivation

The determination of exoplanet habitability via atmospheric remote sensing is fundamentally limited by our ability to interpret ambiguous molecular signatures in context. Robust inference of biosignatures such as O₂, CO₂, or CH₄ from transit or direct-imaging spectra relies on precise emulation of planetary climates—particularly for the distinct, tidally locked waterworlds prevalent among nearby rocky exoplanets targeted by missions like JWST. Global Climate Models (GCMs) provide the only direct means for this, yet computational costs prohibit broad parameter sweeps, uncertainty quantification, or systematic integration with exoplanet observation pipelines.

The paper "ThousandWorlds: A benchmark for climate emulation of potentially habitable exoplanets" (2606.18338) addresses this bottleneck through the introduction of a large-scale, multi-model, machine learning-ready exoclimate dataset. It further contextualizes ThousandWorlds as representing an archetypal, underexplored regime in scientific ML: parameter-to-field regression with high dimensional outputs, sparse simulator data, and multiple imperfect simulators.

Dataset Construction and Domain Coverage

Dataset Composition and Structure: ThousandWorlds comprises ∼1800 equilibrium GCM simulations spanning five distinct GCM frameworks (UM, ExoCAM, ExoPlaSim, LFRic, ExoCAM-pre-2022). The simulations map eight planetary parameters—radius, gravity, rotation period, surface pressure, CO₂ and CH₄ volume fractions, incident stellar flux, and host stellar temperature—onto high-dimensional 3D climate fields (Figure 1). Outputs are 53 geophysical fields on a $32 \times 64$ latitude-longitude grid, forming ∼10⁵ target dimensions per example.

Figure 1: A schematic overview of the dataset, showing 53 fields on the common $32\times64$ grid; the full output space is $\sim10^5$ dimensions.

Planet Class and Input Domain: The focus is on synchronously rotating (tidally locked) waterworlds in or near the habitable zone, with physics-constrained ranges for each input. Equatorial symmetry and ocean-world idealization ensure maximal relevance for exoplanet survey pipelines and enable consistent regridding and normalization.

Structured Missingness and Multi-GCM Nature: Cross-GCM heterogeneity leads to structured missingness (incomplete vertical profiles, omitted variables) after all outputs are mapped to a unified pressure grid and horizontally regridded. Inter-GCM epistemic uncertainty and within-GCM configuration noise are directly represented, supporting both single-simulator and multi-simulator transfer setups.

Evaluation Protocols and Tasks

ThousandWorlds defines three nested benchmark tasks:

• Single-complete: Regression using only a single target GCM (UM), without missing fields.
• Multi-complete: Multi-GCM regression, with complete observations (all 48 shared fields).
• Multi-partial: The full setting, with five GCMs and all 53 fields, featuring structured missing data.

Standard benchmarks report area-weighted RMSE over the test set. A key methodological contribution is the shared-planets protocol, which evaluates emulator error relative to the disagreement between high-fidelity GCMs themselves for identical planetary parameters, emphasizing scientific utility and epistemic uncertainty quantification.

Baselines and Methodology

Seven baselines spanning statistical and ML paradigms are systematically compared:

• Simple Methods: Training-mean, kNN.
• Deep Learning: Coord-MLP (pointwise inference), Coord-DeepONet (operator learning), PCA-MLP (PCA compression with MLP regression).
• Gaussian Processes: PPCA-ICM (probabilistic PCA + intrinsic coregionalization multi-GP), GPLFR (end-to-end GP latent factor regression).

All deep architectures incorporate preprocessing tailored to spatial and physical symmetries; spectral representations are supplied for operator-learning approaches. Hyperparameters are tuned per task subset with cross-validation.

Numerical Results and Model Comparison

RMSE Performance

GP-based methods consistently yield the lowest RMSE across nearly all climate variables and benchmark subsets (Table 1), with GPLFR outperforming PCA-MLP and PPCA-ICM in both absolute accuracy and calibration. PCA-MLP is the top performing deep learning baseline; Coord-MLP performs worse except on select variables.

Cross-GCM Transfer and Task Difficulty

Adding auxiliary GCM data benefits GP methods, but is neutral or even harmful for deep learning approaches depending on the output variable, indicating distinctive challenges in multi-fidelity transfer in low-data regimes for standard architectures.

Scientific Utility: Emulator vs. GCM Disagreement

Importantly, under the shared-planets protocol, bolder claims are substantiated: state-of-the-art emulators (GPLFR) achieve relative RMSEs < 1 across most fields and test cases (Figure 2), meaning their error is smaller than the inherent physical uncertainty manifested by the range of high-fidelity GCMs (Table 2).

Figure 2: Per-planet relative RMSE for GPLFR on Multi-partial, grouped by variable; the dashed line denotes the median inter-GCM disagreement threshold.

Despite this, some climate diagnostics, notably global-mean surface temperature, exhibit absolute errors (∼10–11 K) that are still non-negligible for establishing strict habitability bounds, and per-example prediction errors can exceed GCM disagreement in some cases.

Spatial and Physical Diagnostics

The dataset enables refined spatial diagnostics of emergent behavior (Figures 3, 4, 5):

Figure 3: Spatial maps of mid-tropospheric temperature and wind fields for four representative test exoplanets.

Figure 4: Absorbed shortwave radiation (ASR) maps for four test planets, capturing the variable radiative footprint across planetary rotation states and input parameters.

Figure 5: Vertical profiles of area-weighted temperature and specific humidity (dayside and nightside) for four test planets.

These highlight the emulator's capability to capture complex, multi-scale responses to planetary parameters in the thermodynamic, hydrological, and radiative structure of exoplanet climates.

Further, scalar diagnostics such as jet speed, nightside/day-side temperature contrasts, and ice fractions are systematically predicted with low bias (Figure 6).

Figure 6: Predicted versus true values for six key climate diagnostics; points colored by GCM source and the line denotes perfect prediction.

Implications and Future Directions

Benchmark Utility: ThousandWorlds fills a critical gap for the development and testing of ML surrogates under low-data, high-dimensional, multi-simulator constraints—regimes where tabula rasa deep learning is demonstrably suboptimal, but GP-based and hybrid approaches can yield competitive performance.

Methodological Implications: The results underscore the value of joint representation learning and structured Gaussian process regression, and suggest that exploiting physically informed latent spaces, operator priors, and probabilistic calibration is essential in this domain. Methods that learn output compression and regression jointly are more robust as task realism increases.

Practical Relevance: Emulators trained on ThousandWorlds can accelerate exoplanet atmospheric characterization, rapid parameter sweeps for target selection, and evaluation of orbital/stellar trends in habitability, directly enabling global sensitivity analysis and population-scale inference workflows crucial for interpreting current and upcoming transit and direct imaging observations.

Theoretical Impact and Broader Applications: The benchmark architecture and findings apply broadly to multi-fidelity scientific ML settings: parameter-to-field surrogates with structured outputs and simulator epistemic uncertainty are pervasive in weather, climate, astrophysics, and beyond. Techniques successful here are likely transferrable to, and catalyze, progress in adjacent domains.

Outstanding Challenges and Future Work: Enhancing emulator fidelity to reach error levels significantly below GCM epistemic uncertainty—especially at domain boundaries and for challenging quantities such as clouds—warrants further development of hybrid surrogate models, scalable GPs, and transfer strategies. Extending ThousandWorlds to new planetary classes, integrating more diverse radiative/chemistry schemes, or coupling with inversion pipelines presents immediate next steps.

Conclusion

ThousandWorlds (2606.18338) defines a rigorous, extensible benchmark for climate emulation of habitable exoplanets, marked by high-dimensionality, multi-simulator heterogeneity, and limited data. The demonstrated superiority of structured GP methods highlights critical limitations of standard deep learning in scientific ML parameter-to-field surrogates under such conditions, while establishing a new baseline for scientifically relevant predictive accuracy vis-à-vis intrinsic model spread. This dataset and associated benchmarks offer a robust foundation for the evaluation and improvement of future AI-driven exoclimate surrogates.