Impact of self-supervised pre-training on generalization and physical consistency

Determine whether self-supervised learning approaches for AI weather and climate models, such as masked autoencoder pre-training, improve both out-of-distribution generalization to gray swan extreme events (e.g., Category 5 tropical cyclones) and physical consistency of forecasts, including balances such as gradient-wind balance.

Background

The authors note emerging AI weather/climate models that use self-supervised learning (e.g., masked autoencoders) without explicit physics or mechanisms to address data imbalance. They identify two key limitations observed in FourCastNet: lack of out-of-distribution generalization to gray swans and lack of physical consistency (e.g., gradient-wind balance).

The explicit unresolved question is whether self-supervised pre-training can mitigate these limitations. Clarifying this would inform the design of foundation models and training strategies for robust extreme-event forecasting and emulation.