A Scale-Adaptive Framework for Joint Spatiotemporal Super-Resolution with Diffusion Models

Abstract: Deep-learning video super-resolution has progressed rapidly, but climate applications typically super-resolve (increase resolution) either space or time, and joint spatiotemporal models are often designed for a single pair of super-resolution (SR) factors (upscaling spatial and temporal ratio between the low-resolution sequence and the high-resolution sequence), limiting transfer across spatial resolutions and temporal cadences (frame rates). We present a scale-adaptive framework that reuses the same architecture across factors by decomposing spatiotemporal SR into a deterministic prediction of the conditional mean, with attention, and a residual conditional diffusion model, with an optional mass-conservation (same precipitation amount in inputs and outputs) transform to preserve aggregated totals. Assuming that larger SR factors primarily increase underdetermination (hence required context and residual uncertainty) rather than changing the conditional-mean structure, scale adaptivity is achieved by retuning three factor-dependent hyperparameters before retraining: the diffusion noise schedule amplitude beta (larger for larger factors to increase diversity), the temporal context length L (set to maintain comparable attention horizons across cadences) and optionally a third, the mass-conservation function f (tapered to limit the amplification of extremes for large factors). Demonstrated on reanalysis precipitation over France (Comephore), the same architecture spans super-resolution factors from 1 to 25 in space and 1 to 6 in time, yielding a reusable architecture and tuning recipe for joint spatiotemporal super-resolution across scales.

• The paper introduces a novel scale-adaptive framework that couples a deterministic U-Net with a conditional diffusion module to achieve joint spatiotemporal super-resolution in precipitation fields.
• The methodology decomposes the upscaling task into a smooth mean prediction and a stochastic residual generation while enforcing mass conservation via a dedicated transform.
• The model outperforms conventional methods by reducing pixel-level and high-frequency errors across diverse spatial and temporal scales, offering a systematic recipe for environmental data enhancement.

Scale-Adaptive Joint Spatiotemporal Super-Resolution with Diffusion Models

Introduction and Context

This work introduces a scale-adaptive framework for joint spatiotemporal super-resolution (SR) of precipitation fields using diffusion models. The motivation stems from the challenge in climate and geoscientific data, where conventional deep video super-resolution (VSR) models are tailored for a single spatial and temporal upscaling factor. Such rigidity is prohibitive given the diversity of real-world sources (satellites, radar mosaics, reanalyses) which all exhibit substantial heterogeneity in spatial resolution and temporal cadence. Moreover, precipitation poses a particularly demanding testbed due to its intermittent, heavy-tailed, multiscale nature, requiring models to adequately capture both extremes and fine-scale structure.

The proposed methodology decomposes the upscaling task into two distinct but complementary components: a deterministic prediction of the conditional mean (yielding a smooth backbone for SR) followed by a generative conditional diffusion module tasked with capturing high-frequency and stochastic residuals. Crucially, the architectural core is kept fixed across all spatial and temporal SR factors; only three hyperparameters—diffusion noise schedule amplitude, context length, and a mass-conservation function—are retuned per SR setting. The mass-conservation transform is included to ensure that the physically aggregated precipitation is preserved across resolutions.

This approach demonstrates strong generalizability—the same base model and training procedure is reused across an extensive range of spatial (×1 to ×25) and temporal (×1 to ×6) upscaling factors, refuting the premise that per-resolution manual redesigns are necessary and offering a systematic recipe for scale-adaptive VSR in environmental modeling tasks.

Model Architecture

Deterministic Backbone: U-Net with Spatiotemporal Attention

The deterministic component is a U-Net backbone enhanced with both spatial and temporal self-attention. The input low-resolution (LR) sequence is upscaled with bicubic interpolation and concatenated with auxiliary high-resolution topography, enabling physically informed predictions. Spatiotemporal attention is implemented with multi-head self-attention applied along both spatial neighborhoods (windowed self-attention) and temporal evolutions of each pixel, increasing temporal context as SR factors are increased.

Figure 1: Schematic of the deterministic U-Net encoding multiscale spatiotemporal context for precipitation SR.

Optionally, outputs are post-processed with a mass-conservation transform which scales the predicted high-resolution (HR) field to exactly conserve total precipitation, a critical physical constraint for hydrometeorological interpretability in high-impact applications.

Figure 2: Deterministic U-Net yields a coarse mean field; diffusion head models residuals, with scale-adaptivity achieved by retuning only attention horizon and diffusion noise schedule.

Residual Diffusion Module

Conditioned on the U-Net's mean prediction, a denoising diffusion probabilistic model (DDPM) is trained to generate stochastic fine-scale residuals. The model predicts the velocity (linear blend of clean and noisy residuals) at varying noise levels; inference denoises an initial Gaussian sample through reverse diffusion conditioned on the mean, interpolated LR context, and mass conservation.

Scale-adaptivity in the diffusion stage is accomplished through targeted retuning of the noise schedule. For larger SR factors—where the model is required to generate more HR pixels per LR context, and hence faces greater underdeterminacy—noise amplitude is increased, leading to greater scenario diversity.

Figure 3: Schematic of the super-resolution pipeline: bicubic interpolation feeds an average prediction module, then the diffusion model generates diverse HR scenarios by modeling the stochastic residuals.

Workflow and Pipeline

The complete SR pipeline involves bicubic upscaling, deterministic mean prediction, and scenario generation via the residual diffusion model. Distinct flows are maintained for training (ground truth supervision available) and inference (HR output generated from noise).

Scale-Adaptivity: Hyperparameter Tuning

The only aspects requiring factor-dependent adjustment are the temporal context length $L$, diffusion noise schedule maximum $\beta_{\max}$, and the form/threshold of the mass conservation function $F$. These control (1) input context and receptive field, (2) the model's ability to capture increased stochastic uncertainty as upscaling grows more underdetermined, and (3) output calibration to physically plausible precipitation totals and extremes. All other model, architectural, and loss components are factor-agnostic.

Experimental Setup

Evaluations focus on meteorological precipitation upscaling over metropolitan France using Coméphore radar-gauge reanalysis data (spatial: 1 km, temporal: 1 hour baseline). Multiple super-resolution factors—(1,3), (10,1), (10,3), and (25,6)—are considered to stress-test adaptability. Data covering two full years is partitioned into train and held-out test splits, with 4×4 non-overlapping spatial cross-validation. Outliers are capped based on precipitation’s empirical gamma distribution. All model inputs (precipitation, topography) are min-max normalized.

Results

Qualitative Evaluation

Model outputs illustrate the approach can generate diverse plausible HR sequences for a given LR input, all consistent with both the deterministic mean and underlying physical constraints. Notably, extreme precipitation intensities are better captured via the generative component than any deterministic interpolation baseline. Scenario-wise diversity reflects physically consistent uncertainty.

Figure 4: Qualitative example: LR input/topography, mean prediction, three diffusion scenarios, and ground truth demonstrate high-resolution detail and scenario variability.

Quantitative Benchmarking

Across all tested SR factors, the proposed full architecture achieves superior scores on nearly all climate- and computer-vision-relevant metrics: MSE, MAE, 99th-percentile error (for extremes), log-spectral distance (high-frequency content), earth-moving distance (distributional fidelity), probabilistic calibration (PITD), and CRPS. On some metrics such as SSIM—known to be biased towards overly smooth images—deterministic or interpolation methods occasionally score higher. However, these are not indicative of climatological relevance.

Critically, the diffusion-equipped model reduces both pixel-level and high-frequency spectral error by up to an order of magnitude compared to strong baselines such as EDSR or bicubic interpolation, and is robust under all spatial/temporal upscaling factor settings. The mean- and median-based deterministic U-Net, while computationally efficient, consistently underestimates extremes, echoing known limitations of regression-based SR for heavy-tailed phenomena.

Figure 5: PIT analysis for model calibration—biases in predictive uncertainty are readily identified and corrected via hyperparameter retuning.

Implications and Future Directions

This framework challenges the status quo in applied VSR for physical data, where bespoke model design and costly retraining for each target resolution remain common practice. The demonstrated model is scale-adaptive by design—architecture remains invariant, and only a handful of interpretable, physically motivated hyperparameters are retuned per use-case. This makes the approach systematically extensible to other geoscientific variables and regions, and highly amenable to integration into operational nowcasting and downscaling pipelines.

Practically, the model provides rapid scenario generation (with inference times of minutes per batch on modern hardware), with the caveat that diffusion-based SR is slower relative to purely deterministic approaches. Theoretically, it bridges stochastic generative and physically constrained regression-based modeling, maximizing fidelity to both the distributional and structural characteristics of precipitation while honoring domain-specific conservation laws.

Outstanding research questions include (1) extending full weight sharing across SR factors for a true unified foundation model, (2) investigating transferability to unseen domains, and (3) addressing the reality gap between perfect-model assumptions and noisy downstream products. Efforts to accelerate diffusion inference (e.g., using DDIM, latent diffusion, or progressive distillation) are anticipated to further improve applicability.

Conclusion

This study introduces a scale-adaptive joint spatiotemporal super-resolution framework for environmental data, demonstrated on precipitation. Leveraging a fixed-architecture, attention-augmented U-Net and a conditional residual diffusion generator, the model achieves high accuracy and sharpness across a wide range of upscaling factors, needing only minor hyperparameter retuning. These results advocate for the adoption of scale-adaptive generative models in scientific VSR settings, and highlight future opportunities in truly universal, transferable SR architectures for climate and earth system sciences.