Sat-JEPA-Diff: Satellite Forecasting

• The paper proposes a forecasting method that decouples semantic prediction from texture synthesis using a JEPA-style predictor and a frozen Stable Diffusion backbone.
• The method leverages a lightweight cross-attention adapter and fixed Alpha Earth embeddings to preserve geospatial structure and avoid blurry or hallucinated outputs.
• Empirical results on Sentinel-2 data report leading perceptual metrics, including GSSIM 0.8984 and FID 0.1475, outperforming deterministic baselines in structural fidelity.

Sat-JEPA-Diff is a spatiotemporal forecasting framework for satellite imagery that explicitly separates semantic and structural prediction from texture synthesis by combining an IJEPA-style self-supervised predictor with a frozen Stable Diffusion 3.5 latent diffusion backbone and a lightweight cross-attention adapter. It is formulated for next-frame Sentinel-2 forecasting, where an RGB image at time $t$ is used to predict the image at time $t+1$, and it is motivated by the need to preserve geospatial structure while avoiding the blur typical of pixel-regression models and the hallucinated structure typical of unconstrained generative models. On a global Sentinel-2 benchmark, the model reports leading perceptual scores, including GSSIM $0.8984$ and FID $0.1475$ (Komurcu et al., 14 Mar 2026).

1. Problem setting and motivation

The target task is one-step-ahead satellite forecasting: the input is a Sentinel-2 RGB image $I_t \in \mathbb{R}^{3 \times H \times W}$ with $H=W=128$, and the output is a prediction of the next image $I_{t+1}$. The paper frames this as a proxy for virtual sensing when observations are missing or occluded, including cloud-obscured acquisitions, and evaluates the problem on globally distributed regions of interest from 2017 to 2024 (Komurcu et al., 14 Mar 2026).

Sat-JEPA-Diff is motivated by a failure mode on both sides of the standard forecasting divide. Deterministic video or satellite predictors such as ConvLSTM, PredRNN, and SimVP optimize pixel-wise $\ell_1$ or $\ell_2$/MSE losses and therefore gravitate toward the conditional mean of plausible futures. In this setting, ambiguity in cloud motion, land-cover change, or other multimodal futures suppresses high-frequency detail, yielding regression to the mean, blurry outputs, and weak preservation of edges, thin roads, and small parcels. The reported experiments confirm that PredRNN and SimVP obtain strong PSNR and SSIM while remaining weak on FID and GSSIM. By contrast, generative models such as GANs or latent diffusion models can synthesize sharp textures, but may hallucinate roads, buildings, or river courses in incorrect locations. For remote sensing, this is not merely a visual defect: the paper explicitly identifies hallucinated geospatial structure as a risk for downstream land-use mapping, deforestation tracking, and disaster assessment (Komurcu et al., 14 Mar 2026).

The framework therefore adopts a strict division of labor. Semantic prediction is performed in a latent space aligned with fixed Alpha Earth Foundation Model embeddings, while appearance refinement is delegated to a pretrained generative model. This design is intended to make structure stable first and texture realistic second.

2. System architecture and division of labor

At the system level, Sat-JEPA-Diff consists of three coupled modules: an IJEPA-style semantic predictor, a conditioning adapter, and a frozen Stable Diffusion 3.5 Medium backbone with LoRA adapters. The architecture predicts future semantic tokens from the current frame, fuses those tokens with a coarse RGB reference, and conditions the diffusion model so that fine-grained appearance is synthesized under explicit structural guidance rather than from weak image conditioning alone (Komurcu et al., 14 Mar 2026).

Component	Main function	Key details
IJEPA module	Forecast future semantic structure	ViT-based encoder and predictor aligned to Alpha Earth embeddings
Cross-attention adapter	Convert semantics and coarse RGB into diffusion conditioning	Outputs token-level conditioning $h$ and global conditioning $t+1$0
Stable Diffusion 3.5 backbone	Synthesize texture and appearance	Frozen core transformer, LoRA adapters, VAE with $t+1$1 downsampling

The pipeline is organized as follows. The current RGB frame is encoded into latent tokens by the JEPA encoder, and a predictor transforms those tokens into a forecast of future semantic tokens. An adapter then combines the predicted future tokens with a $t+1$2 downsample of the current RGB frame and produces token-level conditioning $t+1$3 and a global conditioning vector $t+1$4. During training, Stable Diffusion 3.5 receives VAE latents of the ground-truth future frame together with noise and the conditioning pair $t+1$5, and learns a flow-matching denoising objective. During inference, the model predicts the future semantic tokens from $t+1$6, conditions the frozen diffusion backbone, and samples $t+1$7.

The paper summarizes the intended specialization as “IJEPA: where and what” and “Diffusion: how it looks.” This is the defining conceptual feature of Sat-JEPA-Diff: structure is not expected to emerge implicitly from texture synthesis, but is forecast explicitly before generation.

3. Semantic forecasting with IJEPA and Alpha Earth supervision

The semantic backbone is an Instantiated Joint-Embedding Predictive Architecture adapted to forecast Alpha Earth Foundation Model embeddings. The encoder is ViT-based, takes $t+1$8 RGB inputs with patch size $t+1$9, and therefore produces $0.8984$0 patch tokens with hidden dimension $0.8984$1. A 6-layer transformer predictor maps the current-frame tokens to predicted future tokens, and a linear projection maps those 768-dimensional outputs to the 64-dimensional Alpha Earth embedding space. The target encoder is an EMA copy of the encoder, with momentum scheduled from $0.8984$2 to $0.8984$3, and the Alpha Earth embeddings themselves remain fixed (Komurcu et al., 14 Mar 2026).

Supervision is not performed in pixel space. Instead, the predictor is trained against pre-computed Alpha Earth embeddings for the future frame, which makes the JEPA module a predictor of foundation-model semantics. Its objective is hybrid: it combines per-token latent-space $0.8984$4 reconstruction, cosine similarity, a spatial-variance regularizer to prevent spatial collapse, a contrastive InfoNCE-style loss with temperature $0.8984$5, and an additional feature regression term on Alpha Earth features. The reported weights are: $0.8984$6 weight $0.8984$7, cosine $0.8984$8, spatial variance $0.8984$9, contrastive $0.1475$0, and additional feature regression weight $0.1475$1. The masking strategy follows IJEPA multi-block masking in encoder and predictor views, and the paper states that this forces inference of missing regions from visible ones.

The role of the hybrid objective is made explicit by the ablation study. Removing spatial variance or contrastive terms slows alignment to target embeddings and can produce near-zero-variance representations; the full loss maintains higher embedding variance and higher cosine similarity. The paper interprets this as evidence that semantic prediction in a 64-dimensional embedding space is more stable than pixel prediction because it emphasizes land-cover semantics and suppresses sensor-level variability. This suggests that Sat-JEPA-Diff’s structural consistency depends as much on the embedding target and anti-collapse design as on the generative backbone.

4. Diffusion backbone, conditioning adapter, and optimization

The generative component is Stable Diffusion 3.5 Medium used as a latent diffusion or rectified flow transformer. Its core transformer is frozen, except for LoRA adapters on attention layers, and the latent encoder-decoder is a VAE with $0.1475$2 spatial downsampling. For $0.1475$3 inputs this yields latent tensors with $0.1475$4 and $0.1475$5. Domain adaptation is deliberately lightweight: the backbone is not fully retrained on satellite data, and the paper attributes satellite specialization primarily to the learned conditioning adapter and the LoRA modules (Komurcu et al., 14 Mar 2026).

A fixed text prompt is used for all samples: “High-resolution Sentinel-2 satellite image, multispectral earth observation, natural colors RGB composite, 10m ground resolution, clear atmospheric conditions, detailed land surface features.” Sentinel-2 RGB surface reflectance is harmonized to 10 m GSD and normalized to $0.1475$6. The diffusion training objective follows rectified flow or flow matching rather than a standard DDPM parameterization. The model samples Gaussian noise, interpolates between clean latent and noise, and learns a velocity field conditioned on $0.1475$7, together with an SSIM-based regularizer weighted by $0.1475$8.

The conditioning adapter is the bridge between predicted semantics and the diffusion backbone. It receives predicted IJEPA tokens and a coarse $0.1475$9 RGB version of the current frame. The semantic stream is projected with an MLP $I_t \in \mathbb{R}^{3 \times H \times W}$0, while the coarse RGB stream is patchified into 64 tokens and projected by an MLP $I_t \in \mathbb{R}^{3 \times H \times W}$1. Learnable positional embeddings are added to both streams. A learnable sigmoid gate $I_t \in \mathbb{R}^{3 \times H \times W}$2 mixes the semantic and coarse streams, and the reported training outcome is $I_t \in \mathbb{R}^{3 \times H \times W}$3, suggesting equal reliance on semantic and coarse structural cues. Mean-pooled IJEPA tokens are further mapped by a two-layer MLP $I_t \in \mathbb{R}^{3 \times H \times W}$4 to provide global conditioning. The adapter has approximately 25M parameters.

Training is joint across the IJEPA encoder and predictor, the conditioning adapter, and LoRA parameters inside SD3.5. Optimization uses AdamW with base learning rate $I_t \in \mathbb{R}^{3 \times H \times W}$5, warmup from $I_t \in \mathbb{R}^{3 \times H \times W}$6, cosine decay to $I_t \in \mathbb{R}^{3 \times H \times W}$7, and a weight decay schedule from $I_t \in \mathbb{R}^{3 \times H \times W}$8 to $I_t \in \mathbb{R}^{3 \times H \times W}$9. Mixed precision uses bfloat16 for most modules and float32 for the VAE. Batch size is 8 per GPU. A coarse RGB reference dropout of $H=W=128$0 is applied so the model can fall back on semantic embeddings alone. At inference, sampling uses a Flow Matching Euler-Discrete scheduler with noise strength $H=W=128$1 and single-step speed estimation.

5. Dataset, baselines, and empirical performance

The empirical study uses Sentinel-2 Surface Reflectance RGB imagery together with 64-dimensional Alpha Earth embeddings per pixel. The dataset spans 2017–2024 and covers 100 global regions of interest, including urban, agricultural, forest, desert, and deltaic environments. The split is a random shuffle with 80% training and 20% validation; the paper notes that a separate test set is implied by Table 1 even though exact counts are not given. The main task is one-step-ahead prediction, but the appendix also evaluates autoregressive rollouts over longer horizons (Komurcu et al., 14 Mar 2026).

The baseline set includes deterministic methods—Default, PredRNN, and SimVP v2—and generative methods—Stable Diffusion 3.5 with generic conditioning and MCVD. A variant of Sat-JEPA-Diff replaces the ViT encoder with the Panopticon foundation model encoder to test encoder generality. Evaluation uses L1, MSE, PSNR, SSIM, GSSIM, LPIPS, and FID, with GSSIM and FID emphasized as structural and perceptual metrics beyond PSNR or MSE.

On the test set, the main Sat-JEPA-Diff model reports L1 $H=W=128$2, MSE $H=W=128$3, PSNR $H=W=128$4, SSIM $H=W=128$5, GSSIM $H=W=128$6, LPIPS $H=W=128$7, and FID $H=W=128$8. PredRNN achieves stronger PSNR and SSIM—PSNR $H=W=128$9, SSIM $I_{t+1}$0—but much weaker GSSIM $I_{t+1}$1 and FID $I_{t+1}$2. SimVP v2 follows the same pattern, with PSNR $I_{t+1}$3 and SSIM $I_{t+1}$4 but GSSIM $I_{t+1}$5 and FID $I_{t+1}$6. Stable Diffusion 3.5 alone reaches GSSIM $I_{t+1}$7 and FID $I_{t+1}$8 but SSIM falls to $I_{t+1}$9, consistent with the claim that texture sharpness without strong semantic conditioning is insufficient. The Panopticon variant reaches GSSIM $\ell_1$0 and FID $\ell_1$1, which the paper interprets as evidence that the framework is not tied to a single encoder family.

The headline result is the combination of high GSSIM and low FID with competitive distortion metrics. The paper states an 11% relative improvement in GSSIM over the best baseline, while also noting that SSIM and PSNR remain below deterministic predictors, consistent with the perception–distortion trade-off discussed in the text. Qualitative examples on Istanbul, the Amazon, the Corn Belt, the Sahara, and the Mekong Delta show sharper street networks, parcel boundaries, coastlines, and river shapes than deterministic baselines. In autoregressive rollouts over the Rio de Janeiro coastline from 2018 to 2024, PredRNN and SimVP are reported to collapse into blurred scenes after two to three steps, whereas Sat-JEPA-Diff maintains contrast and sharper structure over the seven-year horizon, albeit still under standard autoregressive error accumulation.

6. Interpretation, limitations, and broader JEPA-diffusion context

The paper’s central claim is that Sat-JEPA-Diff works by decoupling structure from texture. The deterministic component predicts what should be present and where in a semantic latent space aligned with Alpha Earth features, while the generative component adds high-frequency appearance under explicit conditioning. This design directly targets two remote-sensing failure modes: mean-regression blur and generative hallucination. The paper nevertheless states several limitations: the full system is heavier than typical deterministic forecasters; long-horizon rollouts still accumulate error; training and evaluation are restricted to Sentinel-2 RGB rather than other sensors or spectral bands; and hallucination risk is reduced but not eliminated, especially under severe distribution shift (Komurcu et al., 14 Mar 2026).

These limits define the practical scope of the method. The paper lists continuous land-cover monitoring, change detection, temporal gap filling under cloud cover, disaster impact mapping, and scenario exploration as candidate applications, but also emphasizes that forecasted imagery should not be conflated with actual observations in critical decision-making. The public release of code and dataset at https://github.com/VU-AIML/SAT-JEPA-DIFF is presented as a reproducibility measure rather than as a claim of deployment readiness.

Within the broader JEPA literature, Sat-JEPA-Diff belongs to a family of methods that shift prediction from input space to latent or semantic space. “Denoising with a Joint-Embedding Predictive Architecture” explicitly turns JEPA into a generative model by combining masked latent prediction with per-token diffusion or flow matching, but does so as an autoregressive continuous-token generator rather than as a remote-sensing forecaster conditioned by foundation-model semantics (Chen et al., 2024). “A Generalization Theory for JEPA-Based World Models” formalizes JEPA pretraining as low-rank factorization of an action-conditioned co-occurrence matrix and derives an approximation–sample error trade-off in the latent dimension, which is relevant to any claim that latent predictive models can outperform input-level prediction when nuisance variation is high (Cui et al., 25 Jun 2026). Other 2026 JEPA work explores structured latent geometry more directly: “Subspace-Decomposed JEPAs” separates progression and content into orthogonal subspaces (Thil et al., 29 May 2026), and “Beyond Isotropy in JEPAs” argues that structural bias should enter the cross-view coupling rather than a fixed isotropic marginal (Alvarez, 19 May 2026). Sat-JEPA-Diff does not instantiate those formulations, but it is congruent with the same general move: semantics are predicted in a constrained latent space, while appearance is delegated to a separate generative mechanism.