WeatherPEFT: Adaptive Fine-Tuning for WFMs

• WeatherPEFT is a parameter-efficient fine-tuning framework that utilizes Task-Adaptive Dynamic Prompting and Stochastic Fisher-Guided Adaptive Selection to tailor Weather Foundation Models.
• It addresses the heterogeneous nature of weather tasks by adapting to variable interactions, resolution changes, and spatiotemporal variations for near full-tuning performance.
• Empirical evaluations on global downscaling, ensemble forecast post-processing, and regional precipitation forecasting demonstrate robust performance with a fraction of the trainable parameters.

WeatherPEFT is a parameter-efficient fine-tuning framework for Weather Foundation Models (WFMs) that couples Task-Adaptive Dynamic Prompting (TADP) in the forward pass with Stochastic Fisher-Guided Adaptive Selection (SFAS) during backpropagation. It was introduced to address a specific mismatch between conventional PEFT methods and weather downstream tasks: weather applications involve variable heterogeneity, resolution diversity, and spatiotemporal coverage variations that are not well handled by uniform adaptation schemes derived from computer vision or natural language processing. In the reported experiments, WeatherPEFT was evaluated on global downscaling, ensemble forecast post-processing, and regional precipitation forecasting, where it was described as achieving performance parity with Full-Tuning using fewer trainable parameters (Cao et al., 26 Sep 2025).

1. Problem setting and rationale

WeatherPEFT is motivated by the observation that weather downstream tasks are unusually heterogeneous. Inputs are multivariate gridded atmospheric states with tens to hundreds of physical variables, including combinations such as surface variables and upper-air fields across pressure levels. Different tasks require different subsets of variables and interactions. The same pretrained WFM may therefore be adapted to tasks that differ not only in target variables, but also in spatial resolution, regional versus global coverage, lead times, accumulation windows, and statistical structure; precipitation, for example, is described as sparse and heavy-tailed (Cao et al., 26 Sep 2025).

This heterogeneity creates a failure mode for conventional PEFT. Methods such as LoRA, DoRA, AdaptFormer, SSF, VPT, APrompt, and BitFit are characterized in the WeatherPEFT study as reusing essentially the same trainable parameter subset across tasks, or as injecting prompts/adapters without encoding task-specific meteorological structure. In weather applications, however, parameter importance is task-dependent: parameters that matter for regional precipitation forecasting may differ from those needed for global downscaling. The paper therefore positions WeatherPEFT as a task-adaptive alternative in which forward conditioning and backward parameter selection are explicitly matched to the meteorological characteristics of the task (Cao et al., 26 Sep 2025).

At a high level, the framework has two coordinated components. TADP conditions the frozen backbone on task-relevant patterns extracted from the encoder’s embedding weights and injects soft prompts before each block. SFAS then estimates parameter importance using Fisher information and updates only the Top-$k$ task-critical parameters, with an annealed stochastic term introduced to stabilize selection. The paper’s central claim is that this coordinated forward and backward adaptation narrows the gap between PEFT and Full-Tuning under weather-specific task shifts (Cao et al., 26 Sep 2025).

2. Task-Adaptive Dynamic Prompting

TADP operates on the encoder’s patch embedding weights

$$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$

where $D$ is the hidden dimension, $V$ is the number of variables, and $P_h \times P_w$ is the spatial kernel size of the embedding window. After flattening the spatial dimensions, WeatherPEFT applies three progressive adapters, each consisting of LayerNorm, down-projection, GELU, and up-projection. These adapters are organized hierarchically: the HW-Adapter learns localized spatial-resolution patterns on $P_h \times P_w$, the V-Adapter models interdependencies among physical variables given the spatial context, and the D-Adapter captures abstract meteorological characteristics in the hidden dimension (Cao et al., 26 Sep 2025).

The internally extracted representation is then passed to an external pattern integration stage. The first two dimensions are merged, self-attention is applied to couple physical quantities with spatial-resolution features, and an MLP produces the final soft prompt tokens

$E_p \in \mathbb{R}^{P \times D},$

where $P$ is the prompt length. These prompt tokens are concatenated with the encoded input token sequence

$T \in \mathbb{R}^{M \times D}$

to form $[E_p; T]$, and this concatenated sequence is injected before each block of the pretrained backbone. In the Aurora instantiation discussed in the paper, this means injection at each 3D Swin U-Net block; more generally, the design is described as per-block prompt injection throughout the encoder and decoder stack (Cao et al., 26 Sep 2025).

The significance of this design is that prompt generation is not based on a generic prompt table or on manually specified task text. Instead, the prompts are extracted from encoder embedding weights and are intended to encode variable couplings, resolution-dependent structure, and task-specific meteorological context. The paper further notes that there are no custom attention score modifications beyond the described self-attention and MLP stages; recalibration emerges from prompt-conditioned attention in the pretrained backbone (Cao et al., 26 Sep 2025).

The reported prompt budgets are task-specific. Table 18, as summarized in the paper details, gives representative values of $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$0 for downscaling, $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$1 for post-processing, and $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$2 for precipitation forecasting. The hidden dimension is given as $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$3, the patch window as $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$4, and compact adapter dimensions such as $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$5, $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$6–$$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$7, $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$8, and an additional $$E \in \mathbb{R}^{D \times V \times P_h \times P_w},$$9 adapter with $D$0 are described as effective. The paper reports that increasing $D$1 beyond approximately $D$2 yields no gains and can degrade performance, while compact adapter dimensions are more effective than large ones (Cao et al., 26 Sep 2025).

3. Stochastic Fisher-Guided Adaptive Selection

SFAS governs which parameters are allowed to update during fine-tuning. The framework uses Fisher information as a sensitivity measure for the predictive distribution $D$3 with respect to the task loss. Because the full Fisher matrix is intractable at WFM scale, WeatherPEFT uses a diagonal approximation and, in practice, a supervised empirical approximation based on ground-truth labels. Larger Fisher values are interpreted as indicating greater parameter importance for the current downstream task (Cao et al., 26 Sep 2025).

Selection is performed at each training step. After computing the empirical Fisher scores, WeatherPEFT adds an annealed stochastic component:

$D$4

where $D$5 is the initial linear decay factor, $D$6 and $D$7 are current and total training steps, and $D$8 is a stochastic vector of the same shape as the Fisher estimate. The paper notes that the printed equation omits “$D$9” but that the textual description clarifies annealing over steps. The Top-$V$0 entries in this stochastic Fisher vector are then selected globally to form a binary Fish Mask. Conceptually, the optimizer applies the mask to gradients so that only selected parameters are updated, while all other parameters remain frozen (Cao et al., 26 Sep 2025).

This mechanism is presented as a response to a second weather-specific problem: not only are tasks heterogeneous, but the subset of parameters that matters can differ substantially across them. In that sense, SFAS contrasts with uniform PEFT, where the same structural update pattern is used regardless of whether the task is multivariate global downscaling, probabilistic post-processing, or sparse regional precipitation forecasting. The paper also reports that annealed randomness is important in early training because raw Fisher scores can be noisy and may mis-prioritize parameters if used as a fixed hard mask from initialization (Cao et al., 26 Sep 2025).

The sensitivity study summarized in the paper gives three practical conclusions. First, performance improves as the selection ratio $V$1 increases from $V$2 toward approximately $V$3–$V$4, after which gains plateau. Second, the stochastic factor $V$5 is often optimal or co-optimal. Third, on sparse precipitation signals, SFAS is reported to be more critical than prompting, although TADP still contributes. These observations are important because they identify WeatherPEFT not merely as prompt tuning, but as a joint prompt-and-selection method (Cao et al., 26 Sep 2025).

4. Training regime, tasks, and quantitative performance

WeatherPEFT was evaluated on three downstream tasks. The global downscaling task used WeatherBench and ERA5 with 68 variables, downscaling from $V$6 to $V$7, trained on 2007–2016 and tested on 2017–2018 at 6-hour resolution. Ensemble forecast post-processing used ENS-10, with a 10-member ECMWF IFS ensemble, 48-hour lead, ERA5 targets at $V$8, and 25 variables, trained on 1998–2015 and tested on 2016–2017. Regional precipitation forecasting used ERA5-CH over China at $V$9, with 70 variables, 12-hour resolution, training on 2010–2019 and testing on 2020; targets were 6-hour accumulated total precipitation at 12h, 24h, and 36h leads (Cao et al., 26 Sep 2025).

The task losses were task-specific. Downscaling used mean square error, with surface loss weighted $P_h \times P_w$0 and upper-air loss weighted $P_h \times P_w$1. Post-processing used Continuous Ranked Probability Score (CRPS), together with the extreme-event weighted variant EECRPS using EFI. Regional precipitation forecasting used mean absolute error. Optimization used AdamW with weight decay $P_h \times P_w$2 and a cosine learning-rate schedule with 3-epoch warmup. The reported task-specific settings were: downscaling with learning rate $P_h \times P_w$3, batch size $P_h \times P_w$4, and $P_h \times P_w$5 epochs; post-processing with learning rate $P_h \times P_w$6, batch size $P_h \times P_w$7, and $P_h \times P_w$8 epochs; precipitation with learning rate $P_h \times P_w$9, batch size $P_h \times P_w$0, and $P_h \times P_w$1 epochs. Training was conducted on $P_h \times P_w$2 NVIDIA A800 80GB GPUs, with approximate training times of about 6 hours for downscaling and about 2 hours each for post-processing and precipitation (Cao et al., 26 Sep 2025).

Task	WeatherPEFT setting	Representative result
Global downscaling	3.48M trainable params on Aurora	T2m RMSE 1.119; U10 1.057; V10 1.051; T850 0.950; Z500 44.922
ENS-10 post-processing	3.18M trainable params on Aurora	Z500 CRPS 72.701 versus 73.760 for Full-Tuning
Regional precipitation	3.38M trainable params on Aurora	Gap to Full-Tuning substantially narrowed; with 52.37M params, 12h SEEPS 0.302 versus 0.304 for Full-Tuning
Backbone transfer	81.99M trainable params on Prithvi-WxC 2.3B	Matches Full-Tuning performance within approximately 4% parameter budget

In global downscaling on Aurora, Full-Tuning used 1239.94M trainable backbone parameters. WeatherPEFT with 3.48M parameters achieved T2m RMSE 1.119, U10 1.057, V10 1.051, T850 0.950, and Z500 44.922, with mean bias close to 0 for all. The best conventional PEFT baseline reported in the same setting, SSF with 3.92M parameters, achieved T2m RMSE 1.180 and Z500 RMSE 48.342. When the parameter budget was increased to approximately 4% of the backbone, WeatherPEFT at 52.47M parameters achieved T2m RMSE 0.916 and Z500 35.076, which the paper characterizes as near Full-Tuning, whose Z500 value was 35.821 (Cao et al., 26 Sep 2025).

In ENS-10 post-processing, WeatherPEFT with 3.18M parameters matched or slightly improved Full-Tuning on some variables; one cited example is Z500 CRPS 72.701 for WeatherPEFT versus 73.760 for Full-Tuning. At about a 4% parameter budget, WeatherPEFT at 52.18M parameters surpassed Full-Tuning in several metrics. In regional precipitation forecasting, the paper reports that conventional PEFT methods exhibited large gaps relative to Full-Tuning, including a LoRA result at 12h with SEEPS +62.8% versus Full-Tuning. WeatherPEFT with 3.38M parameters significantly narrowed this gap, and at 52.37M parameters it outperformed Full-Tuning across all reported metrics, including 12h SEEPS 0.302 versus 0.304, ACC 0.805 versus 0.797, and RMSE 0.174 versus 0.178 (Cao et al., 26 Sep 2025).

The same study also reports backbone transfer. On Prithvi-WxC 2.3B, WeatherPEFT with 81.99M trainable parameters matched Full-Tuning performance within an approximately 4% parameter budget, while LoRA performed poorly on that architecture. In a real-world extreme case involving the 2020 China Mei-yu flood, WeatherPEFT at approximately 4% budget achieved Threat Score comparable to Full-Tuning and better than LoRA, DoRA, and AdaptFormer (Cao et al., 26 Sep 2025).

5. Ablations, hyperparameter behavior, and implementation practice

The WeatherPEFT paper reports consistent synergy between TADP and SFAS. In downscaling ablations, TADP-only and SFAS-only both improved over conventional baselines, but the combined configuration yielded the best results. The same section reports that removing Internal or External pattern extraction from TADP degrades RMSE; for example, T2m RMSE rose to 1.140 or 1.130 versus 1.119 for the full method. Removing the stochastic term from SFAS also increased RMSE, which the authors interpret as confirming the stabilizing role of annealed randomness (Cao et al., 26 Sep 2025).

The hyperparameter analysis provides several operational heuristics. The selection ratio $P_h \times P_w$3 governs the trade-off between efficiency and Full-Tuning parity. Main experiments used $P_h \times P_w$4 for minimal budget and fair comparison with conventional PEFT, corresponding to approximately 0.3% parameters. Increasing $P_h \times P_w$5 toward $P_h \times P_w$6–$P_h \times P_w$7 yields parity with or superiority to Full-Tuning across tasks, after which performance plateaus. Prompt length $P_h \times P_w$8 must remain small and task-dependent; the summary recommends smaller $P_h \times P_w$9 for post-processing, approximately $E_p \in \mathbb{R}^{P \times D},$0 for precipitation, and larger values such as approximately $E_p \in \mathbb{R}^{P \times D},$1 for downscaling, but notes that excessive prompt length can hurt optimization (Cao et al., 26 Sep 2025).

The training procedure is explicitly batch-wise. In the forward pass, soft prompts are computed from frozen encoder embedding weights, concatenated with input tokens, and injected before each block. The backbone then produces outputs and the task-specific loss is computed. In the backward pass, the loss gradient is used to estimate the empirical diagonal Fisher, annealed randomness is added, the Top-$E_p \in \mathbb{R}^{P \times D},$2 Fish Mask is constructed, and the mask is applied so that AdamW updates only the selected parameters. During inference, TADP remains active, but SFAS masks are not needed (Cao et al., 26 Sep 2025).

A practical implication is that WeatherPEFT should be understood as a training-time sparse-update scheme combined with an inference-time prompt-conditioning scheme. It is therefore distinct from PEFT methods that rely solely on low-rank updates, solely on adapters, or solely on prompt tuning. The paper also states that the method was validated on transformer-based backbones, particularly Aurora and Prithvi-WxC, but that it is adaptable to CNNs and GNNs by targeting their embedding spaces and per-layer input concatenations (Cao et al., 26 Sep 2025).

6. Relation to other PEFT methods and terminological extensions

Within the WFM setting, WeatherPEFT is positioned against several PEFT families. LoRA and DoRA are treated as reparameterization methods, while AdaptFormer and SSF are adapter-style approaches, and VPT and APrompt are prompt-based methods. The paper’s comparative argument is that these methods adapt too uniformly across tasks, whereas WeatherPEFT explicitly conditions the forward computation on task-specific meteorological patterns and restricts backward updates to Fisher-critical parameters. The stated result is that this combination closes the performance gap to Full-Tuning with far fewer trainable parameters across heterogeneous weather tasks (Cao et al., 26 Sep 2025).

The term has also appeared in a different context in image restoration. The PVRF study on adverse weather removal presents an “integrated, information-rich summary” from a parameter-efficient fine-tuning perspective and explicitly frames PVRF as “WeatherPEFT”: a perception-guided, PEFT-style conditioning and rectified-flow refinement scheme (Dong et al., 13 May 2026). In that formulation, a frozen vision–LLM in an AWR-QA module produces soft weather-type probabilities and low-level attribute scores. These perceptions condition a restoration backbone through Attribute-Modulated Normalization (AMN), which predicts $E_p \in \mathbb{R}^{P \times D},$3 and applies affine modulation to LayerNorm outputs, and through Weather-Weighted Adapters (WWA), which combine small adapter branches by soft weather probabilities (Dong et al., 13 May 2026).

That same PVRF summary states that the paper’s default setup uses full fine-tuning for the restoration backbone in Stage 1 and trains the rectified-flow correction network in Stage 2, but that a parameter-efficient variant is naturally supported by freezing the backbone and training AMN and WWA, with the refinement network $E_p \in \mathbb{R}^{P \times D},$4 optionally included. The residual rectified flow uses

$E_p \in \mathbb{R}^{P \times D},$5

with a residual-space construction around the posterior-mean anchor $E_p \in \mathbb{R}^{P \times D},$6 and a terminal-consistent velocity parameterization

$E_p \in \mathbb{R}^{P \times D},$7

trained by mean-squared error against the target residual velocity. This suggests a broader usage of “WeatherPEFT” as a label for PEFT-style, weather-conditioned adaptation beyond WFMs, although the formal named framework introduced for weather forecasting and reanalysis tasks is the WFM method of TADP plus SFAS (Dong et al., 13 May 2026).

A common source of confusion is therefore terminological rather than methodological. In the WFM literature, WeatherPEFT refers to the task-adaptive PEFT framework built around dynamic prompting and Fisher-guided sparse updates. In the adverse weather restoration literature, the term is used descriptively for a PEFT-style variant of PVRF rather than as the title of the method itself. The two usages share an emphasis on lightweight adaptation modules and frozen large backbones, but they target different problem classes and different conditioning signals (Cao et al., 26 Sep 2025).

7. Limitations, assumptions, and prospective directions

The reported limitations of WeatherPEFT are primarily about budget, generalization, and physical structure. With very small budgets of approximately 0.3%, a modest gap to Full-Tuning may remain. The paper’s own sensitivity analysis indicates that moving toward 3–4% of trainable parameters closes or exceeds Full-Tuning on many tasks. This establishes an explicit efficiency–performance trade-off rather than claiming a universal dominance of the smallest-budget configuration (Cao et al., 26 Sep 2025).

The framework is also data-driven. The paper reports improved EECRPS and Threat Score under extreme events, and it presents the 2020 China Mei-yu flood case as evidence of robustness, but it states that no explicit out-of-distribution module is included. Likewise, physical priors such as conservation laws or dynamical constraints are not part of the current formulation and are identified as a direction for future work. The authors also emphasize the continuing growth of WFMs toward billions of parameters as the broader context in which resource-constrained adaptation becomes increasingly important (Cao et al., 26 Sep 2025).

A further limitation is architectural scope. The reported evaluations are on transformer-based backbones, especially Aurora and Prithvi-WxC. The claim of portability to CNNs and GNNs is presented as an adaptation route rather than an empirical benchmark result. A plausible implication is that the framework’s core ideas—task-conditioned prompt generation from embedding spaces and Fisher-guided sparse updating—are more general than the specific transformer instantiations tested, but the strongest empirical evidence currently remains in the transformer-based WFM setting (Cao et al., 26 Sep 2025).

Taken together, these characteristics define WeatherPEFT as a weather-specific PEFT framework whose novelty lies in coordinating two kinds of adaptivity: context-aware prompt generation in the forward pass and Fisher-based parameter selection in the backward pass. Its empirical profile is strongest where conventional PEFT struggles most—tasks with strong multivariate coupling, resolution change, or sparse localized targets—and its broader significance lies in showing that PEFT for scientific foundation models may require domain-specific mechanisms rather than direct transplantation from vision or language (Cao et al., 26 Sep 2025).