Smoothed Full Fine-tuning (SFF)

• SFF is a weight interpolation strategy that blends a pre-trained LTSM with a randomly initialized model to form a smoother initialization, addressing sharp and poorly conditioned loss landscapes.
• It stabilizes full fine-tuning by mitigating overfitting and poor generalization, as demonstrated by consistent MSE improvements in forecasting, anomaly detection, and imputation tasks.
• SFF requires no additional inference cost since it is a one-shot preprocessing step that adjusts the loss landscape geometry, making downstream optimization more effective.

Searching arXiv for the specified papers and closely related work on Smoothed Full Fine-tuning. Smoothed Full Fine-tuning (SFF) is a weight-interpolation–based fine-tuning strategy for large time series models (LTSMs) introduced to address a specific trainability failure mode observed after pre-training: pre-trained LTSMs can occupy sharp, poorly conditioned regions of a highly non-convex loss landscape, so direct full fine-tuning becomes unstable, overfits easily, and can perform worse than training from scratch. SFF modifies the initialization of downstream adaptation rather than the downstream optimizer or architecture. Given a pre-trained LTSM and a randomly initialized auxiliary LTSM with the same architecture, it constructs an interpolated parameter vector and then performs standard full fine-tuning from that smoothed starting point (Zhang et al., 7 Jun 2026).

1. Definition and formal construction

SFF is defined for a pre-trained LTSM and a randomly initialized LTSM of identical architecture. Let the pre-trained parameter set be

$\Theta_1 = [\Phi_1, W_{\text{head1}}],$

where $\Phi_1$ denotes the backbone parameters and $W_{\text{head1}}$ the linear prediction head. Let the auxiliary randomly initialized model be

$\Theta_2 = [\Phi_2, W_{\text{head2}}],$

and let $\alpha \in [0,1]$ be an interpolation coefficient controlling retention of pre-trained knowledge. SFF constructs

$$\Theta_3 = \alpha \Theta_1 + (1-\alpha)\Theta_2 = [\Phi_3, W_{\text{head3}}],$$

with

$$\Phi_3 = \alpha \Phi_1 + (1-\alpha)\Phi_2, \qquad W_{\text{head3}} = \alpha W_{\text{head1}} + (1-\alpha)W_{\text{head2}}.$$

The downstream dataset is written as

$\mathcal{D} = \{(\mathcal{X}_i, Y_i)\}_{i=1}^N,$

where $\mathcal{X}_i \in \mathbb{R}^{t \times v}$ is a multivariate time series and $Y_i$ is the downstream target, such as a forecasting target block, masked values for imputation, or an anomaly-related target. The resulting smoothed model is then used as the initialization for full fine-tuning, meaning all parameters are updated.

With full fine-tuning, the forward function after smoothing is written as

$\Phi_1$0

and the downstream optimization objective is

$\Phi_1$1

where $\Phi_1$2 is Mean Squared Error (MSE) (Zhang et al., 7 Jun 2026).

Several operational properties are central to the method. Smoothing is performed once before fine-tuning. There are no extra parameters at inference time. There are no additional forward or backward passes during training relative to ordinary full fine-tuning. In the implementation provided with the paper, interpolation is literally a parameterwise copy operation over paired parameters of the pre-trained and randomly initialized models, after which the interpolated model is fine-tuned as usual (Zhang et al., 7 Jun 2026).

2. Loss-landscape motivation and theoretical interpretation

The motivating observation is empirical and optimization-theoretic. Using Timer as an example on Exchange, Weather, and Electricity, the paper reports that the pre-trained model exhibits a loss surface with sharp protrusions, irregular ridges, and deep narrow valleys, whereas a randomly initialized model with the same architecture exhibits a much smoother, more convex-like landscape. Direct full fine-tuning of the pre-trained model achieves the lowest training loss but often higher test loss than training from scratch, indicating overfitting and poor downstream generalization despite the presence of pre-trained knowledge (Zhang et al., 7 Jun 2026).

This behavior is formalized through Hessian-based sharpness. Let $\Phi_1$3 denote the downstream MSE and let $\Phi_1$4 be a local minimum with Hessian

$\Phi_1$5

Using the largest eigenvalue $\Phi_1$6, the paper defines a sharp minimum by

$\Phi_1$7

for some threshold $\Phi_1$8, and a flat minimum by

$\Phi_1$9

Under a second-order Taylor expansion,

$W_{\text{head1}}$0

so perturbations in directions of large curvature rapidly increase loss. If $W_{\text{head1}}$1 along the top-eigenvector direction, then

$W_{\text{head1}}$2

making the role of $W_{\text{head1}}$3 explicit (Zhang et al., 7 Jun 2026).

SFF is motivated as a sharpness-reduction device. If $W_{\text{head1}}$4 is a sharp pre-trained solution and $W_{\text{head1}}$5 lies in a flatter region, then for

$W_{\text{head1}}$6

the paper argues under a local quadratic approximation along the interpolation path that

$W_{\text{head1}}$7

and therefore

$W_{\text{head1}}$8

Given the flat-versus-sharp contrast, any $W_{\text{head1}}$9 reduces local sharpness relative to the original pre-trained solution. The corresponding claim is that SFF perturbs sharp minima without significantly harming flat regions. If the pre-trained model is already at a flat minimum, interpolation with a flat random initialization remains flat under the same approximation (Zhang et al., 7 Jun 2026).

The auxiliary model is initialized with Kaiming or Xavier schemes. The paper further analyzes the ratio

$\Theta_2 = [\Phi_2, W_{\text{head2}}],$0

arguing that under Kaiming or Xavier initialization this quantity is large, which corresponds to a broad bowl-shaped landscape with mild curvature. This suggests that the role of the auxiliary model is not to contribute semantic knowledge but to contribute a geometrically smoother region of parameter space from which downstream optimization becomes easier (Zhang et al., 7 Jun 2026).

3. Training protocol, hyperparameters, and computational profile

In practice, SFF is inserted before ordinary downstream adaptation. The workflow is: load a pre-trained LTSM, instantiate an auxiliary LTSM with the same architecture, initialize the auxiliary model with Kaiming or Xavier, interpolate parameters with coefficient $\Theta_2 = [\Phi_2, W_{\text{head2}}],$1, and then fine-tune the resulting model on the downstream task. The paper describes this as a pre-processing step applied once before standard fine-tuning, without any special schedule during optimization (Zhang et al., 7 Jun 2026).

The downstream training setup reported in the paper uses Adam with learning rate $\Theta_2 = [\Phi_2, W_{\text{head2}}],$2, 10 fine-tuning epochs, and MSE as the optimization objective. The evaluated tasks are time series forecasting with prediction horizons 96 and 720, anomaly detection using MSE as anomaly score, and imputation via masked-value reconstruction. Data availability ranges from 1% to 100% of the training split in order to test few-shot and low-resource regimes, and results are reported across four random seeds (Zhang et al., 7 Jun 2026).

Several SFF-specific hyperparameters are emphasized. The interpolation coefficient $\Theta_2 = [\Phi_2, W_{\text{head2}}],$3 was searched over $\Theta_2 = [\Phi_2, W_{\text{head2}}],$4 in the experiments. Zero-shot analyses suggest that $\Theta_2 = [\Phi_2, W_{\text{head2}}],$5 is often near-optimal for preserving pre-trained knowledge while smoothing the landscape. Kaiming Normal, Kaiming Uniform, Xavier Normal, and Xavier Uniform all work well, whereas pure standard Gaussian perturbation can hurt performance because it may place the auxiliary model in overly sharp random regions. The default interpolation proportion is 100% of model parameters, though ablations on UniTS indicate that interpolating only a subset of layers can be beneficial on small datasets (Zhang et al., 7 Jun 2026).

A common misconception is that SFF is an optimizer modification akin to Sharpness-Aware Minimization. In the formulation introduced for LTSMs, it is not. SFF does not change the per-step objective, does not require a two-pass update, and does not add memory or compute during training or inference beyond a one-time linear pass over parameters before fine-tuning. This operational distinction is central to its positioning within the paper (Zhang et al., 7 Jun 2026).

4. Empirical evaluation across models and tasks

The experimental study covers eight representative LTSMs: Timer, TimesFM, MOMENT, UniTS, MOIRAI, Chronos, TTMs, and Sundial. Architecturally, the set includes decoder-only models (Timer, TimesFM, Sundial), encoder-only models (MOMENT, MOIRAI), encoder–decoder models (Chronos, UniTS), and an MLP-only lightweight family (TTMs). Reported model sizes range from approximately 3.8 GB for TimesFM down to approximately 3 MB for TTMs (Zhang et al., 7 Jun 2026).

For time series forecasting, the paper uses eight standard multivariate datasets: Exchange rate, ETTh1, ETTh2, ETTm1, ETTm2, Weather, Electricity, and Traffic. MSE and MAE are reported at prediction lengths 96 and 720, across data proportions from 1% to 100%. For anomaly detection, the study uses the UCR Anomaly Archive with 250 datasets and evaluates predicted MSE on anomalous segments, where higher values indicate better anomaly sensitivity. For imputation, the same eight forecasting datasets are used, again measuring reconstruction MSE and MAE on masked values (Zhang et al., 7 Jun 2026).

On Timer forecasting, SFF consistently improves over direct full fine-tuning. At 25% data and prediction length 96, the paper reports Exchange MSE of 0.0805 for SFF, 0.0865 for full fine-tuning, and 0.1441 for training from scratch; ETTh2 MSE of 0.2710 for SFF, 0.2866 for full fine-tuning, and 0.2891 for training from scratch; and Weather MSE of 0.1440 for SFF, 0.1472 for full fine-tuning, and 0.1627 for training from scratch. Over 9 datasets and the higher data proportions 25%, 50%, 75%, and 100%, SFF reduces MSE relative to full fine-tuning by on average approximately 3–7%, and up to approximately 12% in some cases. At small data proportions 1–4%, it improves over full fine-tuning by average 3–4% and over training from scratch by 30–40% in MSE. The paper also reports that SFF and direct full fine-tuning converge in similar steps, both in less than one epoch, but that SFF reaches lower test MSE (Zhang et al., 7 Jun 2026).

The anomaly-detection results on Timer over the 250 UCR datasets are especially pronounced. In one representative group, SFF yields MSE 0.136 and wins 31.3 datasets on average, compared with 0.072 and 4.3 for full fine-tuning, and 0.073 and 5.3 for training from scratch. Across all six groups, SFF wins the majority of datasets in each group, with counts around 30–34 versus 3–8 for the baselines. This indicates that the smoothing effect is not confined to forecasting and extends to reconstruction-based anomaly detection (Zhang et al., 7 Jun 2026).

Similar gains are reported for TimesFM and MOMENT. For TimesFM on Exchange with 25% data, SFF attains 0.1139 versus 0.1276 for full fine-tuning and 0.1209 for training from scratch. For MOMENT on ETTh1 with 100% data, SFF attains 0.3757 versus 0.3951 for full fine-tuning and 0.3870 for training from scratch. Averaged improvements relative to full fine-tuning are reported as approximately 11.45% in MSE for TimesFM and approximately 8.31% for MOMENT. Further examples from other model families include UniTS on ETTh1 at horizon 96 with 0.656 versus 0.678, MOIRAI on Weather at horizon 96 with 0.166 versus 0.173, TTMs on ETTm2 at horizon 96 with 0.170 versus 0.178, and Sundial on ETTm1 at horizon 96 with 0.419 versus 0.428. For Chronos in zero-shot forecasting on Weather at horizon 96, the paper reports 1.087 for SFF versus 1.259 for full fine-tuning (Zhang et al., 7 Jun 2026).

The comparison with other fine-tuning strategies is also explicit. SFF substantially outperforms linear probing and LP $\Theta_2 = [\Phi_2, W_{\text{head2}}],$6 FF. On Timer averaged over 1–3% data, Exchange MSE is 0.0856 for SFF, 0.5943 for linear probing, and 0.4801 for LPFF; on ETTh1, the corresponding values are 0.3722, 0.8806, and 0.7171. Over all datasets and grouped proportions, SFF reduces MSE by 7.17%–41.57% relative to LPFF. Against optimization and regularization baselines on Timer, including label smoothing, SAM, SWA, Mixout, L2-SP, and LoRA with rank 8, the paper reports Exchange MSE values of 0.090 for full fine-tuning, 0.122 for LoRA, 0.088 for SAM, 0.090 for Mixout, 0.090 for L2-SP, and 0.081 for SFF; across six datasets, SFF consistently outperforms these alternatives (Zhang et al., 7 Jun 2026).

An additional observation is that smoothing alone can improve zero-shot performance even before any fine-tuning. For Timer on ETTh1 at horizon 96, the original pre-trained model yields 0.454 MSE, while the smoothed model yields 0.399, approximately a 12% improvement. For TimesFM on ETTh2 at horizon 96, the original model yields 1.865 and the smoothed model 0.382. Averaged over seven datasets, the paper reports 6.13% zero-shot improvement for Timer and 35.75% for TimesFM. This suggests that the pre-trained checkpoint can itself be located in a suboptimal sharp basin, and that interpolation with a random initialization may relocate it to a better local solution even without downstream adaptation (Zhang et al., 7 Jun 2026).

5. Relation to adjacent methods and broader meanings of “smoothed” fine-tuning

SFF is related to, but distinct from, several neighboring lines of work. Relative to SAM, the distinction is procedural and objective-level. SAM explicitly penalizes sharp minima through a neighborhood-based objective and requires two forward-backward passes per update, whereas SFF performs one-shot parameter interpolation before ordinary fine-tuning. Relative to SWA, which averages weights late in training to find wider optima, SFF operates before fine-tuning begins. Relative to Mixout and L2-SP, which keep parameters close to the pre-trained solution, SFF deliberately moves away from a potentially sharp pre-trained basin by mixing with an untrained model (Zhang et al., 7 Jun 2026).

It is also distinct from model soups and other model-merging approaches. Those methods interpolate between multiple trained models, usually to improve test accuracy without extra inference cost. SFF instead interpolates between a pre-trained model and a randomly initialized model, and its purpose is to improve downstream trainability by smoothing the loss landscape rather than to ensemble multiple task-specialized solutions (Zhang et al., 7 Jun 2026).

A broader use of “smoothed full fine-tuning” appears in related literature, but it refers to different mechanisms. In "Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning" (Lee et al., 20 Apr 2025), the proposed SAFT method applies ASAM-style sharpness-aware training during full fine-tuning so that task-specific models become more robust to future merge-induced perturbations. In that setting, smoothing occurs through optimization of a neighborhood-robust objective rather than through one-time weight interpolation. The paper explicitly connects flatness, Hessian eigenvalue control, and joint-task loss linearity, positioning smoothing as a means to improve merge-ability (Lee et al., 20 Apr 2025).

A different formulation appears in "Loss Smoothing for Stable Adaptation Under Distribution Shift" (Patil et al., 1 Jul 2026), where smoothing occurs in objective space rather than parameter space. There, the source objective $\Theta_2 = [\Phi_2, W_{\text{head2}}],$7 and target objective $\Theta_2 = [\Phi_2, W_{\text{head2}}],$8 are interpolated during the early phase of adaptation via

$\Theta_2 = [\Phi_2, W_{\text{head2}}],$9

with $\alpha \in [0,1]$0 ramped over a horizon $\alpha \in [0,1]$1. That method is also full-parameter fine-tuning, but its smoothing mechanism is a gradual transition between training objectives under distribution shift rather than parameter interpolation with a random auxiliary model (Patil et al., 1 Jul 2026).

These related uses of the term can generate ambiguity. In the strict sense established by the LTSM paper, SFF denotes the specific interpolation-based method defined on model weights. In a looser comparative sense, SAFT and loss smoothing are adjacent “smoothed full fine-tuning” approaches because they smooth either the local objective or the adaptation trajectory. The distinction is methodological rather than rhetorical: one smooths initialization in parameter space, another smooths the per-step optimization neighborhood, and the third smooths the source-to-target objective path (Zhang et al., 7 Jun 2026).

6. Scope, limitations, and practical significance

The primary scope of SFF is large time series models. The paper’s central claim is that pre-training can place LTSMs in sharp minima with degraded trainability, creating the specific pathology that full fine-tuning sometimes underperforms training from scratch. SFF is therefore presented not as a universal replacement for fine-tuning, but as a remedy for a pre-training-induced loss-landscape problem observed across diverse LTSM architectures and downstream time-series tasks (Zhang et al., 7 Jun 2026).

The method has several practical advantages within that scope. It requires neither architectural modification nor additional inference cost. It does not add per-step computational overhead in the way SAM does. It is implemented as a short preprocessing pass over model parameters. The paper further reports moderate hyperparameter sensitivity, with a broad good region for $\alpha \in [0,1]$2 and robust performance across Kaiming and Xavier initialization variants. This suggests that SFF is intended as a drop-in replacement for plain full fine-tuning when overfitting, instability, or worse-than-scratch adaptation is observed (Zhang et al., 7 Jun 2026).

At the same time, the empirical results indicate several nontrivial trade-offs. The interpolation coefficient mediates the tension between preserving pre-trained knowledge and smoothing aggressively; $\alpha \in [0,1]$3 reduces to standard full fine-tuning and $\alpha \in [0,1]$4 reduces to random initialization. The interpolation proportion can also matter: full-model interpolation is strongest on larger datasets, while partial interpolation may be slightly better on smaller datasets such as ETTh1 and ETTh2 in UniTS ablations. Standard Gaussian perturbation is explicitly identified as inferior to Kaiming or Xavier initialization. These observations suggest that SFF’s effectiveness depends on preserving the geometric role of the auxiliary model as a flat, well-scaled perturbation source rather than treating interpolation as arbitrary noise injection (Zhang et al., 7 Jun 2026).

A further implication is conceptual rather than conclusively established: because zero-shot performance can improve after smoothing alone, the pre-trained solution itself may be suboptimal even before downstream optimization begins. This suggests that, for LTSMs, downstream fine-tuning difficulties are not only a matter of small-data overfitting or optimizer choice, but can also arise from unfavorable placement of the pre-trained checkpoint within parameter space. SFF addresses this by rearranging the starting point of adaptation rather than by constraining later updates (Zhang et al., 7 Jun 2026).

In summary, SFF names a specific and technically narrow intervention: interpolate a pre-trained LTSM with a randomly initialized auxiliary model of the same architecture, then fully fine-tune the interpolated model. Its significance lies in the claim that trainability of pre-trained LTSMs is limited by loss-landscape geometry, and that a one-shot smoothing step can reduce sharpness, preserve useful pre-trained structure, and consistently improve downstream forecasting, anomaly detection, and imputation across a broad range of architectures (Zhang et al., 7 Jun 2026).