TRACE Framework for Time Series Fine-Tuning

• TRACE is a parameter-efficient fine-tuning framework that addresses time series heterogeneity with a temporally aware gating protocol for LoRA modules.
• It introduces Gated DSIC to dynamically select impactful LoRA modules, ensuring unbiased gradient updates and reducing parameter overhead by up to 90%.
• Its reconstructed prediction head uses depth-wise separable convolutions to achieve superior long-term forecasting accuracy with significantly fewer parameters.

TRACE is a parameter-efficient fine-tuning framework developed for time series foundation models. Compared to prior methods, TRACE addresses unique challenges arising from the diversity in sampling frequencies, channel numbers, and history/prediction lengths that typify time series domains. It introduces two main innovations: (1) Gated DSIC (Gated Dynamic Simulation Importance Calculation), a mechanism to select the most impactful LoRA modules while preserving unbiased parameter updates, and (2) a reconstructed, parameter-lean prediction head for long-term forecasting that achieves competitive or superior performance with drastically fewer parameters than standard linear heads. TRACE has demonstrated quantitative improvements in long-/short-term forecasting, anomaly detection, and even NLP tasks, validated across datasets and use cases.

1. Challenges in Fine-Tuning Time Series Foundation Models

Time series foundation models (pre-trained transformers or CNNs) are being increasingly adopted for tasks such as forecasting and anomaly detection. However, the temporal heterogeneity—variation in sampling rates, channel numbers, length of historical/prediction windows—makes naive fine-tuning suboptimal. Classical parameter-efficient tuning approaches such as LoRA apply static low-rank updates globally, without modeling temporal importance. This results in inefficient adaptation and can exacerbate underfitting or overfitting for tasks with atypical sequence structures. TRACE directly targets this gap with a temporally aware masking and gating protocol for LoRA modules, and with attention to output head design for long-term forecasts.

2. Gated Dynamic Simulation Importance Calculation (Gated DSIC)

Gated DSIC introduces a LoRA module selection mechanism tailored to time-series temporal dynamics. Each LoRA update module is parameterized (for a host weight $W \in \mathbb{R}^{d \times k}$) as $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times k}$.

Module Importance Estimation:

• Calculate score $S$ for each module via the Frobenius norm of its gradient contribution,

$S = \|\partial \mathcal{L} / \partial(AB)\|_F$

or a finite-difference proxy,

$$S \approx \|(f_{\theta+\epsilon \cdot \Delta}(x) - f_{\theta}(x))/\epsilon\|_2$$

• Gating is then applied via a smooth function,

$g(S; \tau, \gamma) = \sigma(\gamma (S-\tau))$

where $\sigma$ is the sigmoid, $\tau$ is the importance threshold, and $\gamma$ the steepness.

• The hard mask is $m = 1_{S \geq \tau}$.

Unbiased Gradient Property:

To ensure that masking does not bias the expected gradient update,

$$\mathbb{E}[g \cdot \partial \mathcal{L} / \partial \theta] = \mathbb{E}[\partial \mathcal{L} / \partial \theta]$$

TRACE sets $\tau$ to the empirical $p$-quantile and pushes $\gamma \to \infty$ for hard selection.

Parameter Efficiency:

By pruning 70–90% of LoRA modules based on DSIC scores, TRACE maintains empirical performance while adding only 1–2% parameter overhead.

3. Reconstructed Prediction Heads for Long-Term Forecasting

TRACE replaces conventional linear probe heads with a multi-stage, compact head:

• Input: Final hidden state $h_T \in \mathbb{R}^d$.
• Stage 1 (Embedding Reducer): Dense($d \to d'$), ReLU.
• Stage 2 (Temporal Expander): Depth-wise separable 1D convolution, projecting $d' \to L \times C$ (prediction length × channel).
• Stage 3 (Channel Mixer): Optional pointwise convolution or gated unit for per-channel mixing.

By setting $d' \ll d$ and using depth-wise separable convolutions, parameter count drops to

$O(d \cdot d' + d' \cdot L + C \cdot d')$

compared to $O(d \cdot L \cdot C)$ for linear heads, achieving comparable or better forecast accuracy.

4. Experimental Protocols

Empirical validation spans forecasting and anomaly tasks, and (unexpectedly) some NLP domains:

• Datasets (typical):
  • Long-term forecasting: ETTh1, ETTh2, ETTm1, Weather, ILI
  • Short-term: Exchange-Rate, Traffic, Electricity
  • Anomaly: NASA SMAP, MSL, Yahoo Webscope
  • NLP: IMDb sentiment classification

Metrics:

• Forecasting: MSE, MAE over horizon
• Anomaly detection: AUROC, P/R at top-$k$
• NLP: classification accuracy/F1

Hyperparameter regimes:

• Learning rate $10^{-4}$, batch sizes 32–64
• LoRA ranks $r = 4$ or $8$; mask ratio 50/$70$/$90$%
• Gate steepness $\gamma = 10$, $\tau$ tuned per validation
• Typical training epochs: 20–50

5. Empirical Results and Ablations

TRACE demonstrates improved error rates and parameter efficiency across benchmarks. For ETTm1 (long-term, 24-step forecast), the following result table is representative:

Model	Params added	MSE ↓	MAE ↓
Full fine-tune	+100 %	0.215	0.325
Linear probe head	+5 %	0.230	0.340
LoRA (r=8)	+4 %	0.228	0.333
TRACE, random mask 70%	+1.2 %	0.225	0.330
TRACE, Gated DSIC, mask 70%	+1.2 %	0.220	0.326
TRACE + recon. head	+0.8 %	0.222	0.328

Across tasks, 3–8% absolute error reductions, AUROC improvements of 1–2%, and parity or gains versus linear heads are reported, all with 1–2% parameter overhead. Ablations show:

• DSIC gating outperforms random/magnitude pruning (up to 2% better MSE).
• Best performance at 70–80% LoRA modules pruned; capacity loss above 90% pruning.
• Reconstruction head matches linear performance at 25% parameter cost, modestly better generalization on longer horizons.

6. Implementation and Deployment Considerations

TRACE may be deployed atop widely used time series backbone architectures (Transformer, CNN, etc.), requiring only that LoRA-style factorized updates are insertable. Gated DSIC module importance can be computed efficiently during fine-tuning; the masking procedure is stateless and can be baked into adapter initialization.

Resource requirements are modest—memory and compute costs are only marginally over vanilla fine-tuning, with most efficiency gains on large, multivariate tasks. The head structure supports convolutional or dense backbones; for extremely low-resource deployments, $d'$ and conv kernel sizes can be further reduced.

TRACE’s unbiasedness relies on empirical calibration of gating thresholds; in domains with highly non-normal loss landscapes, threshold tuning may require bespoke validation. Long-horizon tasks are particularly well suited: the head architecture compresses channel and temporal parameterization and maintains output fidelity.

7. Significance and Limitations

TRACE is the first framework to combine unbiased, data-driven LoRA module selection and efficient output heads for time-series foundation models. Its empirical performance gains and parameter reductions apply not only to canonical forecasting and anomaly problems but are also observed in text classification settings. As only the abstract is publicly available, precise equations, architecture diagrams, and full dataset listings are unavailable. A plausible implication is that future work may extend DSIC-style gating to other modal domains (vision, audio) where frequency/channel heterogeneity is pronounced.

Care must be taken when translating gating mechanisms and reconstructed heads to tasks outside long-horizon or high-channel regimes, and as with all parameter-efficient techniques, the practical sweet spot for the proportion of modules masked/restored may differ by dataset and backbone scale. Further peer-reviewed empirical studies are needed to fully determine TRACE’s behavior under transfer-learning, multi-task scenarios, and highly imbalanced datasets.