Adaptive Dropout for Time Series Forecasting

• The paper introduces DropoutTS, which dynamically adjusts per-sample dropout rates based on spectral noise quantification to better handle heteroskedastic noise in forecasting.
• It employs a spectral noise scorer using FFT and spectral sparsity to generate instance-level dropout rates, ensuring compatibility with diverse deep learning models.
• Empirical results demonstrate significant MSE reductions and training time improvements across synthetic and real-world datasets, confirming its robustness and efficiency.

Sample-Adaptive Dropout for Time Series Forecasting describes strategies that dynamically adjust model capacity on a per-sample basis to improve robustness against heteroskedastic and real-world noise in temporal forecasting tasks. The primary operationalization of this principle is DropoutTS, a model-agnostic plugin that performs instance-level noise quantification using spectral sparsity and modulates dropout rates accordingly, without altering backbone architectures or objectives (Zhong et al., 29 Jan 2026).

1. Framework and Operational Mechanism

DropoutTS is structured as a capacity-centric plugin compatible with a wide range of deep time series backbones, including RNNs, CNNs, Transformers, and MLPs. It comprises two differentiable modules:

• Spectral Noise Scorer: Processes each input sequence $x_i$ to quantify its noise content via reconstruction residuals in the spectral domain.
• Sample-Adaptive Dropout: Maps the computed noise scores to per-sample dropout rates $p_i$, consistently applied in all subsequent feature layers.

All core steps, from spectral noise scoring to adaptive dropout-rate generation, are differentiable and trained end-to-end with the backbone through the same task-triggered loss (e.g., MSE). The spectral-filter parameters co-adapt with the backbone weights during optimization.

2. Spectral Sparsity-Based Noise Quantification

The noise quantification approach in DropoutTS is grounded in principles from compressed sensing and Fourier analysis: clean signals are spectrally sparse, while noise disperses energy broadly in the frequency domain.

Steps:

• Global Linear Detrending: Remove global linear trends from the input sequence.

$$x_{\text{detrend}} = x - (w^* t + b^*)\quad \text{where}~(w^*, b^*) = \arg\min_{w, b} \sum_{t=1}^L \|x_t - (wt + b)\|^2$$

• Fourier Transform and Normalization: Apply FFT, compute log-amplitudes, and perform instance-wise min–max normalization in log-amplitude space:

$$Z = \mathcal{F}(x_{\text{detrend}}),\quad A = |Z|,\quad L = \log(1 + A),$$

$$\hat{A} = \frac{L - \min(L)}{\max(L) - \min(L) + \epsilon} \in [0,1]$$

• Spectral Flatness Measure (SFM): Compute SFM, a metric near zero for sharply peaked (signal-like) spectra and near one for flat (noise-like) ones:

$$\mathrm{SFM}(A) = \frac{ \exp\left( \frac{1}{K} \sum_k \ln A_k \right) }{ \frac{1}{K} \sum_k A_k }$$

• Adaptive Spectral Masking: Use a sigmoid-transformed, learnable threshold based on SFM:

$\tau = \sigma(w_s \cdot \mathrm{SFM}(A) + b_s)$

$$M = \sigma\left( \mathrm{Softplus}(\alpha) \cdot (\hat{A} - \tau) \right)$$

• Residual Score Computation: Perform inverse FFT with masked spectra and compute the mean absolute error (MAE) between input and reconstruction:

$$x'_{\text{detrend}} = \mathcal{F}^{-1}(Z \odot M),\quad x' = x'_{\text{detrend}} + (w^* t + b^*)$$

$$s = \frac{1}{L C} \sum_{t, c} \left|x_{t, c} - x'_{t, c}\right|$$

The resulting scalar $s$ quantifies noise or outlier intensity at the sample level.

3. Adaptive Dropout Rate Assignment

The residual scores $s_i$ exhibit dataset-dependent ranges, requiring normalization:

• Batch-wise min–max scaling: $\hat{s}_i \in [0,1]$
• Dropout rate mapping:

$$\tilde{s}_i = \tanh(\hat{s}_i \cdot \mathrm{Softplus}(\gamma))$$

$$p_i = p_{\min} + (p_{\max} - p_{\min}) \cdot \tilde{s}_i$$

where $p_{\min}, p_{\max}$ are user-specified bounds (e.g., [0.05, 0.5]) and $\gamma$ controls sensitivity.

Clean samples ($\hat{s}_i \approx 0$) receive minimum dropout, preserving fidelity, while noisier samples ($\hat{s}_i \approx 1$) are regularized more strongly to avoid memorization of artifactual components.

4. Training Algorithm and Implementation

The main computational workflow within each mini-batch is:

1. Per-sample spectral noise scoring (detrend, FFT, log-norm, SFM, mask, IFFT, residual computation).
2. Batch-wise scaling and rate mapping (compute $p_i$).
3. Forward pass with sample-adaptive dropout using the straight-through estimator so that gradients flow through discrete dropout masks.
4. Loss computation and gradient updates flow through both backbone and spectral scorer parameters.

Key differentiators from standard pipelines are the per-sample dropout mask and differentiability via the straight-through estimator.

Pseudocode Summary:

Step	Description	Key Details
(1) Spectral Scoring	Per-sample FFT, residual MAE	Differentiable, instance noise quant.
(2) Rate Mapping	Normalize $s_i$, assign $p_i$	Adaptive, learnable sensitivity
(3) Forward & Dropout	STE-based per-sample dropout	Gradients flow via STE
(4) Loss & Backprop	Joint update	Backbone & scorer co-learned

Parameter overhead is minimal: four additional scalars per backbone ($\alpha, w_s, b_s, \gamma$). At inference, no additional latency is incurred.

5. Empirical Evaluation and Results

Extensive experiments evaluate DropoutTS on both synthetic and real-world benchmarks.

• Synthetic Benchmark (Synth-12): Covers four signal regimes with three noise profiles (Gaussian, heavy-tail, missing), across multiple noise intensities ($\sigma \in \{0.1, ..., 0.9\}$), with SNR ranging from 23.8 dB to 7.4 dB.
• Real-world Data: Seven benchmarks, including ETTh1/h2, ETTm1/m2 (electricity transformer temperatures), Electricity (consumption), Weather, and ILI (influenza activity). Backbone models include Informer, Crossformer, PatchTST, iTransformer, TimesNet, and TimeMixer.

Key Results:

• On Synth-12, Informer achieved up to 48.2% MSE reduction for $\sigma = 0.3$, averaging 46.0% MSE drop, while more robust models saw 0.7%–4.9% gains.
• On real-world data, MSE improvements for Informer reached 34.2% (ETTh1), 47.6% (ETTh2), and 68.0% (Electricity). State-of-the-art backbones exhibited 0.5%–13.8% gains (e.g., 13.8% for TimeMixer on Weather, 19.4% for TimesNet on ILI).
• The mean MSE improvement across seven datasets and six backbones was approximately 9.8%.
• Training time was reduced by up to 31% for Informer due to faster convergence, despite ~20% additional per-batch latency from FFT/IFFT, resulting in a net 11–31% reduction in total GPU time.

DropoutTS addresses the Fixed Dropout Paradox: fixed rates can under-regularize noisy instances and over-regularize clean ones, producing non-monotonic error curves. Sample-adaptive dropout restores logical monotonicity (more noise → more regularization → controlled variance).

6. Theoretical Foundations and Generalization Properties

Theoretical analysis establishes that in heteroskedastic linear models, the optimal dropout strength increases strictly with sample noise variance, implying that a fixed dropout rate is intrinsically suboptimal for non-uniform noise distributions.

Additionally, DropoutTS tightens the Rademacher-complexity-based generalization bound by selectively discounting noisy samples from capacity allocation while maximizing utilization on clean samples. This mechanism yields uniformly lower generalization error bounds than any fixed-rate dropout scheme.

DropoutTS acts in the function/capacity space and is complementary to data-pruning and latent prior strategies. Empirically, combining DropoutTS with Selective Learning on the ILI benchmark produced an additional ~2% MSE gain, reinforcing the stacking potential of capacity-centric and data-centric approaches.

7. Limitations, Failure Cases, and Future Directions

Limitations

• In very high-SNR, optimally tuned fixed dropout may already approach the theoretical best, limiting the advantage of adaptive approaches.
• The framework presupposes look-back windows of sufficient length for meaningful spectral decomposition (very short sequences reduce scoring efficacy).

Failure Modes

• For signals whose "clean" form is inherently broadband (e.g., true white noise prediction), signal and noise become indistinct in the spectral domain, reducing score validity.
• In scenarios with highly non-stationary or piecewise-varying trends, linear detrending can be insufficient, impairing residual-based noise estimation.

Future Directions

• Extending capacity modulation mechanisms to control network depth, width, or weight decay on a per-sample basis.
• Application to other domains with spectral sparsity structure, such as vibration analysis or speech.
• Integrating multiresolution (e.g., wavelet) transforms for handling nonstationarity across scales.
• Joint spatial–spectral scoring for structured multivariate time series, including graph-based domains (traffic, sensor networks).

In summary, DropoutTS reframes the issue of robustness in time series modeling from selective data removal or direct regularization toward dynamic per-sample allocation of model capacity, grounded in both spectral signal analysis and end-to-end learnability, with consistent theoretical and empirical improvements across diverse regimes (Zhong et al., 29 Jan 2026).