Forecast collapse of transformer-based models under squared loss in financial time series

Abstract: We study trajectory forecasting under squared loss for time series with weak conditional structure, using highly expressive prediction models. Building on the classical characterization of squared-loss risk minimization, we emphasize regimes in which the conditional expectation of future trajectories is effectively degenerate, leading to trivial Bayes-optimal predictors (flat for prices and zero for returns in standard financial settings). In this regime, increased model expressivity does not improve predictive accuracy but instead introduces spurious trajectory fluctuations around the optimal predictor. These fluctuations arise from the reuse of noise and result in increased prediction variance without any reduction in bias. This provides a process-level explanation for the degradation of Transformerbased forecasts on financial time series. We complement these theoretical results with numerical experiments on high-frequency EUR/USD exchange rate data, analyzing the distribution of trajectory-level forecasting errors. The results show that Transformer-based models yield larger errors than a simple linear benchmark on a large majority of forecasting windows, consistent with the variance-driven mechanism identified by the theory.

• The paper finds that transformer-based models converge to the trivial Bayes predictor under squared loss due to the weak conditional mean in financial series.
• It demonstrates that increasing model expressivity leads to variance amplification and consistently higher error, as shown in high-frequency FX data.
• The findings urge a shift to alternative forecasting objectives, such as distributional modeling and uncertainty quantification, in finance.

Forecast Collapse of Transformer-Based Models Under Squared Loss in Financial Time Series

Problem Formulation and Theoretical Analysis

This paper rigorously addresses the systematic failure of highly expressive Transformer-based models, such as PatchTST, in forecasting financial time series under squared loss objectives. The analysis is presented in terms of empirical risk minimization (ERM) for trajectory-to-trajectory learning with rich hypothesis spaces.

The main focus is on prediction tasks over time series governed by low conditional signal structure, specifically where the conditional mean of future trajectories is degenerate—flat for asset prices and zero for returns, as justified under standard martingale assumptions for financial markets. The learning setup involves observing a fixed-size past window and predicting a multi-step future trajectory. The loss function is the standard trajectory-level mean squared error (MSE).

It is established that, under squared loss, the Bayes-optimal predictor is always the conditional mean of the future trajectory given the past. For structured domains with inherent predictability (e.g., energy consumption, traffic flow), the conditional mean is informative and rich models can fit this target effectively. However, financial time series typically exhibit vanishingly weak conditional mean structure. In such settings, even in the asymptotic regime with unlimited data and optimized training, any expressive model class converges to the trivial Bayes predictor, regardless of architectural innovations.

Impact of Model Expressivity and Variance Amplification

The paper's key claim is that expanding model expressivity (as with large Transformers) does not improve predictive performance in weakly predictable regimes. Instead, it leads to variance amplification through the interpolation of noise present in the training data. The absence of signal implies a strict bias–variance trade-off: highly flexible predictors cannot reduce the (zero) bias but systematically amplify estimator variance via noise reuse.

This phenomenon is formalized by comparing prediction errors between a simple, well-specified linear benchmark and a highly expressive interpolating model. Theoretical results demonstrate that the test error of the interpolating predictor is at least twice the irreducible noise level, while the parametric linear model achieves nearly optimal risk with estimation error decaying as the sample size grows. Specifically, for a forecast of horizon $H$ and innovation variance $\sigma^2$, the out-of-sample error for the interpolating predictor $\hat{f}_n$ satisfies:

$$\mathbb{E}\|\mathbf{Y}_t^{(H)}-\hat f_n(\mathbf{X}_t^{(L)})\|_2^2 \ge 2H\sigma^2$$

compared to the linear predictor

$$\mathbb{E}\|\mathbf{Y}_t^{(H)}-\hat f_n^{\ell}(\mathbf{X}_t^{(L)})\|_2^2 = H\sigma^2 + O\left(\frac{H\sigma^2}{n}\right)$$

showing strictly inferior generalization for highly expressive models in this regime.

Empirical Evaluation on High-Frequency FX Data

The theoretical insights are substantiated via careful empirical analysis on high-frequency EUR/USD intraday price data. Both a PatchTST Transformer and a linear autoregressive model are trained to forecast 30-step ahead price trajectories. Evaluation is performed across 233 disjoint intraday forecasting windows.

Performance is assessed in terms of the distribution of squared trajectory errors. The error distribution for the PatchTST model is consistently and substantially worse than the linear baseline, both in aggregate and across individual forecast windows.

Figure 1: Empirical cumulative distribution functions of trajectory errors and the distribution of log error ratios for PatchTST versus linear regression on EUR/USD high-frequency data.

Panel (a) in Figure 1 shows a uniform rightward shift of the error distribution for the Transformer. Panel (b) quantifies this gap: in 92% of instances, the PatchTST incurs strictly higher error than the linear model, with the mean test risk ratio at approximately 1.71. This performance deficit remains or worsens as model size increases and is not attributable to architectural or optimization limitations, but is a direct consequence of the statistical learning objective in noise-dominated processes.

Implications for Research and Practice

The analysis has broad implications for model selection, methodology, and future directions in financial time-series forecasting and beyond:

• Model Misspecification Resilience: It is demonstrated that when the data-generating process offers no predictable trajectory-level signal, overparameterization and model class expansion degrade out-of-sample performance. This runs counter to perspectives where larger models are universally beneficial.
• Limits of Architectural Innovation: The results are agnostic to specific Transformer instantiations, attention mechanisms, or patching strategies. Any ERM-based approach with expressive hypothesis spaces under squared loss converges to variance-amplifying solutions in this regime.
• Necessity of New Learning Targets: Given the degenerate nature of conditional means in finance, point forecasting under squared loss is shown to be uninformative. Research should pivot to distributional forecasting—such as full conditional density estimation or uncertainty quantification—where additional model capacity may prove salient.
• Understanding the Signal-to-Noise Threshold: The analysis foregrounds the importance of quantifying the transition between structured and noise-dominated regimes. This threshold determines when model expressivity ceases to be an asset and becomes a liability for generalization.
• Guidance for Benchmark Design: The credibility of long-term forecasting benchmarks must account for underlying process-level predictability. Synthetic or engineered benchmarks with strong signal structure may not be representative of financial domains.

Conclusion

The paper provides a theoretically complete and empirically validated process-level explanation for the ineffectiveness, and even counterproductivity, of Transformer-based ERM under squared loss in financial time-series forecasting. It establishes that, in the absence of non-trivial predictable dynamics, model expressivity merely amplifies variance without reducing bias, leading to systematic performance collapse relative to simple, well-specified models.

This work suggests a redirection of research focus, emphasizing the need for alternative objectives—particularly distributional modeling and uncertainty-aware forecasting—in application domains characterized by highly stochastic or martingale-like structure. As such, the results offer a strong explanatory framework for negative empirical findings in financial ML and propose a clear path toward more statistically principled modeling in weak-signal time-series environments.