Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Series

Abstract: Neural networks applied to financial time series operate in a regime of underspecification, where model predictors achieve indistinguishable out-of-sample error. Using large-scale volatility forecasting for S$&$P 500 stocks, we show that different model-training-pipeline pairs with identical test loss learn qualitatively different functions. Across architectures, predictive accuracy remains unchanged, yet optimizer choice reshapes non-linear response profiles and temporal dependence differently. These divergences have material consequences for decisions: volatility-ranked portfolios trace a near-vertical Sharpe-turnover frontier, with nearly $3\times$ turnover dispersion at comparable Sharpe ratios. We conclude that in underspecified settings, optimization acts as a consequential source of inductive bias, thus model evaluation should extend beyond scalar loss to encompass functional and decision-level implications.

• The paper demonstrates that optimizer selection imposes an implicit prior on neural network functions in financial time series forecasting.
• The paper employs impulse response and geometric analyses to show that SGD yields simpler, near-linear mappings while adaptive optimizers capture richer nonlinear dynamics.
• The paper shows that these optimizer-induced functional divergences lead to distinct economic behaviors, significantly affecting trading strategy metrics like portfolio turnover.

The Optimizer as Implicit Inductive Bias in Financial Time Series: Functional and Decision-Level Divergence Under Tied Error

Introduction and Problem Formulation

This paper interrogates the consequence of optimizer choice in neural network-based volatility forecasting for S&P 500 equities where a regime of persistent underspecification results in multiple models achieving nearly identical out-of-sample error. By systematically varying both architecture (MLP, CNN, LSTM, Transformer) and optimizer (SGD, Adam, Muon) while keeping data and hyperparameters comprehensively tuned, the study exposes a profound disconnect between metric equivalence in error and the actual function class realized by different model–optimizer pairs. The central claim is that, in low signal-to-noise domains such as financial time series, the optimizer is not a benign engineering detail but a core source of inductive bias, with direct implications for functional behavior, feature attribution, temporal dependence, and—critically—downstream economic decisions.

Empirical Foundation: Predictive Equivalence and Its Fragility

The first set of results firmly establishes predictive equivalence. Across all considered architectures, neither deep nor linear models (OLS, LASSO) are able to statistically outperform each other in one-step-ahead volatility prediction on a high-quality, survivorship-bias-free S&P 500 panel. This parity persists even under rigorous, optimizer-specific hyperparameter optimization, eliminating concerns about unfair comparisons or premature convergence. As documented, differences in NMSE across optimizer–architecture pairs remain within statistical noise, and replicate across random seeds.

However, this apparently robust metric equivalence is shown to be highly deceptive. Standard leaderboard evaluations, when used in isolation, are demonstrated to collapse fundamentally distinct model behaviors and decision rules into a single scalar, reserving the possibility for radical divergence in model function and application. This Rashomon regime introduces the critical necessity for evaluation protocols extending beyond predictive scores.

Functional Divergence: Impulse Responses and Geometric Analysis

Moving past error metrics, the paper applies systematic functional diagnostics to dissect the specific functional forms selected by each (architecture, optimizer) pair. Impulse response analysis—perturbing individual volatility lags and observing model output—reveals that:

• SGD induces simpler, near-linear mappings: Response surfaces are flat or predominantly monotonic. Complexity and memory are suppressed; the learned function primarily weights recent lags with minimal nonlinear correction.
• Adaptive optimizers (Adam, Muon) enable nontrivial nonlinearity and long memory: These models manifest curved, sigmoidal impulse profiles and can encode lag structures suggestive of multi-timescale volatility spillovers or economically motivated cycles, particularly in LSTM and Transformer architectures.

This divergence is illustrated in the optimizer-difference surfaces between Adam and Muon for identical architectures, which show highly structured, non-planar landscapes, contradicting the naive expectation of optimizer redundancy under tied loss.

Figure 1: At lag $t-1$, SGD (green) produces a flat response, while Adam and Muon (blue/red) encode nonlinear dampening, all yielding similar predictive error but radically distinct mappings.

Figure 2: LSTM response surface under Muon, highlighting highly nonlinear, lag-specific shock sensitivity.

The dynamical and mechanistic underpinnings are further illuminated via curvature measurement. Adaptive optimizers consistently converge to solutions with significantly higher Hessian spectral norm (max eigenvalue) compared to SGD, substantiating that flatter minima (SGD) are associated with simpler mappings, whereas the ability to traverse sharper regions (Adam/Muon) gives rise to richer, more complex functionals.

Feature Attribution and the Role of Optimization Geometry

Feature attribution analysis via SHAP exposes that temporal dependence and effective receptive field are optimizer-mediated even under identical architectures. In LSTMs, for example, Muon-learned models utilize long-horizon lags, while Adam compresses importance toward recent history. In contrast, SGD-trained models consistently display acute recency bias, regardless of architecture. These observations confirm that the learned lag structure—interpretable as the model's economic view of how information propagates through volatility—is optimizer-dependent, not a rigid property of the neural template.

Figure 3: CNN feature attribution indicates optimizer-dependent selection of relevant temporal lags.

Mechanistic Diagnostics: Edge of Stability and Optimizer Interventions

The convergence behavior of SGD is shown to exhibit Edge of Stability (EoS) phenomena, where the top Hessian eigenvalue saturates at the stability boundary proportional to the step size, directly constraining the admissible function complexity.

Figure 4: Evolution of $\lambda_{max}$ in MLP SGD training traces the boundary of algorithmic stability, reflecting constrained complexity.

Optimizer-swap interventions provide further causality evidence. Starting from a converged Adam minimum, switching to SGD immediately induces a collapse to the characteristic flat SGD solution, and vice versa, even with early interventions. This demonstrates that the optimizer's update geometry—not initial conditions or pure stochasticity—dictates the selection among admissible functions:

Figure 5: Swapping optimizers post-convergence decisively steers functional complexity: SGD simplifies, Adam complexifies.

Figure 6: Early intervention with Adam followed by SGD yields the same attractor as baseline SGD, confirming optimizer-determined selection.

These findings support the assertion that optimization acts as a selection principle within the feasible hypothesis class, with the local geometry (flat/sharp minima, stability regime) controlling the realized model.

Economic and Decision-Level Implications: Trading Strategy Divergence

The paper demonstrates that functional divergence among models with identical predictive loss causes substantial variation in decision-level outcomes when model outputs are embedded in volatility-sorted portfolio rules. Specifically, metrics such as portfolio turnover—a proxy for trading intensity and practical implementability—vary by nearly 3× between SGD and adaptive-optimizer-trained models, despite similar risk-adjusted returns (Sharpe ratios):

Figure 7: Sharpe–Turnover frontier reveals similar Sharpe ratios but large turnover dispersion by model/optimizer, demonstrating materially different trading behaviors.

Time-series analysis confirms these distinctions persist across market regimes and stress episodes. Crucially, higher turnover, induced by adaptive optimizers, can erode realized utility once transaction costs are considered—even if ex ante predictive power is unchanged. This exposes the practical risk in using scalar error metrics for model selection in finance: models are not substitutable in production, and optimizer-dependent functional forms directly affect realized economic outcomes.

Theoretical and Practical Implications, Directions for Future Research

This work significantly sharpens our understanding of the modeling process in low-SNR, high-dimensional time series. In such regimes, model selection is function selection: optimizer choice, often relegated to a "technical" detail, is in fact a substantive modeling assumption that encodes socioeconomic hypotheses (e.g., volatility memory structure, market responsiveness) and governs real-world deployability (e.g., trading frequency, cost efficiency).

Implications for research and practice include:

• Model benchmarking in finance must extend beyond loss-based metrics: Interpretability, lag structure, and downstream economic behavior should be standard evaluation axes.
• Ensembling across optimizer–architecture pairs yields error reductions: The existence of partially orthogonal residuals among tied models suggests that exploring the Rashomon set has practical value, and that consensus or distributional approaches to downstream decisions may offer enhanced robustness.
• Further theoretical analysis is required: Understanding the precise mechanism by which update geometry and curvature constraints couple to function learning in underdetermined settings remains an open problem. The dynamic stability boundary may serve as a fruitful analytical locus in such investigations.

Conclusion

The study uncovers that optimizer choice is a consequential, intrinsic component of modeling financial time series with neural networks, acting as an implicit prior even under strict metric equivalence. Functional divergence is optimizer-driven, manifests in temporal feature attributions and surface geometry, and has direct, persistent consequences for financial decisions such as turnover and strategy implementability. As model complexity and dataset scale increase, especially in the context of foundation models for time series, explicit scrutiny of the optimizer's role as an inductive bias will be essential for robust, interpretable, and economically meaningful machine learning in finance.

---

References:

"Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Series" (2603.02620).