Neural Network-Based Estimation of Time-Dependent Parameters in AR(p) Processes

Abstract: We investigate a forecasting framework based on a simple discrete-time dynamic model with coefficients varying in time. The parameters of the model are recovered within a deep learning framework, which makes it possible to retain a transparent parametric structure while simultaneously accounting for complex and nonstationary patterns in the observed phenomenon. Our analysis covers two specifications of the noise process. Besides the standard Gaussian setting, we also consider Laplace-distributed noise, which can offer a more adequate description in the presence of heavier tails and sharper local fluctuations. For both cases, we formulate the predictive scheme of the model and analyze the associated uncertainty quantification, including the construction of prediction intervals. The results illustrate that a relatively simple model, when combined with time-dependent parameter estimation, can serve as a mathematically tractable and practically flexible tool for forecasting complex dynamics under different noise assumptions. The general model is stated for TVAR($p$), while the prediction-interval formulas and the numerical experiments are developed for the TVAR(1) case.

• The paper introduces a neural network framework that estimates time-dependent AR parameters using a likelihood-based inference methodology.
• It demonstrates effective recovery of AR coefficients with low MSE and derives recursive formulas for multi-step prediction intervals under both Gaussian and Laplace noise.
• The approach proves practical in applications like electricity spot price forecasting, enhancing predictive accuracy and model interpretability in nonstationary contexts.

Neural Network-Based Estimation of Time-Dependent Parameters in AR(p) Processes

Introduction

The paper "Neural Network-Based Estimation of Time-Dependent Parameters in AR(p) Processes" (2607.00470) addresses the estimation of time-varying parameters in autoregressive (AR) models, leveraging feedforward neural networks for function approximation within a likelihood-based statistical framework. The focus is on developing interpretable time series models that retain a parametric, transparent structure while handling highly nonstationary data, accommodating both Gaussian and heavy-tailed (Laplace) noise specifications.

The methodology bridges classical state-space and smooth deterministic trend modeling with modern neural approaches, contrasting with fully black-box deep learning predictors that forfeit parameter interpretability. Explicit neural parameterizations of the local mean, AR coefficients, and noise scale are optimized directly through minimization of negative log-likelihood. Theoretical treatment centers on TVAR(1), with formulas for point prediction and exact multi-step uncertainty quantification, and is instantiated in both synthetic and real-world datasets to empirically study the recoverability and expressivity of the approach.

Model Specification

The authors consider the general TVAR($p$) process:

$$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$

where $c(t)$ (intercept), $\Phi_j(t)$ (AR coefficients), and the noise scale (variance $\sigma^2(t)$ in the Gaussian case or $b(t)$ in the Laplace case) are unknown, nonparametric functions of time. Central to the approach is the approximation of these functions via a neural network, embedding neural outputs as time-varying statistical parameters directly into the likelihood.

Given an observed sample $\{\tilde y_0, \ldots, \tilde y_N\}$, the neural network is trained to minimize either the conditional Gaussian log-likelihood or its Laplace analog, which drives estimation of $c(\cdot)$, $\Phi_j(\cdot)$, and the appropriate scale.

Noise Specifications and Loss Formation

Under the Gaussian noise assumption, the negative log-likelihood reduces to a conditional, time-local quadratic loss. For Laplace-distributed innovations, the objective is the sum of the log-scale and the normalized absolute error, leading to an $\ell_1$-type loss generalized across time with a time-dependent scale:

$$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$0

Neural models are tasked with directly outputting parameter values at each $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$1. In practice, a feedforward architecture takes time indices (or features) as input and returns $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$2. For $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$3, the AR(1) setting is emphasized, allowing for explicit predictive recursion and tractable uncertainty quantification.

Prediction and Uncertainty Quantification

A core contribution is the explicit derivation of recursive formulas for point forecasts and multi-step prediction intervals under both Gaussian and Laplace innovations in the TVAR(1) case. Critical to this is propagation of parameter estimates through the predictive distribution, utilizing the plug-in principle where necessary.

For Laplace innovations, the construction of multi-step prediction intervals is handled by recursion over the prediction error process. The $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$4-step ahead forecast error is a symmetric, mixture-of-exponentials random variable whose distribution is exactly characterized at each step (see Proposition 1 and Appendix A). The prediction interval radius must be computed as the quantile of the sum of scaled Laplace variables, obtained as the (unique) solution to a transcendental equation dependent on the realized parameter paths.

Synthetic Data Experiments

To assess accuracy and identifiability, the framework is examined on synthetic TVAR(1) data with known ground-truth parameter trajectories and both Gaussian and Laplace noise realizations. Parameter functions $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$5, $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$6, and scale are chosen to be nonlinear and nonstationary, with neural networks of moderate width and depth (GELU activations) trained to minimize the respective negative log-likelihoods.

The neural estimators demonstrate high-fidelity recovery of the time-varying AR coefficients (notably, MSE $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$7 for $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$8 in both cases), with largest errors arising in the intercept and scale—a characteristic attributable to error propagation in likelihood-based inference under finite data.

Figure 1: Synthetic trajectories generated from the TVAR(1) model under Gaussian and Laplace noise.

Figure 2: True and estimated time-dependent parameters in the synthetic experiments. Panels (A)--(C): Gaussian; (D)--(F): Laplace.

The Laplace model yields particularly low reconstruction error for the scale function, suggesting robustness to outliers and heavy-tailed behavior. The relatively low network capacity required to resolve the parametric forms indicates favorable inductive bias in the neural architecture for this low-sample, structured regime.

Real-World Data: Electricity Spot Price Forecasting

Empirical validation proceeds on a univariate daily spot price series extracted from Nordic electricity markets. Two distinct training regimes are considered: short (81 observations) and long (995 observations) time windows. Parameter estimation is performed via fixed feedforward architectures outputting $$y_t = c(t) + \sum_{j=1}^{p} \Phi_j(t)\, y_{t-j} + \varepsilon_t,$$9, $c(t)$0, and either $c(t)$1 or $c(t)$2 based on the assumed noise family.

Figure 3: First 5000 records from the real-world electricity spot price dataset (Denmark and neighboring countries).

Figure 4: Data windows used in the real-data experiments: upper panel (training length 81), lower panel (995).

Visualizations of the estimated parameters in both cases reflect significant in-sample nonstationarity and sharp local structural changes, especially in the scale coefficients, supporting the necessity of a TVAR, non-Gaussian specification in realistic settings.

Figure 5: Estimated time-dependent parameters for the case with 81 training observations.

Figure 6: Estimated time-dependent parameters for 995 observations used in training.

To further probe the generative adequacy of the estimated models, simulated trajectories from the neural parameterizations are overlaid with actual test data, confirming the expressive power of the time-dependent AR structure.

Figure 7: Observed and simulated trajectories for the 995-length dataset window for Gaussian (upper) and Laplace (lower) specifications.

Quantitative evaluation, while illustrative and limited to point-wise and one/two-step ahead forecast errors on a hold-out split, suggests that longer training windows substantially improve performance. Notably, both noise models yield similar short-horizon predictive error on this dataset; the Laplace noise model produces smaller outlier reconstruction errors, supporting its relevance for heavy-tailed or volatile energy market time series.

Implementation Details

All experiments are performed using TensorFlow/PyTorch, with specific loss functions matching the negative log-likelihood for each noise family. Training is performed on ordered batches (no shuffling), with progressive learning rate decay and mini-batch approximation for longer windows. Full code and experimental notebooks are available per the paper’s supplementary material.

Recursive, likelihood-consistent multi-step prediction intervals are implemented for both noise settings, with Gaussian intervals exploiting the plug-in formula for error variance, and Laplace intervals requiring numerical root-finding over a quantile equation involving mixtures of exponentials.

Theoretical and Practical Implications

The presented framework unifies neural network regression and time-local statistical inference, supporting interpretable, nonstationary time series modeling. The explicit likelihood-based formulation for time-dependent parameters, as opposed to standard “black-box” predictors, allows uncertainty quantification via classical statistical machinery extended through neural approximation.

The derivation and implementation of exact prediction intervals under Laplace innovations, including closed-form recursions and scalar quantile equations for error propagation, expand the toolkit for probabilistic forecasting in non-Gaussian heavy-tailed regimes.

On a practical level, this neural likelihood methodology provides a computationally efficient alternative to more complex state-space and Bayesian filtering paradigms, particularly in low-data, high-volatility applications (e.g., energy forecasting, financial econometrics), where nonstationarity and heavy-tailed noise dominate.

Future Directions

Open problems include extension to general Normal-Laplace and broader non-Gaussian noise models, multivariate/matrix AR parameterization, theoretical analysis of neural estimator identifiability, as well as systematic rolling-window or expanding-window backtesting against both classical and deep learning baselines.

Investigation into the statistical properties of the plug-in intervals under finite-sample and growing sample-size regimes is warranted, as is deeper analysis of the smoothness, capacity, and regularization required of the neural function approximator for practical identifiability and generalization.

Conclusion

The paper rigorously establishes a neural likelihood approach to time-varying autoregressive modeling, supporting transparent, interpretable, and nonstationary statistical inference in both Gaussian and heavy-tailed settings. The explicit theoretical and empirical treatment of uncertainty quantification, direct comparison against true parameters, and applicability to volatile energy time series substantiate its utility for interpretable forecasting under dynamic and complex real-world regimes.