Extraction of informative statistical features in the problem of forecasting time series generated by It{ô}-type processes

Abstract: In this paper, we consider the problem of extraction of most informative features from time series that are regarded as observed values of stochastic processes satisfying the It{ô} stochastic differential equations with unknown random drift and diffusion coefficients. We do not attract any additional information and use only the information contained in the time series as it is. Therefore, as additional features, we use the parameters of statistically adjusted mixture-type models of the observed regularities of the behavior of the time series. Several algorithms of construction of these parameters are discussed. These algorithms are based on statistical reconstruction of the coefficients which, in turn, is based on statistical separation of normal mixtures. We obtain two types of parameters by the techniques of the uniform and non-uniform statistical reconstruction of the coefficients of the underlying It{ô} process. The reconstructed coefficients obtained by uniform techniques do not depend on the current value of the process, while the non-uniform techniques reconstruct the coefficients with the account of their dependence on the value of the process. Actually, the non-uniform techniques used in this paper represent a stochastic analog of the Taylor expansion for the time series. The efficiency of the obtained additional features is compared by using them in the autoregressive algorithms of prediction of time series. In order to obtain pure conclusion that is not affected by unwanted factors, say, related to a special choice of the architecture of the neural network prediction methods, we used only simple autoregressive algorithms. We show that the use of additional statistical features improves the prediction.

• The paper introduces a method to extract latent statistical features by reconstructing local mixture models from univariate Itô process data.
• It employs modified maximum likelihood (EM) and ℓ2-minimization techniques with adaptive weighting to estimate unknown drift and diffusion coefficients.
• Empirical results demonstrate that incorporating these nonparametric features significantly improves prediction metrics such as MAE and directional accuracy.

Informative Statistical Feature Extraction for Forecasting Time Series Modeled by Itô Processes

Problem Framework

The paper addresses the principled extraction of statistical features from univariate time series generated by stochastic processes governed by Itô-type SDEs with unknown, random drift and diffusion coefficients. Unlike standard approaches in finance or physics, no stationarity assumptions or parametric forms for the coefficients are imposed. This leads to a general feature extraction problem where only the observed data from the time series is available, with no external structural prior.

The main goal is to improve time series prediction by augmenting the feature space with latent statistical structure derived from the observed data, specifically by reconstructing mixture-based probabilistic models that locally fit the empirical distributions of increments.

Methodological Innovations

Statistical Reconstruction of Coefficients via Mixture Models

The fundamental modeling assumption is that the process $X(t)$ follows an SDE of the form

$dX(t) = a(t)dt + b(t)dW(t),$

with both $a(t)$ (drift) and $b(t)$ (diffusion) being random, time-varying, and entirely unknown. Since the increments of $X(t)$ are then, conditionally, mixtures of normal random variables (or more generally, scale-location mixtures), the empirical distribution of increments over a time window is modeled as a finite mixture of normal distributions with unknown weights ($p_k$), means ($a_k$), and scales ($b_k$):

$$P(X_i - X_{i-1} < x) \approx \sum_{k=1}^K p_k \Phi\left(\frac{x - a_k}{b_k}\right),$$

where $\Phi$ is the standard normal CDF.

These parameters are estimated locally in sliding windows along the time axis ("moving separation of mixtures"), providing a time-adaptive, non-parametric summary of local dynamics. The techniques for reconstructing the mixture coefficients are:

• Maximum Likelihood via EM: The conventional EM algorithm is employed for fitting mixtures but is modified to use weighting (exponential, linear, or data-driven) to alleviate lag and over-smoothing. Constraints such as non-negativity of $dX(t) = a(t)dt + b(t)dW(t),$0 and positivity of $dX(t) = a(t)dt + b(t)dW(t),$1 apply.
• $dX(t) = a(t)dt + b(t)dW(t),$2-distance Minimization: An alternative is to match the empirical and theoretical CDFs of increments in each window by minimizing the sum of squared errors at order statistics, implemented with fast sequential quadratic programming (SLSQP). This method regularizes the ill-posed maximum likelihood estimation and can be hybridized by including a log-likelihood penalty.
• Weighted Approaches: Both likelihood and $dX(t) = a(t)dt + b(t)dW(t),$3 minimization are extended to account for temporal ordering, emphasizing recent data for feature estimation.

Feature Engineering

Three approaches to feature extraction within each window are distinguished:

• Uniform Features: Averages (expectation), medians, or modes of the reconstructed mixture distributions over all data in the window, yielding time series for $dX(t) = a(t)dt + b(t)dW(t),$4 and $dX(t) = a(t)dt + b(t)dW(t),$5. These can be plugged into standard (vector) autoregressive schemes or used as additional prediction covariates.
• Non-uniform (State-dependent) Features: The window is partitioned into bins by current $dX(t) = a(t)dt + b(t)dW(t),$6 value; within each, the increment distribution is separately fitted and summarized (mean, mode, median), yielding a functionally state-dependent estimator of the coefficients. This approach is interpreted as a stochastic analog of Taylor expansion: repeated conditioning and recursion yield estimates for higher "derivatives" of the latent dynamics.
• Distributional Features: Instead of the raw mixture parameters, quantiles or CDF values of the reconstructed mixture are used as features, or even histogram-based representations. This is aligned with recent advances in distributional time series modeling and interval/histogram-valued predictions.

Alternative Kernels

Recognizing that empirical increment distributions often exhibit heavier tails than the normal law, kernels other than Gaussian are used for mixture modeling:

• Student’s t-kernels: These offer more flexibility and capture heavy tails, with shape parameters estimated alongside means and scales.
• Logistic kernels: Exploit the logistic distribution's property as a normal scale mixture with tractable tail behavior and computational benefits.

Computational Results and Numerical Experiments

Empirical evaluation is conducted on data from interplanetary magnetic field measurements ($dX(t) = a(t)dt + b(t)dW(t),$7 component), with high-frequency observations (1-minute intervals). Both the raw and smoothed series are considered to assess noise effects.

Key findings include:

• Feature Augmentation Increases Predictive Power: Adding statistical features (mixture moments, modes, quantiles) extracted via the proposed methods consistently improves forecast metrics (MAE, RMSE, DIR).
• Window Size Matters: Performance gains become significant only for adequate window sizes that balance estimation bias and variance.
• Weighted Methods Outperform Unweighted: Weighting, especially linear, confers improvements, suggesting a premium on responsiveness to nonstationary changes in dynamics.
• Choice of Kernel Affects Tail Modeling: Mixtures with Student or logistic kernels produce gains, particularly for directions prediction and in smoothed (de-noised) series.
• No Universal Optimum: The optimal combination of feature extraction method, mixture kernel, and weighting is metric- and data-dependent; adaptivity is required for best results.

Strong numerical results are reported; for instance, on the original series with a window of size 4320 (approx. three days of data), Taylor expansion-based reconstruction achieves a best MAE of 0.2696, and mixtures with Student or logistic kernels plus linear weighted EM give the highest DIR (direction prediction accuracy). On smoothed data, DIR exceeding 54% is attainable, a notable margin over baseline AR predictors.

Theoretical and Practical Implications

The work advances data-driven, nonparametric feature engineering for complex stochastic time series with unknown governing equations. By allowing both the drift and diffusion to be entirely general and extracting their statistical signatures directly from data via principled mixture modeling, it broadens applicability to strongly nonstationary or poorly specified systems.

From a methodological standpoint, the hybridization of likelihood- and discrepancy-based estimation regularizes notoriously unstable mixture fitting, a key development for practical deployment. The introduction of weighted empirical distribution functions as a forecasting tool is a technical innovation.

The flexible augmentation of feature spaces using nonparametric estimates of latent SDE parameters opens avenues for adaptation in modern machine learning pipelines, including neural network and ensemble models. Furthermore, the stochastic Taylor expansion approach can, in principle, be extended to multivariate or high-dimensional SDEs, albeit with nontrivial identification and computational challenges.

Future Directions

• Extension to Multivariate SDEs: Generalization of moving mixture methods to vector-valued processes, with corresponding latent coefficient modeling.
• Integration with Deep Learning: Incorporation of extracted statistical features in deep sequence models (e.g., LSTM, Transformer architectures) for applications such as econometrics, climatology, and astrophysical forecasting.
• Further Kernel Generalization: Exploration of other non-Gaussian, possibly nonparametric, base kernels for mixture model fitting, especially for processes with apparent regime switches or extreme value behavior.
• Real-time Adaptive Weighting: Online adaptation of weighting schemes in model fitting for rapidly nonstationary series or streaming applications.

Conclusion

This study provides a rigorous, statistics-grounded approach to the extraction of informative latent features for forecasting Itô-type stochastic process time series, without recourse to strong model assumptions or exogenous information. By leveraging local mixture models and modern regularization strategies for estimation, and systematically validating the efficacy of extracted features, the methodology robustly enhances predictive accuracy over classical univariate approaches, especially for complex, dynamically evolving systems (2604.16865).