Linear Predictivity in Statistical Models

• Linear predictivity is a framework that uses Hilbert space projections to construct the best linear predictor for minimizing mean-square error.
• It underpins diverse methodologies such as time series forecasting, regression (kriging), point process modeling, and extreme value analysis through closed-form solutions and innovations algorithms.
• Practical implementations address high-dimensional challenges, missing data, and heavy-tailed distributions via regularization and specialized estimation techniques.

Linear predictivity encompasses the theory, methodology, and application of optimal linear prediction in a wide range of statistical and dynamical settings. At its core, linear predictivity concerns the conditions under which a response can be optimally predicted, in the mean-square sense, by a linear function of past or observed data—often exploiting the geometry of Hilbert spaces, structural properties of stochastic models, and the computational tractability of linear projections. The concept extends from classical time series and regression, through spatial statistics, functional data, point processes, and even to settings with extremes, missing data, or high-dimensional predictors.

1. Fundamental Concepts of Linear Predictivity

Linear predictivity is defined relative to a given structure (time series, stochastic process, random field, or high-dimensional vector) and a prediction target. The principal object is the best linear predictor (BLP), that is, the measurable linear functional or map that minimizes mean-square error among all linear predictors given observed data. In the $L^2$ setting, this corresponds to the orthogonal projection of the target variable onto the closed linear span of the predictors. In specific models—such as Gaussian processes, Markovian time series, or linear regression—the BLP coincides with the conditional expectation; in more general settings, the optimal predictor may exhibit more complex dependencies or even fail to be strictly linear in the inputs.

Several settings articulate different aspects of linear predictivity:

• Time Series: If $(X_t)_{t\in\mathbb{Z}}$ is a stationary process, the best linear prediction of $X_0$ based on $\{X_{-p}:p\in P\}$ is the orthogonal projection of $X_0$ onto their closed span. The property of a set $P\subset\mathbb{N}$ being linearly predictive is characterized by whether this projection recovers $X_0$ for all relevant processes (Chandgotia et al., 2019).
• Regression and Kriging: For a process $y(x)=\theta^\top f(x)+\varepsilon(x)$ observed on a domain $D$, the BLUP of $y(x_0)$ given $(X_t)_{t\in\mathbb{Z}}$0 over $(X_t)_{t\in\mathbb{Z}}$1 minimizes MSE and takes a closed-form via continuous analogues of discrete kriging (Dette et al., 2019).
• Point Processes: In marked temporal point processes, the best linear predictor of future increments is the orthogonal projection of the target increment onto the history, and is characterized by a Wiener–Hopf integral equation for the prediction kernel (Aigner et al., 2022).
• Extremes: In heavy-tailed settings, the predictor is defined via projection in an extremal inner-product space, replacing conventional covariance by tail pairwise dependence matrices (Lee et al., 2021).

2. Optimal Linear Prediction: Mathematical Structure and Generalization

The best linear predictor is constructed via Hilbert-space geometry. Let $(X_t)_{t\in\mathbb{Z}}$2 be the target and $(X_t)_{t\in\mathbb{Z}}$3 the observed data, all in $(X_t)_{t\in\mathbb{Z}}$4. The BLP of $(X_t)_{t\in\mathbb{Z}}$5 given $(X_t)_{t\in\mathbb{Z}}$6 is the orthogonal projection $(X_t)_{t\in\mathbb{Z}}$7, where $(X_t)_{t\in\mathbb{Z}}$8 is the closed linear span. Explicit formulas depend on the context:

• Finite-Dimensional Regression: In the classical model $(X_t)_{t\in\mathbb{Z}}$9, the BLP is the ordinary least squares estimator.
• Spatial Statistics & Continuous Kriging: The continuous observation BLUP for $X_0$0, given data on $X_0$1, is

$X_0$2

where $X_0$3 solves Fredholm equations incorporating the covariance kernel $X_0$4 and the regression structure. The mean squared error admits closed-form expressions involving $X_0$5 and the measure $X_0$6 (Dette et al., 2019).

• Functional Data: For functional linear processes, the optimal predictor is the projection onto the finite-dimensional principal subspace (via functional principal components), and the innovations algorithm is used for iteratively optimal prediction in increasing dimensions (Klepsch et al., 2016).

Generalizations involve:

• Predicted objects (e.g., derivatives of functions, increments of processes).
• Constraints (e.g., unbiasedness, or requirement that coefficients respect certain symmetries).
• Augmented or transformed feature spaces (e.g., instance normalization, trend–seasonality splitting in time series), which can often be recast as unconstrained linear regression over an augmented set (Toner et al., 2024).
• Adaptation to missing data, where the optimal predictor involves multiway interactions between observed variables and missingness indicators; simple linear augmentation achieves statistical guarantees under finite samples, while deep architectures can attain full Bayes risk if sample size allows (Morvan et al., 2020).

3. Linear Predictivity in Time Series, Regression, and Predictive Sets

Classical and High-Dimensional Regression

In both low- and high-dimensional settings, the prediction framework generalizes from basic autoregression to ensemble prediction and variable screening:

• Time Series Forecasting: Variations of linear models (with trend–seasonality decomposition, different normalization schemes) retain the property of being equivalent to linear regression over constructed features; empirical results show that the closed-form OLS solution dominates or matches more elaborate architectures in out-of-sample accuracy, and no regularization is needed in most practical regimes (Toner et al., 2024).
• Predictive Screening: In the context of many candidate predictors, aggregate tests based on out-of-sample MSE differences—comparing one-variable regression models to an intercept-only benchmark—yield asymptotically normal test statistics under the null, robust to predictors’ persistence properties. The same framework enables post-hoc screening of the “most active” linear predictors (Gonzalo et al., 2023).

Linear Predictive Sets

The property of a subset $X_0$7 being linearly predictive for a stationary process is closely related to harmonic analysis. For prediction of $X_0$8 from $X_0$9, $\{X_{-p}:p\in P\}$0 is linearly predictive exactly when the constant function $\{X_{-p}:p\in P\}$1 lies in the closed span of $\{X_{-p}:p\in P\}$2 for every singular spectral measure—this relates to the Riesz set property in Fourier analysis (Chandgotia et al., 2019). In the Gaussian case, linear predictivity and general predictivity coincide due to the structure of conditional expectations.

4. Advanced Contexts: Functional Data, Missing Information, Point Processes, and Extremes

Functional Data

For functional linear processes, projection onto leading principal components reduces the infinite-dimensional problem to a multivariate projection, solvable via the Multivariate Innovations Algorithm. Under suitable growth of the truncation dimension, the predictor converges to the theoretical optimum (Klepsch et al., 2016).

Systems with Missing Data

In linear regression with missing values, the Bayes-optimal predictor is not linear in the observed and missingness mask variables; rather, it is a multilinear function of observed data and all possible missing-value indicators. In moderate dimensions, a linear predictor on zero-imputed values augmented with missing indicators provides a statistically efficient approximation, and deeper architectures (one-layer ReLU MLPs with $\{X_{-p}:p\in P\}$3 units) can represent the exact Bayes predictor (Morvan et al., 2020).

Point Processes

For stationary marked point processes, linear prediction is characterized by solving a Wiener–Hopf integral equation relating the kernel of the predictor to autocovariances. The unique solution generalizes the Hawkes process kernel, and recursive differential or innovations-based algorithms allow explicit computation in both AR- and MA-type point process models (Aigner et al., 2022).

Extreme Value Linear Prediction

Prediction in heavy-tailed or extremal regimes is formulated in terms of tail pairwise dependence matrices (TPDMs), with the optimal “transformed-linear predictor” arising as a projection in an extremal Hilbert space. This replaces covariance-based geometry with tail-integration, yielding predictors and uncertainty quantification that are well-calibrated for extreme events (Lee et al., 2021).

5. Practical Algorithms and Applications

A spectrum of applied methodologies for linear predictivity has been developed:

Area	Linear Predictivity Methodology	Reference
Continuous kriging/blup for random fields	Fredholm equations, vector-measure BLUP, MSE formula	(Dette et al., 2019)
Functional time series	Principal component truncation + multivariate innovations	(Klepsch et al., 2016)
Time series forecasting	Feature-augmented OLS, closed-form solutions, normalization	(Toner et al., 2024)
High-dimensional predictive regression	Out-of-sample MSE screening, key-player identification	(Gonzalo et al., 2023)
Financial multi-asset prediction	Explicit OOS Sharpe formulas under OLS overfitting	(Jacquier et al., 7 Jan 2025)
Locally adaptive interpretable regression	Meta-learners controlling OLS coefficient bands	(Munkhdalai et al., 2020)
Chaotic dynamical systems	Delay embedding, small LMEs, ensemble/voting with shrinkage	(LuValle, 2013)
Sports/seasonal linear forecasts	Line fit on cumulative points, team/table accuracy metrics	(Jurman, 2015)
Predictive sets/theory	Hilbert projection, spectral and Riesz set criteria	(Chandgotia et al., 2019)
Point process/hawkes inference	Wiener–Hopf kernel, ODE/innovations-analytic recursions	(Aigner et al., 2022)
Extreme value prediction	Projection via TPDMs, angular/polar intervals	(Lee et al., 2021)
Surrogate predictivity in dynamical systems	LR Green functions, causality-based PI for ranking observables	(Tomasini et al., 2020)
Missing data: block expansion, mask features	Full Bayes 2ᵈ-block, linear + mask, MLP representation	(Morvan et al., 2020)

Application domains range from spatial-temporal forecasting (kriging and call-center workload), financial Sharpe evaluation, macroeconomic and chaotic climate forecasting, sporting outcome prediction, to data-efficient variable screening in economics.

6. Extensions, Theoretical Anchors, and Limitations

Key theoretical developments include:

• Hilbert Space Geometry: Linear predictivity is inherently a projection problem in $\{X_{-p}:p\in P\}$4, with structure and rates depending on input covariance/spectral properties—connecting directly to harmonic analysis, Riesz sets, and Hilbert module theory (Chandgotia et al., 2019, Klepsch et al., 2016).
• Semiparametric Efficiency: In regression with unknown nonlinear links, the “linearity condition” on covariates ensures the adaptive estimator achieves the same efficiency as if the link were known, via the invariance of the tangent space of the nuisance (Shao et al., 2010).
• Consistency and Convergence: For innovations algorithms, local regularity of the covariance/spectral density and shrinkage (e.g., James–Stein) are critical to ensure consistency and avoid overfitting in high- or infinite-dimensional models (Klepsch et al., 2016, LuValle, 2013).
• Limitations:
  • In functional or infinite-dimensional settings, regularization is required to prevent instability or non-existence of the optimal linear predictor.
  • For missing values, estimating the full block-interaction model is infeasible for moderate $\{X_{-p}:p\in P\}$5; practical regimes require approximation (Morvan et al., 2020).
  • Point process predictors or kernel inversion may be computationally demanding, mitigated by algorithmic recursion or exploiting banded/Toeplitz structure (Aigner et al., 2022).
  • In heavy-tailed and extreme value contexts, classical MSE-based predictors and Gaussian-based intervals severely misrepresent tail uncertainty; tail-integrated geometry corrects this (Lee et al., 2021).

Linear predictivity thus constitutes both a deep mathematical theme—governed by projection geometry, harmonic analysis, and optimality criteria—and a foundation for a wide array of modern prediction methodologies, spanning parametric, semiparametric, and nonparametric regimes across the sciences and engineering.