Auto-Regressive Processes

• Auto-regressive processes are stochastic models defined by recursive linear functions of past values and random noise, widely used in time series analysis.
• They achieve stationarity and stability under specific parameter conditions, ensuring accurate variance and spectral estimates.
• Extensions such as random coefficient models, nonstationary approximations, and auto-regressive decoding expand their applications in physics, queueing, and machine learning.

Auto-regressive (AR) processes are a foundational class of stochastic models characterized by their recursive dependence: each value in the time series is expressed as a linear (or, in some generalizations, nonlinear or matrix-valued) function of a finite number of previous values and a stochastic innovation term. Such models form the backbone of modern time series analysis, with critical roles in statistics, physics, signal processing, and machine learning. AR processes also underpin sequential generative mechanisms adopted by deep learning models, from text decoders to continuous-time random processes.

1. Mathematical Foundations of Auto-Regressive Processes

The canonical AR process of order $p$, denoted AR($p$), is defined as

$X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$

where $\{\varepsilon_t\}$ are i.i.d. innovations, typically Gaussian white noise with zero mean and variance $\sigma^2$, and parameters $\{\phi_i\}$ control the dependence structure.

For $p=1$, the AR(1) recursion simplifies to

$X_t = \phi X_{t-1} + \varepsilon_t,$

with the stationarity (and causality) condition $|\phi|<1$ (0709.2963). The AR(2) model introduces a second lag with parameter $\phi_2$: $p$0 The theory generalizes to vector-valued, operator-valued, or even matrix-variate recursions for applications such as time-varying covariance modeling or functional data (Fox et al., 2011, Benth et al., 2017).

The structure of the AR process is governed by the roots of the characteristic polynomial $p$1, with the spectral radius $p$2 (where $p$3 are roots of $p$4) controlling stability and decay regimes (Dembo et al., 2019).

2. Stationarity, Stability, and Transient Behavior

For infinite-length AR(1) with $p$5, the process is strictly stationary and has variance $p$6. In contrast, finite-length simulations initialized from non-stationary values exhibit pronounced transients, with the variance evolving as

$p$7

if the process is started with $p$8 (0709.2963). These transients decay exponentially for moderate $p$9, but for $X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$0 near unity, convergence may require $X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$1 time steps.

To ensure stationarity from the outset in a simulated process, it is necessary to initialize $X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$2 (0709.2963). Failure to do so leads to systematic bias in variance and power spectral estimates, especially at low frequencies—a phenomenon particularly acute for strongly persistent ($X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$3) processes.

For general AR($X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$4), the detailed structure of the roots of $X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$5 controls the stability regime:

• If $X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$6, all disturbances decay exponentially.
• If $X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$7, the process is unstable, and disturbances grow exponentially.
• If $X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$8, polynomially bounded oscillations (“critical stability”) appear. Persistence properties—the probability of a trajectory staying non-negative (or in some convex set)—are tightly linked to the positivity and multiplicity of the dominant root, yielding a taxonomy of exponential, stretched exponential, and polynomial decay rates, with sharp transitions at certain parameter values (Dembo et al., 2019).

3. Extensions: Random Coefficients, Reflected Recursions, and Nonstationarity

Random Coefficient AR Processes (RCA)

The RCA model allows each $X_t = \sum_{i=1}^p \phi_i X_{t-i} + \varepsilon_t,$9 (or $\{\varepsilon_t\}$0) to be a random variable, with recursion

$\{\varepsilon_t\}$1

where $\{\varepsilon_t\}$2 are i.i.d. standard normals, $\{\varepsilon_t\}$3, $\{\varepsilon_t\}$4 i.i.d. random variables (Ślęzak et al., 2019). Under $\{\varepsilon_t\}$5, RCA processes admit stationary solutions, whose marginal distributions are non-Gaussian mixtures: linear MSD yet with exponential or stretched-Gaussian tails for $\{\varepsilon_t\}$6. RCA models have been shown to reconcile certain “Brownian yet non-Gaussian” phenomena in physical diffusion (Ślęzak et al., 2019).

Reflected AR and Integer-Valued AR

In applications such as queueing, “reflected” AR recursions arise, with

$\{\varepsilon_t\}$7

where $\{\varepsilon_t\}$8 and $\{\varepsilon_t\}$9 are possibly dependent service and interarrival times, and $\sigma^2$0 (the AR parameter) may be constant, random, or even negative (Dimitriou et al., 2023). The analysis of stationary laws involves Laplace transforms and often yields infinite product solutions parameterized by the distributions of $\sigma^2$1, $\sigma^2$2, and $\sigma^2$3.

Generalizations include discrete-state (integer-valued) reflected AR recursions for retrial queueing, where the PGF of the stationary distribution is given by an analogous infinite-product formula (Dimitriou et al., 2023).

Nonstationary and Locally Stationary AR Approximations

For nonstationary or locally stationary time series, it is possible to globally approximate the process by a white-noise-driven AR($\sigma^2$4) process of growing order. Under uniform positive-definiteness and short-range dependence, such AR approximations yield provably small $\sigma^2$5 errors, with predictive coefficients estimated via sieve or high-dimensional OLS techniques. Adaptive statistical inference is available for detecting time-variation in AR coefficients through high-dimensional $\sigma^2$6 tests and multiplier bootstrap methods (Ding et al., 2021).

4. Statistical and Computational Analysis of AR Processes

Detrended Fluctuation Analysis (DFA)

DFA provides a principled technique to probe the scaling exponent and the range of correlation in both AR(1) and AR(2) models. The short-range DFA exponent $\sigma^2$7 is found to increase exponentially in $\sigma^2$8 for AR(1), following $\sigma^2$9. For AR(2) with positive $\{\phi_i\}$0, both the exponent and the range $\{\phi_i\}$1 increase, while negative $\{\phi_i\}$2 primarily reduces the range (0707.1437). Analysis of $\{\phi_i\}$3 and $\{\phi_i\}$4 supports model identification and diagnostic discrimination between AR(1) and AR(2) dynamics.

Testing, Inference, and Forecasting

In nonstationary or high-dimensional contexts, AR coefficients can be estimated via high-dimensional sieves, with theoretical bounds on estimation and forecasting error. Tests for time-invariance (“correlation stationarity”) of coefficients are constructed via high-dimensional quadratic forms, yielding asymptotically normal statistics under the null, and with practical implementation via multiplier bootstrap (Ding et al., 2021).

For variance matrices (multivariate “volatility”), the stationary inverse Wishart AR(1) (IW-AR(1)) defines the recursion

$\{\phi_i\}$5

with explicit conditions and marginal inverse-Wishart law under suitable choices of innovation distributions and parameter constraints (Fox et al., 2011). Filtering, smoothing, and posterior inference are implemented via forward-filtering-backward-sampling strategies in Bayesian computation.

5. Continuous-Time and Functional Generalizations

The continuous-time ARMA (CARMA) framework extends the AR concept to $\{\phi_i\}$6-valued processes (where $\{\phi_i\}$7 is a Hilbert space) driven by Lévy processes. The model is defined by

$\{\phi_i\}$8

where $\{\phi_i\}$9 is a polynomial in the time-derivative operator acting on $p=1$0, and $p=1$1 is white noise in $p=1$2 (Benth et al., 2017). Stationarity holds if the spectrum of the companion operator $p=1$3 lies in the left half-plane.

Sampling a CARMA($p=1$4,$p=1$5) at uniform intervals yields a discrete ARMA($p=1$6,$p=1$7) process. Functional ARMA models are thereby naturally identified as sampled versions of CARMA processes in Hilbert space, relevant for infinite-dimensional time series.

6. Auto-Regressive Decoding in Sequential Generative Models

Auto-regressive decoding underpins sequence generation in LLMs and similar architectures. Standard decoding is strictly sequential, with each token depending on its full prior context. Recent advancements, such as Auto-Parallel Auto-Regressive (APAR) decoding, leverage the conditional independence structure in hierarchical outputs (e.g., lists, tree-structured paragraphs) to enable parallelized AR thread spawning. APAR instruct-tunes LLMs to use special control tokens indicating fork points, enabling parallel execution and substantial reductions in both time and cache requirements, with up to $p=1$8 speedup in practice, and even $p=1$9 when combined with speculative decoding (Liu et al., 2024). Crucially, segment-wise attention and efficient cache management produce gains without degradation in output quality, as measured by human or automated benchmarks.

7. Applications and Broader Implications

AR processes support a wide spectrum of applications:

• In statistical physics, RCA models clarify the universal signature of Brownian yet non-Gaussian diffusion in heterogeneous environments (Ślęzak et al., 2019).
• In queueing and service system theory, reflected AR recursions admit exact performance measures for a broad class of systems with dependent service and arrival processes (Dimitriou et al., 2023).
• In computational neuroscience, IW-AR(1) processes model nonstationary innovation covariances for multivariate EEG and neural data (Fox et al., 2011).
• In modern machine learning, AR decoding mechanisms are central to the deployment and inference efficiency of dominant LLMs (Liu et al., 2024).

AR process theory links classical time series analysis with operator theory, functional analysis, stochastic processes in infinite dimensions, and state-of-the-art machine learning, enabling both fundamental mathematical insights and practical tools for modeling, inference, and prediction.