Bernstein Inequalities in Sparse VAR Models

• The paper establishes Bernstein-type inequalities for tail probabilities in nonlinear VAR models, quantifying concentration under weak dependence.
• Methodology employs basis expansion with group-Lasso penalty to enforce sparsity and control estimation bias in high-dimensional, non-parametric additive models.
• Results show scalable network recovery validated on gene-expression and synthetic data, with high AUROC and AUPR demonstrating practical effectiveness.

A high-dimensional non-parametric sparse additive model provides a flexible and interpretable statistical framework for modeling complex dependencies in settings where the number of variables is large, the relationships are nonlinear, and the underlying structure is assumed to be sparse. These models generalize classical linear sparse models by replacing scalar coefficients with unknown smooth functions. The framework is especially valuable in time series (VAR) and regression contexts where additive, non-parametric, and sparse architectures provide both modeling flexibility and control over model complexity.

1. Mathematical Formulation and Model Structure

The canonical high-dimensional non-parametric sparse additive model in time series, as formulated in "Estimation of High-dimensional Nonlinear Vector Autoregressive Models" (Han et al., 23 Nov 2025), takes the form: $X_t = h(X_{t-1}) + \epsilon_t,$ where $X_t \in \mathbb{R}^p$ is a high-dimensional vector, $h(\cdot)$ encodes the dynamic structure, and $\epsilon_t$ are i.i.d. noise. Additivity is imposed via

$h_j(x) = \sum_{k=1}^p h_{jk}(x_k),$

for each $j = 1,\ldots, p$, with each $h_{jk}$ an unknown univariate function. Sparsity is enforced by restricting the set $S = \{(j, k): h_{jk} \not\equiv 0\}$ to cardinality much less than $p^2$. This structure generalizes linear sparse VARs ($X_t = \Theta X_{t-1} + \epsilon_t$) by replacing static coefficients with functions.

The model is also fundamental in regression contexts: for i.i.d. responses,

$Y_i = \sum_{j=1}^p f_j(X_{ij}) + \varepsilon_i,$

where most $f_j$ are null, and the nonzero $f_j$ capture nonparametric signal.

2. Basis Expansion and Sparse Estimation

Estimation of the unknown univariate functions $h_{jk}$ is achieved via truncated basis expansions. Each $h_{jk}$ is written as

$$h_{jk}(x) \approx \sum_{l=1}^L b_{jk}^{(l)*} \psi_{k,l}(x),$$

where $\{\psi_{k,1}, \ldots, \psi_{k,L}\}$ is an orthonormal basis (e.g., splines, wavelets) on a compact domain, and $L$ governs the approximation rate relative to the smoothness parameter $\beta$.

Collecting all basis coefficients into a vector $b^* \in \mathbb{R}^{p^2 L}$, the model for $X_t$ can be reformulated as

$$X_t \approx \Psi(X_{t-1})^\top b^* + r_t + \epsilon_t,$$

where $\Psi(X_{t-1})$ is a block-diagonal basis design matrix and $r_t$ is the truncation bias.

Group-sparsity is enforced via a group-Lasso penalty: $$\hat b = \arg\min_{b \in \mathbb{R}^{p^2 L}} \frac{1}{n} \sum_{t=1}^n \|X_t - \Psi(X_{t-1})^\top b\|_2^2 + \lambda \sum_{j,k} \|\Sigma_k^{1/2} b_{jk}\|_2,$$ with each $b_{jk} \in \mathbb{R}^L$ the block of coefficients for interaction $(j,k)$ and $\lambda$ the tuning parameter. This block-structured penalty induces sparsity at the interaction level (i.e., entire functions).

Numerical optimization proceeds via block coordinate descent, alternating between updates for each block $b_{jk}$ and global residual updating.

3. Statistical Theory: Rates and Concentration

A principal technical contribution in (Han et al., 23 Nov 2025) is the derivation of sharp Bernstein-type inequalities for sums of functions of the nonlinear VAR process. Assuming a componentwise-Lipschitz condition on $h$ ($\|H\|_\infty < 1$) and suitable moment conditions on $\epsilon_t$, for any Lipschitz $g$,

$$\Pr\left(\left|\sum_{t=1}^n \{g(X_t) - \mathbb{E}[g(X_t)]\}\right| \ge z\right) \le 2 \exp\left\{- \frac{z^2}{c_1 \tau^2 n + c_2 \tau z} \right\},$$

matching the rate of classical Bernstein inequalities under weak dependence.

Theoretical results (Theorem 3.1 in (Han et al., 23 Nov 2025)) establish that if

$$\lambda \gtrsim \sqrt{\frac{L \log(pL)}{n} + s_0 L^{1-\beta}}, \quad n \gtrsim s_0 L \log n \log(pL) + L^2 \log n \log(pL),$$

then, with high probability,

$$\|\hat b - b^*\|_2 \le C \sqrt{s} \lambda, \qquad \sum_{j,k} \|\hat h_{jk} - h_{jk}\|_{L_2}^2 \le C s \lambda^2 + C s L^{-2\beta},$$

where the first term is stochastic (variance-driven) error and the second is basis truncation bias. The estimation rate is thus governed by sparsity $s$, number of active influences per output $s_0$, sample size, smoothness ($\beta$), and number of basis functions $L$.

When additional incoherence assumptions (Assumption 3.4) are imposed, exact support recovery ($\Pr(\widehat S = S) \to 1$)—that is, consistent variable selection—can be demonstrated.

4. Empirical Performance and Practical Implementation

Extensive simulation studies under various network structures (random, banded, clustered) with variable dimensions ($p = 20, 50, 100$) and time series lengths ($n = 50, 100, 200, 500$) demonstrate robust performance of the sparse non-parametric additive VAR estimator. For $n=500$, AUROC up to 0.92 and AUPR up to 0.94 are achieved. Degradation with increasing $p$ is gradual, indicating scalability.

On biological gene-expression data for the E. coli SOS repair network ($p = 8$, $n = 50$), the method recovers six out of nine known regulatory edges (AUROC ≈ 0.812) and identifies key hubs, outperforming $\ell_1$-regularized linear VAR in both network recovery and absence of spurious links.

The framework is modular: wavelets, splines, or other bases can be substituted; alternative decomposable penalties (e.g., SCAD or MCP) can replace the group-Lasso to tune for different types of sparsity or smoothness. The block-structured optimization algorithm enables scalability to hundreds of series.

5. Relation to Broader High-dimensional Non-parametric Regression

The sparse additive non-parametric VAR model (Han et al., 23 Nov 2025) is a special case and significant extension of high-dimensional sparse additive modeling, previously explored in a variety of regression and estimation settings (e.g., (Haris et al., 2016, Wahl, 2014, Tan et al., 2017, Shang et al., 2013, Chatla et al., 6 May 2025, Sardy et al., 2022)). At their core, these models embrace:

• Additivity: each predictor's effect is modeled via a univariate (potentially nonlinear) function, accommodating general nonlinear dependencies and mitigating the curse of dimensionality.
• Sparsity: only a small subset among the many possible components are truly active, enabling effective variable selection and control of model complexity.
• Non-parametric estimation: basis expansions (splines, wavelets, RKHS, etc.) or reproducing kernel methods are systematically used to estimate unknown functions, often under smoothness constraints.
• Penalized estimation: convex penalties (group-Lasso, hierarchical, non-concave group, etc.) are the principal techniques for enforcing sparsity and controlling overfitting.

Theoretical frameworks consistently provide minimax-optimal or near-optimal convergence rates under high-dimensional scaling, with robustness to non-Gaussian errors (Chatla et al., 6 May 2025) and uniform asymptotic inference tools (Bach et al., 2020). Empirical process theory, oracle inequalities, and concentration inequalities underpin finite-sample guarantees throughout the literature.

6. Implications and Extensions

The high-dimensional non-parametric sparse additive model, particularly in the VAR context, strikes an effective compromise between interpretability and dynamical flexibility. Its success in recovering true networks in gene regulatory and other coupled time series settings validates the additive, sparse, and non-parametric paradigm. In broader regression and machine learning domains, similar models form the foundation for robust, scalable, and interpretably structured non-parametric learning.

A notable implication is the sharp interplay between stochastic variance, bias due to smoothness complexity, and the cost of nonlinearity, as quantitatively explicated by the dependence on $L$, $\beta$, and sparsity in the error rates. The modularity of the estimation pipeline (e.g., substituting penalty/basis types) enhances adaptability to domain-specific requirements and computational resources.

In summary, the high-dimensional non-parametric sparse additive model provides a theoretically grounded, computationally scalable, and empirically validated approach to uncovering complex sparse nonlinear structures in high-dimensional time series and regression, extending the interpretability of classical sparse models to a vastly more expressive functional domain (Han et al., 23 Nov 2025).