Positive Definite Covariance Functions

• Positive definite covariance functions are symmetric bivariate mappings that ensure any finite covariance matrix is Hermitian positive semidefinite, forming the basis for valid Gaussian processes.
• They are essential in constructing reproducing kernel Hilbert spaces and extend to non-Euclidean and multivariate, space–time models in statistics and machine learning.
• Recent advances focus on optimization and regularization techniques—such as eigenvalue constraints and nonparametric methods—to enforce positive definiteness in high-dimensional settings.

A positive definite covariance function is a symmetric, bivariate mapping—typically, $K: D \times D \rightarrow \mathbb{R}$ or $\mathbb{C}$—such that, for any finite collection of points $\{x_i\}_{i=1}^n \subset D$, the corresponding matrix $[K(x_i, x_j)]_{i,j}$ is Hermitian positive semidefinite. This requirement is fundamental in probability, statistics, and machine learning, as it ensures that the covariance matrix of any finite set of random variables, fields, or processes is nonnegative definite, guaranteeing existence and well-posedness of associated Gaussian measures, kriging predictors, reproducing kernel Hilbert spaces (RKHSs), and various other structures. The paper of positive definite covariance functions encompasses their characterization and construction in both Euclidean and non-Euclidean spaces, their employment in high-dimensional estimation under structural constraints, their role in graphical models, their adaptation for multivariate and hierarchical data, and their generalization to nonparametric or operator-valued forms.

1. Definitions, Core Properties, and Connections to Hilbert Spaces

A classical covariance function $K(x, y)$ is positive definite if, for any finite set $\{x_i\}_{i=1}^n$ and any $\{c_i\} \subset \mathbb{C}$,

$$\sum_{i=1}^n \sum_{j=1}^n \overline{c_i} c_j K(x_i, x_j) \ge 0.$$

This property renders $K$ a general kernel in the sense of RKHS theory and ensures existence and uniqueness of a Hilbert space $\mathcal{H}(K)$ with the reproducing property, $f(x) = \langle f, K(\cdot, x)\rangle_{\mathcal{H}}$ for all $f \in \mathcal{H}(K)$. Every $K$ of this type arises as the covariance function of a (centered) Gaussian process $\{V_x: x \in D\}$ via

$\mathbb{E}[\overline{V_x} V_y] = K(x, y),$

and, conversely, the Kolmogorov consistency theorem guarantees that any such $K$ induces a valid (finite-dimensional) Gaussian process (Jorgensen et al., 2019).

Positive definite kernels also admit explicit integral factorizations:

$$K(x, y) = \int_{B} \overline{k_x^{(B)}(b)}\, k_y^{(B)}(b) d\mu(b)$$

over measurable auxiliary “boundary” spaces $(B, \mathcal{F}_B, \mu)$, providing a geometric (or probabilistic) decomposition corresponding to the Karhunen–Loève representation and leading to boundary-based harmonic analysis, as for the Drury–Arveson kernel or iterated function system fractals (Jorgensen et al., 2019).

2. Positive Definiteness on Non-Euclidean Domains

While numerous families of positive definite functions are known in $\mathbb{R}^d$, extending them to non-Euclidean spaces requires careful analysis of the underlying geometry. A necessary and sufficient condition for a metric space $(D, d)$ to admit positive definite covariance functions of the form $r(d(s,t))$ is for $d^2$ to be conditionally negative definite (CND) (Schoenberg’s theorem):

$$\sum_{i,j} a_i a_j d(s_i, s_j)^2 \le 0, \quad \text{whenever} \,\, \sum_{i=1}^m a_i = 0.$$

If $d^2$ is CND, then $r_\nu(d) = \exp(-\kappa d^\nu)$ is positive definite for $\nu \in (0,1]$ and $\kappa > 0$, generalizing the powered exponential, Matérn, and rational quadratic kernels (Godoy et al., 21 Feb 2025).

Schoenberg’s framework encompasses many spatial domains of interest, including spheres (with great-circle distance), certain manifolds, and other non-Euclidean geometries. If a metric satisfies the CND condition, all standard radial positive definite functions constructed via the Hilbert space embedding $d(s,t) = \|F(s) - F(t)\|_{L_2}$ remain positive definite on $(D, d^{1/2})$ (Godoy et al., 21 Feb 2025).

3. Multivariate, Cross, and Nonseparable Covariance Functions

Multivariate and spatiotemporal models require the construction of matrix-valued positive definite covariance functions, $C(h, u) = [C_{ij}(h, u)]$, with both marginal and cross-covariances. For multivariate space–time modeling, a flexible extension of the Gneiting class was proposed:

$$C_{ij}(h, u) = \frac{\sigma_i \sigma_j}{[\alpha |u|^{2a} + 1]^\tau} \rho_{ij} \mathcal{F}\left( \frac{h}{[\alpha|u|^{2a} + 1]^{b/2}}; r_{ij}, \nu_{ij} \right),$$

where $\mathcal{F}$ is either the Matérn or generalized Cauchy spatial function, and sufficient conditions for positive definiteness are provided through proper mixing of marginal and cross parameters (Bourotte et al., 2015).

Application-specific sufficient conditions exploit mixture representations and analytic inequalities (e.g., Pólya-type conditions for the powered exponential and Cauchy bivariate models) (Moreva et al., 2016), or spectral/series expansions (e.g., with spherical harmonics on the sphere (Buhmann et al., 2021)). For space-time data on spheres, adaptations of the Gneiting class involving distance on $S^d$ are shown to be positive definite on $S^d \times \mathbb{R}$ for all $d$ (White et al., 2018), using a characterization via Gegenbauer (spherical polynomial) transforms and harmonic analysis with Plancherel measure (Berg, 2020).

4. Structural and High-Dimensional Estimation: Sparsity, Regularization, and Positive Definiteness

Estimation of sparse positive definite covariance or precision matrices in high dimensions is achieved using penalized convex or nonconvex optimization. Classical methods based on thresholding often result in indefinite matrices; positive definiteness is enforced via:

• Explicit eigenvalue (or cone) constraints, e.g.

$$\min_{\Sigma \succeq \epsilon I} \frac{1}{2}\|\Sigma - \widehat{\Sigma}_n\|_F^2 + \lambda \|\Sigma\|_1,$$

with $\ell_1$-penalized ADMM algorithms guaranteeing PD solutions (Xue et al., 2012, Duan et al., 2023, Wen et al., 2016).

• Condition number (well-conditioning) constraints:

$$\mathcal{C}_\kappa = \left\{\Omega \succ 0 : \lambda_{\max}(\Omega)/\lambda_{\min}(\Omega) \leq \kappa \right\}$$

enabling solution path algorithms and efficient projections via operator splitting, ensuring not only PDness but also numerical robustness (Oh et al., 2015, Choi et al., 2016).

• Linear shrinkage or explicit correction methods, such as the FSPD estimator:

$$\widehat{\Sigma}^* = \alpha \widehat{\Sigma} + (1-\alpha)\mu I$$

with choices of $(\alpha, \mu)$ enforcing $\widehat{\Sigma}^* \succeq \epsilon I$ while preserving the support of the original estimator, all in closed form (Choi et al., 2016).

• Penalized linear regression formulations on vectorized matrices, with constraints to fix unbiased diagonal estimates and force positivity via eigenvalue thresholding within an ADMM loop; such estimators accommodate noise correlation in the sample covariance and yield sparse, positive definite, and unbiased diagonal matrices (Kim et al., 12 Mar 2025).

Recent advances include the development of positive definite estimators using nonconvex penalties (e.g., SCAD, $\ell_q$, hard-thresholding), which reduce shrinkage bias yet are efficiently computed and theoretically justified (Wen et al., 2016), as well as approaches that extend the framework to repeated-measurement (hierarchical) data with estimation at multiple covariance layers (between-/within-subject) via constrained convex optimization (Duan et al., 2023).

5. Geometric, Harmonic, and Nonparametric Perspectives

Covariance functions can be interpreted, constructed, and manipulated via geometric and harmonic analysis:

• RKHSs provide the foundation for function-valued data and statistical learning. Positive definite kernels admit both probabilistic (Gaussian process) and geometric (boundary integral) factorizations, connecting stochastic analysis, operator theory, and machine learning (Jorgensen et al., 2019).
• On homogeneous spaces (e.g., spheres), positive definiteness is characterized by series expansions in zonal or spherical functions—with coefficients themselves PD over the dual group; such expansions unify spatial statistics, representation theory, and harmonic analysis (Berg, 2020).
• Nonparametric regression under PD constraints is achieved by representing estimators as integral transforms of positive surrogate measures (using Bochner's or Schoenberg's theorems) and optimizing pseudo data via evolutionary algorithms, yielding estimators that are inherently PD and can be further constrained to be isotropic or monotonic. Such methods yield reliable long-range behavior and are tailoring to kriging and spatial prediction (Kang, 2023).
• Positive definite independent (PDI) kernels generalize HSIC and distance covariance by enforcing positivity on zero-marginal quadratic forms over product domains, characterized via integral representations of Bernstein functions in the radial case (Guella, 2022).
• On polynomial hypergroups, kernels of the form

$K(n, m) = \int_{D_s} P_n(x) P_m(x) d\mu(x)$

endow nonstationary stochastic sequences with positive definite covariance, unifying spectral and prediction theory, supporting fast Levinson-type algorithms, and enabling generalized Wiener-type theorems for detection of discrete spectral components (Hösel, 25 Nov 2024).

6. Advanced and Application-Driven Scenarios

Positive definite covariance functions underpin multifidelity estimation—e.g., regression on the manifold of symmetric PD matrices ($\mathcal{P}_d$)—with intrinsic geometry dictating computation (e.g., via affine-invariant Riemannian distances), guaranteeing PDness in large-scale assimilation, metric learning, or hierarchical inference (Maurais et al., 2023). In Bayesian networks and graphical models, the possibility and uniqueness of a PD completion of a partial covariance matrix is characterized by graph-theoretic conditions (chordality or perfection for DAGs), with explicit, polynomial-time recursion schemes and closed-form determinant/inverse formulas sharply enabling parameter estimation and likelihood computation (Ben-David et al., 2011).

Flexible nonseparable cross-covariance constructions for multivariate space-time data, parameterized to allow separate smoothness and scale for each margin, are defined with sufficient PD conditions via scale-mixture or Pólya-type analytic inequalities, and estimated by composite likelihood (Bourotte et al., 2015, Moreva et al., 2016). Similarly, matrix-valued kernels constructed from completely monotone functions and generalized Aitken integrals provide a unified framework for nonseparable, cross-covariance modeling (generalizing the Gneiting class) across multivariate interpolation and probabilistic modeling (Menegatto et al., 2021).

7. Summary and Outlook

Positive definite covariance functions are central to the modeling, estimation, and prediction of probabilistic structures in both finite and infinite-dimensional settings. Their construction requires careful analysis of the underlying geometry and function class, with isometric embedding and CND/PD criteria playing a foundational role. Modern developments exploit optimization, harmonic analysis, nonparametric surrogates, geometric and manifold-valued tools, and stochastic process theory to construct, estimate, and exploit positive definite covariance functions across a vast range of applications: from high-dimensional and sparse estimation, to multivariate and space–time modeling, to machine learning, spatial statistics, and beyond.

Emerging directions involve enhanced scalability, integration of structure (e.g., sparsity, conditioning), generalization to operator- or tensor-valued kernels, and adaptation to complex measurement or sampling regimes (such as repeated/hierarchical data, manifolds beyond spheres, or function spaces with hypergroup structure). The unification of probabilistic, geometric, and computational perspectives continues to broaden the applicability and theoretical underpinnings of positive definite covariance functions in statistics, data science, and applied mathematics.