Asymmetric Gaussian Kernel Matrices

• Asymmetric Gaussian kernel matrices are defined by Gaussian similarity functions that use different metrics for each data space, capturing directional information.
• They facilitate nonlinear SVD and asymmetric Nyström methods by preserving distinct row and column geometries without resorting to symmetrization.
• They underpin advances in local metric learning for Gaussian Processes, improving performance in tasks like graph reconstruction and node classification.

Searching arXiv for the cited papers and closely related work on asymmetric Gaussian kernel matrices. Asymmetric Gaussian kernel matrices are matrices whose entries are generated by Gaussian-type similarity functions but are not constrained to be symmetric, typically because the kernel depends on the direction of evaluation or couples two different data spaces. In the literature considered here, asymmetry arises in two principal ways: through kernels of the form $\kappa:\mathcal X\times\mathcal Z\to\mathbb R$ for nonlinear SVD on row and column spaces, and through center-dependent Gaussian Process kernels in which each center uses its own local metric. These constructions depart from the standard Mercer setting, where a single symmetric positive semidefinite kernel is defined on one space. They therefore alter both the algebraic structure of the kernel matrix and the associated learning theory, while also making it necessary to distinguish true asymmetry from related but different phenomena such as anisotropy or loss of positive definiteness on curved spaces (Tao et al., 2023, Pintea et al., 2018).

1. Definition and conceptual scope

The standard kernel-method baseline in this area is a Mercer kernel $\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$, that is, a symmetric positive semidefinite kernel on one data space. Such kernels are sufficient for eigenvalue problems, including conventional kernel PCA, but they are insufficient when the data inherently contains directionality or row/column asymmetry. The motivating examples explicitly include directed graphs, attention matrices in Transformers, bipartite or row/column structured data, and rectangular datasets (Tao et al., 2023).

In this setting, an asymmetric kernel matrix is not merely a symmetric Gram matrix with different hyperparameters. Rather, it is a matrix $G\in\mathbb R^{N\times M}$ with entries $G_{ij}=\kappa(x_i,z_j)$, where the two arguments need not belong to the same space and the matrix need not satisfy $G_{ij}=G_{ji}$. A distinct construction appears in center-based Gaussian Process regression, where the kernel from $\overline{\mathbf x}_i$ to $\overline{\mathbf x}_j$ is evaluated using the metric attached to the first center, so that each row is produced by a different local Gaussian geometry (Pintea et al., 2018).

This scope excludes an important nearby class: generalized Gaussian kernels parameterized by a symmetric positive definite precision matrix. Those kernels are anisotropic and adaptive, but not asymmetric in the sense of $k(x,y)\neq k(y,x)$. The distinction is consequential because asymmetry changes the linear-algebraic problem from an eigenvalue problem to a left/right singular-vector problem, whereas anisotropy within a Mercer kernel preserves symmetry and RKHS structure (Wada et al., 2018).

2. Principal constructions of asymmetric Gaussian kernel matrices

One construction begins from an arbitrary data matrix $A\in\mathbb R^{N\times M}$, which may be non-square and asymmetric. The row samples and column samples are defined separately as

$$\mathcal X=\{x_i=A[i,:]\}_{i=1}^N\subset \mathbb R^M,\qquad \mathcal Z=\{z_j=A[:,j]\}_{j=1}^M\subset \mathbb R^N.$$

Two nonlinear feature maps into a common feature space $\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$0 are then introduced,

$\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$1

and the asymmetric kernel is defined by

$\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$2

The induced kernel matrix

$\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$3

is not required to be symmetric and generally satisfies $\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$4 (Tao et al., 2023).

A second construction is explicitly Gaussian. In center-based GP regression, the model uses a smaller set of data centers

$\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$5

and replaces a global Gaussian kernel by a source-center-specific kernel

$\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$6

This yields asymmetry because

$\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$7

In the univariate version each center $\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$8 learns its own lengthscale $\hat{k}(x_i,x_j)=\hat{k}(x_j,x_i)$9,

$G\in\mathbb R^{N\times M}$0

while the multivariate version uses a learned precision matrix $G\in\mathbb R^{N\times M}$1,

$G\in\mathbb R^{N\times M}$2

The asymmetry is therefore induced by individualized local metrics: from $G\in\mathbb R^{N\times M}$3 to $G\in\mathbb R^{N\times M}$4, the metric of $G\in\mathbb R^{N\times M}$5 is used; from $G\in\mathbb R^{N\times M}$6 to $G\in\mathbb R^{N\times M}$7, the metric of $G\in\mathbb R^{N\times M}$8 is used (Pintea et al., 2018).

These two constructions share a common principle: asymmetry is used to preserve information that symmetrization would remove. In the two-space formulation, the preserved information is the separation between row and column roles. In the GP formulation, it is the local target variance and multi-modal structure encoded by center-specific kernel neighborhoods.

3. Nonlinear SVD, left/right geometry, and asymmetric Nyström approximation

The most explicit algebraic treatment of asymmetric kernel matrices appears in KSVD, a nonlinear extension of matrix SVD through asymmetric kernels. The method introduces

$G\in\mathbb R^{N\times M}$9

with coefficient matrices $G_{ij}=\kappa(x_i,z_j)$0 and $G_{ij}=\kappa(x_i,z_j)$1, and maximizes the variance of each mapped data source when projected onto the subspace spanned by the other source, subject to a mutual orthogonality coupling. The optimization problem is

$G_{ij}=\kappa(x_i,z_j)$2

subject to

$G_{ij}=\kappa(x_i,z_j)$3

Equivalently,

$G_{ij}=\kappa(x_i,z_j)$4

The interpretation given in the paper is direct: $G_{ij}=\kappa(x_i,z_j)$5 gives the left singular directions in the asymmetric feature space, $G_{ij}=\kappa(x_i,z_j)$6 gives the right singular directions, and the constraint couples them analogously to the bi-orthogonality of SVD (Tao et al., 2023).

Through Lagrangian duality, the KKT stationarity conditions become

$G_{ij}=\kappa(x_i,z_j)$7

together with

$G_{ij}=\kappa(x_i,z_j)$8

where $G_{ij}=\kappa(x_i,z_j)$9. If

$G_{ij}=G_{ji}$0

is the compact SVD, then the solution is

$G_{ij}=G_{ji}$1

Accordingly, KSVD recovers the left and right singular vectors of the asymmetric kernel matrix $G_{ij}=G_{ji}$2, not of a symmetrized surrogate such as $G_{ij}=G_{ji}$3, $G_{ij}=G_{ji}$4, or $G_{ij}=G_{ji}$5 (Tao et al., 2023).

The same paper extends the Nyström method to asymmetric cases by starting from a pair of adjoint eigenfunctions $G_{ij}=G_{ji}$6 and $G_{ij}=G_{ji}$7 associated with the asymmetric kernel operator. For i.i.d. samples $G_{ij}=G_{ji}$8 and $G_{ij}=G_{ji}$9, the finite-sample system is

$\overline{\mathbf x}_i$0

This yields out-of-sample approximations

$\overline{\mathbf x}_i$1

up to the scaling factors introduced in the derivation, and full-matrix approximations based on sampled submatrices,

$\overline{\mathbf x}_i$2

The method speeds up KSVD because singular subspaces are learned from a smaller sampled submatrix $\overline{\mathbf x}_i$3, thereby reducing kernel evaluation cost, storage cost, and SVD cost. Empirically, the reported outcomes are that KSVD outperforms PCA, SVD, and KPCA on node classification for Cora, Citeseer, and Pubmed directed graph data; gives the best graph reconstruction results; is often best or competitive with KPCA on UCI datasets; and that the asymmetric Nyström method is faster than truncated SVD, RSVD, and symmetric Nyström on the tested KSVD problems (Tao et al., 2023).

4. Center-dependent Gaussian Processes and local target variance

The GP construction is motivated by the claim that a single global kernel metric is too restrictive when the target distribution is multi-modal or has region-dependent variance. The model is framed as learning the target space variance in the neighborhood of a sample through metric learning. Different parts of the input space may correspond to different local target behaviors, and a single global lengthscale cannot simultaneously represent a very smooth region, a highly variable region, or regions with different local shapes and orientations in feature space (Pintea et al., 2018).

With centers $\overline{\mathbf x}_i$4, prediction takes the form

$\overline{\mathbf x}_i$5

At test time,

$\overline{\mathbf x}_i$6

so each row of the kernel vector is center-specific rather than globally shared. The training kernel matrix is asymmetric because each row is generated by the metric of its corresponding center (Pintea et al., 2018).

The parameters are not learned by maximizing the GP marginal likelihood in the standard way. Instead, the paper uses discriminative metric learning with a regularized squared prediction loss over non-center training samples,

$\overline{\mathbf x}_i$7

For the univariate case the regularizer is $\overline{\mathbf x}_i$8; for the multivariate case it is $\overline{\mathbf x}_i$9, and the paper states that the multivariate regularizer uses the Frobenius norm. The parameters are learned with SGD, all parameters are updated jointly at each iteration, batch SGD uses minibatches of 64, momentum is $\overline{\mathbf x}_j$0, the regularization weight is $\overline{\mathbf x}_j$1, and a validation set is used to select among local optima because the objective is non-convex. In the multivariate model, a cone projection step is applied after each update so that $\overline{\mathbf x}_j$2 remains positive definite. Because the kernel matrix is asymmetric, the inverse is computed via SVD rather than standard Cholesky decomposition (Pintea et al., 2018).

The reported empirical interpretation is that the asymmetric kernel captures local structure in target variability rather than only input similarity. In the 2D blob illustration, univariate AGP learns different sizes for two blobs, while multivariate AGP learns both sizes and shapes. In the consistency experiment, learned per-center lengthscales approach a known true value as more training data is used. Across So2, Temp, Airlines, UCSD, and VOC-2007, the AGP variants often outperform standard GP, center-GP, sparse GP baselines such as DTC, FITC, and PIC, and prior counting models in the image tasks (Pintea et al., 2018).

5. Positive definiteness, Mercer structure, and the asymmetry–anisotropy distinction

Asymmetric Gaussian kernel matrices are only one way in which Gaussian-kernel practice departs from the classical Euclidean Mercer setting. A separate and very strong negative result concerns the symmetric Gaussian kernel on curved metric spaces. For a metric space $\overline{\mathbf x}_j$3, the Gaussian kernel is

$\overline{\mathbf x}_j$4

and

$\overline{\mathbf x}_j$5

On the unit circle $\overline{\mathbf x}_j$6 with its intrinsic geodesic metric, the result is

$\overline{\mathbf x}_j$7

Thus the Gaussian kernel $\overline{\mathbf x}_j$8 is never positive definite on the circle, regardless of $\overline{\mathbf x}_j$9. Because positive definiteness is inherited by restriction to isometrically embedded subsets, if $k(x,y)\neq k(y,x)$0 then

$k(x,y)\neq k(y,x)$1

hence $k(x,y)\neq k(y,x)$2. The concrete examples listed include spheres $k(x,y)\neq k(y,x)$3, real projective spaces $k(x,y)\neq k(y,x)$4, and real Grassmannians $k(x,y)\neq k(y,x)$5, with further remarks that other Riemannian manifolds may admit such embeddings, including tori and some negatively curved examples (Costa et al., 2023).

The significance of that result is not asymmetry but global failure of positive definiteness. For these spaces, Gaussian kernel matrices built from arbitrary data are not guaranteed to be PSD; one cannot assume the existence of a valid RKHS feature map; and methods such as SVMs, kernel PCA, or Gaussian-process-style models can fail or require indefinite-kernel adaptations. The paper emphasizes a subtlety: for any fixed finite dataset, taking $k(x,y)\neq k(y,x)$6 drives the Gram matrix toward the identity, so it becomes PD for large enough $k(x,y)\neq k(y,x)$7, but no single $k(x,y)\neq k(y,x)$8 works uniformly for all data on the full space (Costa et al., 2023).

This provides a useful correction to a common misconception. A Gaussian kernel matrix may deviate from standard Mercer theory either because it is genuinely asymmetric, or because it is symmetric but not positive definite on the underlying geometry. Conversely, a generalized Gaussian kernel need not be asymmetric at all. In generalized Gaussian kernel adaptive filtering, the kernel is

$k(x,y)\neq k(y,x)$9

with $A\in\mathbb R^{N\times M}$0 symmetric positive definite and interpreted as an inverse covariance matrix or precision matrix. The paper explicitly states that this is a symmetric positive definite kernel function, and that the method should be understood as anisotropic, adaptive, and nonstationary in the filtering sense, not asymmetric in the kernel sense. Its main technical contribution is to update precision matrices on the SPD manifold by MEG and NMEG rules so as to preserve symmetric positive definiteness by construction (Wada et al., 2018).

6. Broader mathematical and spectral perspectives

The broader theory of nonsymmetric kernels includes probabilistic constructions in which Gaussian structure remains a reference point but symmetry is relaxed. A permanental vector with a symmetric kernel and index $A\in\mathbb R^{N\times M}$1 is a squared Gaussian vector, while the general definition extends squared Gaussian vectors to nonsymmetric kernels and positive indexes. For such kernels, diagonal equivalence, effective equivalence, and symmetrizability become central notions. The main result highlighted here is that in dimension $A\in\mathbb R^{N\times M}$2, if there exists at most one subset of three indices such that the $A\in\mathbb R^{N\times M}$3 principal submatrix is symmetrizable, then the permanental vector is infinitely divisible. The paper presents this as evidence that truly new nonsymmetric permanental kernels may be very limited, and it sharpens the boundary between kernels explained by Gaussian-square structure and those forced into the infinitely divisible regime (Eisenbaum, 2017).

A distinct but conceptually relevant perspective comes from random-matrix theory for real asymmetric sparse Gaussian matrices. A multiparametric Gaussian ensemble with arbitrary means and variances can encode sparsity and bandedness through entry-dependent parameters $A\in\mathbb R^{N\times M}$4 and $A\in\mathbb R^{N\times M}$5, and the spectral evolution can be reduced to a single effective complexity parameter $A\in\mathbb R^{N\times M}$6. The paper states a universality claim: many different sparse or banded real-asymmetric Gaussian ensembles have the same spectral density if they have the same $A\in\mathbb R^{N\times M}$7 and the same initial spectral condition. In the reported crossover, real-eigenvalue densities are Gaussian for small $A\in\mathbb R^{N\times M}$8 and semicircular for large $A\in\mathbb R^{N\times M}$9, while complex-spectrum densities become nearly angle-independent and approach uniformity in the bulk as $$\mathcal X=\{x_i=A[i,:]\}_{i=1}^N\subset \mathbb R^M,\qquad \mathcal Z=\{z_j=A[:,j]\}_{j=1}^M\subset \mathbb R^N.$$0 increases (Dutta et al., 28 Jul 2025).

That random-matrix work does not directly study Gaussian kernel matrices. A plausible implication is that non-symmetric Gaussian kernel matrices, especially when they are sparse, banded, or built from heterogeneous directional interactions, may admit useful spectral diagnostics in terms of the fraction of real eigenvalues, the radial density of complex eigenvalues, and effective complexity rather than only entrywise construction. The paper itself explicitly notes such relevance only in a conceptual sense (Dutta et al., 28 Jul 2025).