Kernel–Target Alignment (KTA)

• Kernel–Target Alignment (KTA) is defined as the normalized Frobenius inner product between a kernel and a target kernel, quantifying their similarity.
• It serves as a criterion for kernel selection, aids in tracking spectral dynamics in NTKs, and guides feature selection and transfer learning.
• KTA is also applied in quantum kernels and online learning, demonstrating its broad impact on optimization, kernel learning, and statistical performance.

Kernel–Target Alignment (KTA) is a measure of similarity between a kernel and a target kernel, usually expressed through a normalized Frobenius inner product between two Gram matrices. In supervised settings the target is commonly $T = y y^\top$ or its centered analogue, while in other settings the “target” can be a source-domain kernel, a class-indicator kernel, or a reconstruction-induced kernel. Across recent work, KTA appears in at least three distinct but related roles: as a scalar criterion for kernel selection and kernel learning, as a spectral notion describing how target energy is distributed across kernel eigenmodes, and as a dynamical phenomenon in which evolving kernels—especially NTKs—become more aligned with task-relevant directions during training (Feng et al., 2021, Shan et al., 2021, Amini et al., 2022).

1. Formal definitions and variants

The classical formulation uses the Frobenius inner product

$$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$

and defines normalized alignment as

$$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$

For scalar labels $y$, a standard target kernel is $T = y y^\top$, so that $\langle K,yy^\top\rangle_F = y^\top K y$. This form appears in classical KTA, in regression analyses of tree-ensemble kernels, and in quantum-kernel work (Feng et al., 2021, Coelho et al., 12 Feb 2025).

A centered variant is often used to remove mean effects. With

$$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$

centered alignment is

$$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$

where $y_c = y - \bar y\,1$. This centered form is emphasized in work on tree-ensemble kernels, non-linear feature selection, and the HSIC connection (Feng et al., 2021, Athanasakis et al., 2013).

Several papers deliberately use uncentered alignment. In NTK analysis, the alignment metric on the training set is

$$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$

so that changes in $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$0 directly track evolution of the training-set NTK toward the label kernel (Shan et al., 2021). In one online-kernel-learning formulation, “kernel alignment” is instead

$$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$1

which acts as a mismatch measure: large kernel polarization $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$2 makes $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$3 small (Li et al., 2022).

The target kernel itself depends on the problem class. In binary classification it is often $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$4; in multi-class classification it can be a class-indicator kernel such as $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$5; in unsupervised transfer learning there are no target labels, so alignment is computed between a source data kernel $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$6 and a target-side combination kernel $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$7; in unsupervised feature selection via matrix factorization the target kernel is the linear Gram matrix of reconstructed data,

$$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$8

and alignment is between $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$9 and a centered kernel on the original data (Fang et al., 2020, Redko et al., 2016, Lin et al., 2024).

2. Spectral, statistical, and information-theoretic viewpoints

A central perspective is that KTA is fundamentally spectral. If

$$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$0

then

$$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$1

The coefficients $$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$2 quantify how much the $$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$3-th eigenvector captures the target, so “target-aligned components” are eigenvectors with large $$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$4. Tree-ensemble kernel analyses operationalize alignment by the component-wise scores $$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$5, and report that good performance is associated with strong target-aligned components concentrated in a low-dimensional subspace (Feng et al., 2021).

A sharper version appears in truncated kernel ridge regression (TKRR), where KTA is not a single scalar but an alignment spectrum. For empirical kernel matrix

$$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$6

and target $$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$7, the empirical alignment scores are

$$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$8

The exact empirical MSE of TKRR is

$$A(K,T)=\frac{\langle K,T\rangle_F}{\|K\|_F\|T\|_F}.$$9

with $y$0. Under polynomial eigenvalue decay $y$1 and polynomial alignment $y$2, there is an over-aligned regime $y$3 in which TKRR achieves

$y$4

while full KRR is capped at $y$5. The paper’s conclusion is that target alignment can be better leveraged by utilizing spectral truncation (Amini et al., 2022).

A related statistical-mechanics treatment of kernel regression defines population-level task-model alignment through the kernel eigenfunctions $y$6 and target expansion $y$7. The population alignment is

$y$8

and the cumulative power distribution

$y$9

measures how much target energy lies in top kernel modes. This formulation ties alignment to spectral bias: high-eigenvalue modes are learned faster, so high alignment predicts better sample efficiency (Canatar et al., 2020).

Centered KTA also has direct information-theoretic and dependence-measure interpretations. In unsupervised transfer learning, the centered trace term $T = y y^\top$0 is exactly the biased empirical HSIC up to constants, and with Parzen window estimates it also equals the QMI estimator. The paper states that KTA coincides with the biased estimate of HSIC when centered kernels are used (Redko et al., 2016). The non-linear feature-selection literature makes the same connection: centered KTA is a normalized HSIC-like quantity, whereas the unnormalized trace $T = y y^\top$1 is the HSIC numerator (Athanasakis et al., 2013).

3. Neural tangent kernels, feature learning, and specialization

In neural-network theory, KTA is used to describe a structural change in the NTK during training. For a network $T = y y^\top$2, the NTK is

$T = y y^\top$3

and under gradient flow with square loss,

$T = y y^\top$4

If the kernel is static, $T = y y^\top$5, then

$T = y y^\top$6

so the loss decays as a mixture of exponentials determined by the spectrum of $T = y y^\top$7 (Shan et al., 2021).

The KTA viewpoint isolates what changes when the kernel is not static. Writing

$T = y y^\top$8

alignment increases when more spectral mass is placed on directions $T = y y^\top$9 with large overlap with $\langle K,yy^\top\rangle_F = y^\top K y$0, or when eigenvalues grow along those directions. In the paper’s toy model of optimal feature evolution, the kernel obeys the conservation law

$\langle K,yy^\top\rangle_F = y^\top K y$1

and the final kernel is

$\langle K,yy^\top\rangle_F = y^\top K y$2

The $\langle K,yy^\top\rangle_F = y^\top K y$3 term boosts the projection of the kernel along $\langle K,yy^\top\rangle_F = y^\top K y$4, and the loss contracts faster than with a static kernel (Shan et al., 2021).

The same paper gives architecture-specific mechanisms. In depth-$\langle K,yy^\top\rangle_F = y^\top K y$5 deep linear networks with scalar output, the NTK on the training set satisfies the central asymptotic result

$\langle K,yy^\top\rangle_F = y^\top K y$6

Alignment therefore arises without nonlinearities, and depth amplifies the aligned component linearly in $\langle K,yy^\top\rangle_F = y^\top K y$7. The kernel does not fully collapse to $\langle K,yy^\top\rangle_F = y^\top K y$8, so alignment saturates below $\langle K,yy^\top\rangle_F = y^\top K y$9 (Shan et al., 2021).

For two-layer ReLU networks, the NTK decomposes as

$$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$0

where $$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$1 comes from gradients with respect to second-layer weights and $$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$2 from gradients with respect to first-layer weights. The derived dynamics show pronounced anisotropy in $$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$3: growth along the target direction is $$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$4 faster than in average directions, whereas $$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$5 grows approximately isotropically. In multi-output nonlinear networks the same framework defines the kernel specialization matrix

$$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$6

and empirically finds diagonally dominant KSMs for nonlinear MLPs and CNNs but flat KSMs for linear networks. In the two-layer ReLU theory, specialization comes primarily from $$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$7, with approximately twofold faster growth along the correct class target than along off-target directions (Shan et al., 2021).

4. Kernel learning, transfer, feature selection, and discriminant methods

Outside neural tangent regimes, KTA is widely used as an objective for kernel construction or as a diagnostic for whether a kernel encodes target-relevant structure. In tree-ensemble kernel learning, RF and XGB kernels are defined by leaf co-membership, and KRR performance on continuous targets is empirically monotonic in alignment measures computed from the kernel eigensystem. Strong peaks in $$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$8 indicate that the supervised signal is concentrated near a lower-dimensional manifold spanned by target-aligned components; landmark learning shows that such peaks persist under perturbations of the kernel representation (Feng et al., 2021).

In random Fourier feature methods, earlier two-stage approaches learn the feature map by maximizing alignment between a learned kernel $$H = I - \frac{1}{n}11^\top,\qquad K_c = H K H,\qquad T_c = H T H,$$9 and a pre-defined target kernel $$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$0, often the ideal label kernel $$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$1 in binary classification or $$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$2 in multi-class settings. A later end-to-end formulation argues that the pre-defined target kernel is not necessarily optimal for the downstream linear learner, and replaces explicit KTA maximization with joint empirical-risk minimization of the feature generator and classifier (Fang et al., 2020).

In unsupervised transfer learning, KTA is used to align a source data kernel $$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$3 with a target-side combination kernel

$$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$4

chosen by solving a quadratic program that maximizes unnormalized alignment with $$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$5. The resulting aligned kernel is then coupled to kernel NMF and convex NMF through a “bridge matrix,” producing the BC-NMF pipeline (Redko et al., 2016). A plausible implication is that KTA can function as a domain-matching criterion even when no target labels are available, provided that alignment is defined between data kernels rather than between a data kernel and a label kernel.

Feature-selection work uses KTA both as a relevance criterion and as a computational bottleneck. One line of work notes that greedy optimisation of Centred Kernel Target Alignment exhibits strong results in terms of generalisation accuracy and sparsity, but is computationally prohibitive for large datasets; the proposed randSel algorithm estimates feature contributions by random subsets and provides probabilistic guarantees for preserving relevant features (Athanasakis et al., 2013). Another line embeds unnormalized centered alignment into matrix factorization for unsupervised feature selection. There the core term is

$$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$6

and an MKL extension learns a convex combination of centered kernels to avoid dependence on a single a priori kernel choice (Lin et al., 2024).

KTA has also been used to derive discriminant criteria. In kernel-alignment-inspired LDA, the goal is to maximize the alignment between a projected data kernel and a class-indicator kernel. The resulting objective becomes

$$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$7

which is closely related to classical LDA and extends naturally to multi-label data (Zheng et al., 2016).

5. Quantum kernels and online learning

Quantum-kernel work often adopts KTA as a training or model-selection criterion. A standard quantum embedding kernel maps $$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$8 to $$A(K,y) = \frac{\langle K_c,T_c\rangle_F}{\|K_c\|_F\|T_c\|_F} = \frac{y_c^\top K_c y_c}{\|K_c\|_F\|y_c\|^2},$$9 and uses either the overlap kernel $y_c = y - \bar y\,1$0 or the fidelity kernel

$y_c = y - \bar y\,1$1

In variational quantum-kernel methods, KTA is used as a cost function for choosing circuit parameters, but evaluating the full kernel matrix at every step is costly. One recent direction combines KTA-trained quantum embedding kernels with the Nyström approximation

$y_c = y - \bar y\,1$2

to reduce the number of quantum circuit executions needed to construct the kernel matrix (Coelho et al., 12 Feb 2025).

Another quantum line emphasizes that many practical methods do not optimize normalized KTA directly. A sub-sampling approach for variational training defines classical KTA and centered KTA, but optimizes an SVM-style dual loss over a variational quantum kernel, evaluating only $y_c = y - \bar y\,1$3 sub-kernels per iteration. The reported effect is a substantial reduction in circuit counts while maintaining classification accuracy (Sahin et al., 2024). A Pegasos-based method goes further and jointly updates the primal SVM weights and the quantum feature-map parameters, using the fidelity kernel and SPSA; its objective is hinge-loss-based rather than explicit KTA, but the method is presented as simultaneous kernel alignment and QSVM training, and it is reported to be particularly effective for non-stationary data (Gentinetta et al., 2023).

A fully quantum SVM pipeline uses uncentered normalized KTA

$y_c = y - \bar y\,1$4

to rank quantum kernels before solving the SVM dual as a QUBO. In that study, the top three configurations by alignment are a 30-qubit SU2HR map with one repetition, an 8-qubit Z-feature map with two repetitions, and a 16-qubit SU2RR map with one repetition, with respective KTA values $y_c = y - \bar y\,1$5, $y_c = y - \bar y\,1$6, and $y_c = y - \bar y\,1$7. The best end-to-end model, using the highest-alignment kernel, achieves $y_c = y - \bar y\,1$8, close to a classical RBF-SVM at $y_c = y - \bar y\,1$9 (Bifulco et al., 5 Sep 2025).

In online kernel learning, alignment enters directly into regret bounds. With

$$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$0

an optimistic mirror-descent algorithm obtains regret $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$1 under exponential eigenvalue decay of the kernel matrix, with $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$2 computational complexity, and $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$3 otherwise, with corresponding budgeted complexity $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$4. The batch extension yields an excess-risk bound $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$5 (Li et al., 2022).

6. Limitations, misconceptions, and open directions

A recurring misconception is that any increase in kernel norm implies better alignment. NTK theory explicitly rejects this: norm increases in $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$6 alone do not explain accelerated training; what matters is $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$7 and the concentration of spectral mass along target-relevant directions (Shan et al., 2021). A related misconception is that KTA is a single universally accepted formula. The literature includes centered and uncentered Frobenius alignment, mismatch-style online quantities such as $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$8, and spectral alignment spectra such as $$A(t)=\frac{\langle \Theta_t,yy^\top\rangle_F}{\|\Theta_t\|_F\|yy^\top\|_F} =\frac{y^\top \Theta_t y}{\|\Theta_t\|_F\|y\|^2},$$9; these objects are related but not identical (Li et al., 2022, Amini et al., 2022).

High alignment is also not sufficient by itself. In the quantum SVM pipeline, an 8-qubit configuration with KTA $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$00 yields only $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$01, which the authors attribute to PCA compression from 30 to 8 dimensions and resulting information loss. This shows that KTA is informative but not sufficient by itself; it must be interpreted together with representational adequacy and optimization quality (Bifulco et al., 5 Sep 2025).

Many analyses are strongly assumption-dependent. NTK results are derived for mean-squared error, full-batch gradient descent or gradient flow, and specific architectural conditions such as static gates in wide ReLU networks or small, balanced initialization in deep linear networks; extending alignment theory from the training Gram matrix to test-set kernels and predicting generalization gains remains an open direction (Shan et al., 2021). TKRR theory is stated for fixed design, additive zero-mean noise, and empirical norms, and further work is needed for classification losses and minimax optimality in the over-aligned regime (Amini et al., 2022). Tree-ensemble alignment results focus on regression with continuous targets; classification requires appropriate target encoding and often benefits from centered alignment (Feng et al., 2021).

Optimization itself can become difficult even when the model is underparameterized. A one-qubit toy model for cloud detection optimizes uncentered KTA for the kernel

$$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$02

and finds that the landscape possesses either many local extrema or becomes flat with a narrow global extremum. In that model, the global optimum lies at $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$03, the peak width satisfies

$$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$04

and the average alignment scales as $$\langle K, T\rangle_F = \operatorname{trace}(K^\top T),$$05, indicating that optimization becomes harder as more data are introduced (Miroszewski et al., 2023).

Open directions are correspondingly diverse. Tree-ensemble studies propose extending KTA assessment to oblique forests, rotation forests, mixup forests, Mondrian forests, BART, and survival kernels (Feng et al., 2021). Quantum-kernel work emphasizes improved sampling, low-rank approximation, and noise robustness (Coelho et al., 12 Feb 2025, Sahin et al., 2024). Neural-kernel theory identifies the need for a fully rigorous treatment of ReLU alignment dynamics and for extensions of alignment theory from training dynamics to test-set generalization (Shan et al., 2021). Together, these directions suggest that KTA is best understood not as a single metric, but as a family of alignment principles linking kernel structure, target structure, optimization dynamics, and statistical performance across a wide range of kernelized models.