Bayesian Tensor Network Kernel Machines

• Bayesian Tensor Network Kernel Machines are probabilistic models that integrate structured tensor decompositions with hierarchical sparsity priors for automatic rank and feature selection.
• They leverage Bayesian inference via mean-field variational methods or MCMC to quantify uncertainty and regularize predictions while retaining computational efficiency.
• Their design enables scalable learning and interpretable insights by dissecting latent interactions and pinpointing relevant features in high-dimensional data.

Bayesian Tensor Network Kernel Machines are a fully probabilistic class of kernel-based machine learning models in which the parameters (weights) of the kernel machine are represented by a structured tensor network (such as a low-rank CP or tensor train decomposition), and all network components—including hyperparameters governing rank and feature dimensions—are endowed with sparsity-inducing hierarchical priors. This framework enables simultaneous uncertainty quantification, automatic selection of model complexity (tensor rank and feature relevance), and interpretability, while retaining the computational tractability expected from tensor network approaches. Bayesian inference, typically via mean-field variational methods or MCMC, is used to approximate the joint posterior over all latent variables and hyperparameters, allowing for robust learning, predictive uncertainty evaluation, and data-driven regularization (Kilic et al., 15 Jul 2025).

1. Probabilistic Formulation with Tensor Networks

Bayesian Tensor Network Kernel Machines (BTN-KMs) formalize the kernel prediction function as

$f(x) = \phi(x)^\top w$

where $\phi(x)$ is a kernel-defined feature map, and $w$ is a parameter vector exponentially large in the input and feature dimension. To address the storage and computational intractability, the parameter $w$ is factorized using a low-rank tensor network, typically the CP (CANDECOMP/PARAFAC) decomposition:

$$w = \sum_{r=1}^R w_r^{(1)} \otimes w_r^{(2)} \otimes \cdots \otimes w_r^{(D)}$$

where $R$ is the chosen (maximal) tensor rank and $w_r^{(d)} \in \mathbb{R}^{M_d}$ are the factors for the $d$th mode (see (Kilic et al., 15 Jul 2025), Eqn (4)). Each $w_r^{(d)}$—as well as all hyperparameters (noise and regularization parameters)—are treated as latent variables with prior distributions.

The prior over each factor matrix is Gaussian:

$$p(\mathrm{vec}(W^{(d)}) | \lambda_R, \lambda_{M_d}) = \mathcal{N}(0, \Lambda_R^{-1} \otimes \Lambda_{M_d}^{-1})$$

where $$\Lambda_R = \operatorname{diag}(\lambda_1, \ldots, \lambda_R)$$ encodes column-wise (rank) sparsity and $$\Lambda_{M_d} = \operatorname{diag}(\lambda^{(M,d)}_1, \ldots, \lambda^{(M,d)}_{M_d})$$ enforces feature-wise selection per mode. The likelihood for labels/targets, under Gaussian noise for regression, is

$$p(y|\{W^{(d)}\}, \tau) = \prod_n \mathcal{N}(y_n|\phi(x_n)^\top w,\tau^{-1})$$

with a Gamma prior placed on the noise precision $\tau$ [(Kilic et al., 15 Jul 2025), Eqns (7)-(12)]. This hierarchical Bayesian construction allows the model to regularize complexity automatically.

2. Automatic Rank and Feature Selection

A distinguishing characteristic of BTN-KMs is their ability to adapt both the effective tensor rank and the set of active features without requiring explicit cross-validation or manual tuning. This is achieved by imposing independent Gamma hyperpriors on the rank-precision vector $\lambda_R$ and the per-mode feature-precision vectors $\lambda_{M_d}$. As variational inference proceeds, unnecessary components in the network (columns in factor matrices for rank, rows for features) are shrunk toward zero due to large inferred precisions, thus “pruning” the model structure directly from data.

This approach draws on methods for automatic relevance determination (ARD), extended to the tensor network domain. In practical terms, components with high posterior means for the associated precision variables are interpreted as negligible and the corresponding factors are excluded from inference and prediction [(Kilic et al., 15 Jul 2025), Sec. 3]. Empirical results on synthetic data confirm that the rank and feature dimensions discovered by the model match ground truth even when starting from significantly over-parameterized configurations.

3. Model Interpretability via Sparsity-Inducing Priors

Sparsity at both the component and feature levels enhances interpretability. The hierarchical prior structure ensures that:

• The low-rank tensorization decomposes the learned function into interpretable additive/multiplicative interactions along each mode.
• The $\lambda_{M_d}$ variables indicate which original input features in each mode are most relevant, facilitating variable selection.
• High values in $\lambda_R$ point to redundant or irrelevant latent interactions, revealing effective model rank.

Posterior summaries of these parameters (e.g., histograms of $\lambda$’s posterior means) provide direct insight into which components of the network are driving predictions, and which are confidently being ignored, as shown in Figure 1 of (Kilic et al., 15 Jul 2025).

4. Mean-Field Variational Inference and Bayesian ALS

To approximate the intractable joint posterior, mean-field variational inference is employed:

$$q(\Theta) = \prod_{d=1}^D q(W^{(d)}) \prod_{d=1}^D q(\lambda_{M_d})\, q(\lambda_R) \, q(\tau)$$

Each factor is chosen in the exponential family for tractable updates (i.e., Gaussian for weights, Gamma for precisions). Updates follow standard coordinate ascent:

$$\ln q_j(\theta_j) = \mathbb{E}_{q(\Theta \setminus \theta_j)}[\ln p(y, \Theta)] + \mathrm{const}$$

The crucial aspect is that the update for each factor matrix $W^{(d)}$ has the same structure as in alternating least squares (ALS), with the addition of prior-based regularization and uncertainty propagation:

$$\Sigma^{(d)} = [ \mathbb{E}[\tau] \mathbb{E}[G^{(d)} G^{(d)\top}] + \mathbb{E}[\Lambda_R] \otimes \mathbb{E}[\Lambda_{M_d}] ]^{-1}$$

$$\operatorname{vec}(\hat{W}^{(d)}) = \mathbb{E}[\tau]\, \Sigma^{(d)}\, \mathbb{E}[G^{(d)}]\, y$$

[(Kilic et al., 15 Jul 2025), Eqn (22)]. Despite the uncertainty computations, the computational complexity of each variational iteration matches that of deterministic ALS because of exploitable tensor network algebra and careful reuse of intermediates. Thus, the Bayesian model delivers uncertainty quantification at no extra cost.

5. Empirical Performance and Uncertainty Quantification

BTN-KMs are demonstrated on both synthetic and real-world datasets, including Airfoil, Concrete, Energy, and Adult (Kilic et al., 15 Jul 2025). Key findings include:

• Prediction accuracy: Superior or competitive compared to deterministic and Bayesian tensor models (such as T-KRR and SP-BTN), as measured by RMSE, misclassification rate, and negative log-likelihood.
• Uncertainty quantification: Predictive distributions obtained via the mean-field posterior yield reliable uncertainty intervals, e.g., using the mean and variance to parameterize a Student’s t distribution for predictions.
• Scalability: Capable of operating efficiently on problems with large feature dimension and data size due to the parameter compression of tensor networks, with automatic pruning safeguarding against overfitting as initial rank or feature dimension increases.
• Robustness: The model is insensitive to over-specification of rank; unnecessary capacity is pruned during inference rather than requiring user intervention.

6. Theoretical Foundations and Future Perspectives

The Bayesian Tensor Network Kernel Machine framework is grounded in established principles of sparse Bayesian modeling and the computational paradigm of tensor networks. It extends Bayesian tensor factorization, as surveyed by recent work (Shi et al., 2023), to the supervised kernel learning scenario, providing both theoretical adaptivity (i.e., the capability to recover the correct model order and feature set) and practical tractability.

Hierarchical priors, such as the multiplicative gamma process, furnish automatic complexity control, and variational inference ensures feasibility on modern datasets. This approach aligns with ongoing efforts to merge scalable nonparametric Bayesian inference with expressive structured models, opening routes to deeper architectures, further integration with non-Gaussian likelihoods, and applications in high-dimensional structured prediction and scientific domains.

Table: Core Components and Their Roles

Component	Role/Function	Impact on Learning
Factor matrices $W^{(d)}$	Parameterization of weight tensor	Efficient, compressed modeling
Rank precisions $\lambda_R$	Sparsity prior over latent components	Automatic rank selection
Feature precisions $\lambda_{M_d}$	Sparsity prior per feature/mode	Feature selection and interpretability
Noise precision $\tau$	Observation noise variance	Calibrated uncertainty
Mean-field variational inference	Posterior approximation	Scalable, uncertainty-aware learning

Summary

Bayesian Tensor Network Kernel Machines provide a scalable, interpretable, and uncertainty-quantified alternative to deterministic tensor network models by introducing hierarchical sparsity-inducing priors and variational Bayesian inference. The resultant framework accommodates automatic selection of both model rank and relevant features via data-driven posterior inference, delivering robust prediction and reliable uncertainty estimates with no greater computational burden than standard tensor network ALS approaches (Kilic et al., 15 Jul 2025). This model family represents a significant advance in the principled unification of Bayesian nonparametrics and tensor network machine learning.