Tensor Network-Constrained Kernel Machines as Gaussian Processes (2403.19500v1)

Abstract: Tensor Networks (TNs) have recently been used to speed up kernel machines by constraining the model weights, yielding exponential computational and storage savings. In this paper we prove that the outputs of Canonical Polyadic Decomposition (CPD) and Tensor Train (TT)-constrained kernel machines recover a Gaussian Process (GP), which we fully characterize, when placing i.i.d. priors over their parameters. We analyze the convergence of both CPD and TT-constrained models, and show how TT yields models exhibiting more GP behavior compared to CPD, for the same number of model parameters. We empirically observe this behavior in two numerical experiments where we respectively analyze the convergence to the GP and the performance at prediction. We thereby establish a connection between TN-constrained kernel machines and GPs.

• The paper demonstrates that TN-constrained kernel machines converge to Gaussian Processes in the limit of large TN ranks with i.i.d. Gaussian priors.
• The paper shows TT-constrained models achieve GP behavior faster than CPD models in higher dimensions due to their hierarchical architecture.
• The paper details how traditional regularization in MAP estimation approximates the log-prior, leading to efficient and scalable kernel learning.

Tensor Network-Enforced Kernel Machines Converge to Gaussian Processes

Introduction

In the field of kernel machines, the integration of Tensor Networks (TNs) as a mechanism for constraining model parameters has garnered attention for its effectiveness in mitigating computational and storage complexities. This paper presents a formal exploration into how TN-constrained kernel machines, more specifically those subjected to Canonical Polyadic Decomposition (CPD) and Tensor Train (TT) constraints, manifest as Gaussian Processes (GPs) when imbued with i.i.d. priors over their components. Beyond establishing this connection, the paper explores the implications of such a relationship, providing insights into model behavior, convergence properties, and potential practical applications.

Theoretical Foundations and Main Results

The paper sets the stage by introducing the necessary background on GPs, TNs, and their application in kernel machines. It succinctly articulates how TNs enable the representation of large tensors with a significantly reduced number of parameters, thus offering a solution to the curse of dimensionality often encountered in machine learning.

Convergence of CPD and TT Models to GPs

A pivotal contribution of this research is the formal proof that outputs of CPD and TT-constrained kernel machines converge to a GP in the limit of large TN rank. Specifically, for CPD-constrained models, the paper demonstrates that as the rank parameter $R$ tends to infinity, the models approximate a fully characterized GP. Similarly, TT-constrained models exhibit GP behavior as the TT ranks ($R_i$) grow. This convergence is rigorously proven under the condition that the components of TNs are subjected to i.i.d. Gaussian priors.

Comparative Analysis of CPD and TT Models

An insightful analysis presented in the paper asserts that TT-constrained models reach GP behavior more swiftly compared to CPD counterparts for an identical number of parameters, particularly in higher input dimensions. This is attributed to TT's hierarchical architecture, which, unlike CPD’s linear structure, captures an expansive range of model interactions.

Implications for Regularization and MAP Estimation

The research further extends to discuss the implications of these findings in the context of Maximum A Posteriori (MAP) estimation. Notably, it provides a nuanced view of regularization within TT and CPD-constrained kernel machines, elucidating how traditional regularization terms approximate the log-prior in a GP context as the model parameters increase.

Empirical Validation

The theoretical assertions are substantiated through two numerical experiments focusing on GP convergence and prediction behavior. The experiments illustrate the GP convergence for CPD and TT models and highlight the predictive capabilities as the number of parameters grows. These empirical findings align with the theoretical discourse, showcasing the models' tendency toward GP behavior with increased parameters and affirming the faster convergence of TT models in higher dimensions.

Conclusion and Future Directions

The paper conclusively establishes a clear linkage between TN-constrained kernel machines and GPs, elucidating the conditions under which such models approximate GPs and the inherent differences in convergence behavior between CPD and TT constraints. The resultant insights have profound implications, shedding light on model behavior, informing regularization strategies, and potentially influencing the design of scalable and efficient kernel-based learning algorithms. Looking forward, the foundational work laid in this paper paves the way for deeper exploration into TNs' role in enhancing the interpretability and performance of kernel machines, along with a broader application in tackling complex high-dimensional data challenges in machine learning.