SVM-STL: SVM for Supervised Tensor Learning

• SVM-STL is a matrix-based generalization of classical SVMs that retains intrinsic row and column structures in data like images or biomedical signals.
• It leverages a learnable weight matrix to define a matrix-induced inner product and employs matrix-valued kernels for both linear and nonlinear classification.
• An SMO-style optimization algorithm with proven theoretical convergence and generalization bounds adapts the method to structured input domains.

The SVM-STL ("Support Vector Machine for Supervised Tensor Learning") formulation, as codified in the Kernel Support Matrix Machine (KSMM), generalizes classical support vector machines to directly handle data naturally represented as matrices, such as images or biomedical signals. By defining a matrix-induced inner product via a learnable weight matrix and leveraging matrix-valued kernels, SVM-STL enables the construction of hyperplanes in matrix Hilbert spaces, capturing intrinsic data structure that would otherwise be lost by vectorization. The method includes theoretical guarantees and a SMO-style optimization algorithm, achieving generalization rates on par with standard SVMs, but adapted for matrix inputs (Ye, 2017).

1. Matrix-Induced Primal Formulation

Let $\{(X_i, y_i)\}_{i=1}^N$ be a set of training pairs with $X_i \in \mathbb{R}^{m \times n}$ and $y_i \in \{-1, +1\}$. The analysis is conducted within the matrix Hilbert space $\mathcal{H} = \mathbb{R}^{m \times n}$, equipped with the Frobenius inner product $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$. A symmetric weight matrix $V \in \mathbb{R}^{n \times n}$ ($V = V^\top$, $\|V\| \neq 0$) defines the matrix-induced inner product: $$\langle\!\langle X, Y \rangle\!\rangle_{\mathcal{H}, V} = \langle X^\top Y, V \rangle = \operatorname{tr}(V^\top X^\top Y)$$ and the associated norm

$$\|X\|_{\mathcal{H}(V)} = \sqrt{\langle\!\langle X, X \rangle\!\rangle_{\mathcal{H}, V/\|V\|}}$$

The KSMM corresponds to the following primal problem: $X_i \in \mathbb{R}^{m \times n}$0 The KSMM decision function is specified as: $X_i \in \mathbb{R}^{m \times n}$1 This approach maintains both row and column structural information of matrix-valued data in classification tasks (Ye, 2017).

2. Dual Problem and Representer Property

The Lagrangian for the primal includes dual variables $X_i \in \mathbb{R}^{m \times n}$2, $X_i \in \mathbb{R}^{m \times n}$3. Stationarity conditions yield: $X_i \in \mathbb{R}^{m \times n}$4 Substitution into the Lagrangian leads to the dual: $X_i \in \mathbb{R}^{m \times n}$5 where $X_i \in \mathbb{R}^{m \times n}$6 in the linear case.

The stationary condition further leads to the matrix representer theorem: the solution $X_i \in \mathbb{R}^{m \times n}$7 always resides in the span of the transformed matrices $X_i \in \mathbb{R}^{m \times n}$8, paralleling the classical SVM structure (Ye, 2017).

3. Matrix-Valued Kernels and Nonlinear Extensions

For nonlinear classification, a feature mapping $X_i \in \mathbb{R}^{m \times n}$9 is introduced with a matrix-valued reproducing kernel: $y_i \in \{-1, +1\}$0

Three kernel families are exemplified:

Kernel Type	Formula	Parameterization
Linear matrix kernel	$y_i \in \{-1, +1\}$1	$y_i \in \{-1, +1\}$2
Polynomial kernel	$y_i \in \{-1, +1\}$3	$y_i \in \{-1, +1\}$4
Gaussian (columnwise)	$y_i \in \{-1, +1\}$5	$y_i \in \{-1, +1\}$6

All kernels constructed in this manner are symmetric positive semidefinite $y_i \in \{-1, +1\}$7 matrices, ensuring the positive-definiteness requisite for the optimization framework (Ye, 2017).

4. Sequential Minimal Optimization Solver

KSMM employs a Sequential Minimal Optimization (SMO) strategy, updating two Lagrange multipliers per iteration:

• Identify $y_i \in \{-1, +1\}$8 violating its KKT conditions; select $y_i \in \{-1, +1\}$9 randomly
• Denote $\mathcal{H} = \mathbb{R}^{m \times n}$0, $\mathcal{H} = \mathbb{R}^{m \times n}$1
• Optimize the one-dimensional objective:

$\mathcal{H} = \mathbb{R}^{m \times n}$2

Using the first and second derivatives:

$\mathcal{H} = \mathbb{R}^{m \times n}$3

$\mathcal{H} = \mathbb{R}^{m \times n}$4

• Update $\mathcal{H} = \mathbb{R}^{m \times n}$5 via constrained Newton’s method:

$\mathcal{H} = \mathbb{R}^{m \times n}$6

• Update $\mathcal{H} = \mathbb{R}^{m \times n}$7
• Update bias $\mathcal{H} = \mathbb{R}^{m \times n}$8 according to standard pair-wise SVM update schemes

The dual objective is monotonically increased and convergence is asymptotically ensured (Ye, 2017).

5. Generalization Bound via Rademacher Complexity

For loss function $\mathcal{H} = \mathbb{R}^{m \times n}$9, where $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$0 is $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$1-Lipschitz in $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$2 and bounded by $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$3, the following generalization bound holds (Theorem 3.2): $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$4 where $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$5, $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$6 and $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$7 denote the expected and empirical risks, respectively. The bound exhibits an $$\langle A, B \rangle_\mathcal{H} = \operatorname{tr}(A^\top B)$$8 convergence rate, matching that of classical SVMs but leveraging the matrix-induced norm and thus the structured nature of the data (Ye, 2017).

6. Significance and Context

SVM-STL, realized by KSMM, extends the SVM paradigm to directly operate on matrix-structured data, eliminating the need for vectorization and retaining richer structural and contextual dependencies within samples. The formulation unifies and generalizes prior approaches such as support tensor machines, with improvements in computational tractability, robust kernelization, and theoretical guarantees via Rademacher bounds. The asymptotic convergence of the SMO-like algorithm and the adaptability to matrix-valued kernels position SVM-STL as a foundation for subsequent research in supervised tensor learning and related structured input domains (Ye, 2017).