---
title: Linear VC Dimension in Classifier Design
url: https://www.emergentmind.com/topics/linear-vc-dimension
type: topic
---

# Linear VC Dimension in Classifier Design

Searching arXiv for relevant papers on linear VC dimension, hyperplane classifiers, and related linearized model classes.
Linear VC dimension is the Vapnik–Chervonenkis dimension of hypothesis classes defined by linear decision rules, most classically affine hyperplanes in Euclidean space and, more generally, models whose predictions are linear in the input but whose parameterization imposes structural constraints. In its classical form, it measures the largest finite subset of $\mathbb R^n$ that can be shattered by classifiers of the form $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$; for the family of all oriented affine hyperplanes in $\mathbb R^n$, the VC dimension is $n+1$ [1410.7372]. More recent work studies how this capacity changes under margin constraints, exact $\Theta$-bounds, tensor-network parameterizations, zero sets of linear combinations, and algebraic parameter spaces [1408.2803] [2106.11827] [2109.04805] [2504.11345].

## 1. Classical definition for linear hyperplane classes

Let $X=\mathbb R^n$ and let $\mathcal H$ denote the class of all oriented affine hyperplanes
$$
h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.
$$
The VC dimension $\gamma(\mathcal H)$ is the cardinality of the largest finite set $S\subset X$ that can be shattered by $\mathcal H$, where shattering means that for every labeling $y:S\to\{+1,-1\}$ there exists a hyperplane $h_{w,b}\in\mathcal H$ such that $\operatorname{sign}(w^\top x+b)=y(x)$ for all $x\in S$ [1410.7372].

For the family of all affine hyperplanes in $\mathbb R^n$,
$$
\gamma(\mathcal H)=n+1
$$
[1410.7372]. In the same vein, a set of points in $\mathbb R^n$ can be shattered by an affine hyperplane provided $n+1$ general position points [1503.03148]. This is the baseline notion of linear VC dimension: absent further restrictions, capacity is determined by ambient feature dimension.

The classical viewpoint also underlies the comparison classes used in later structured models. In particular, for a full linear separator in $\mathbb R^D$, one has $\mathrm{VC}=D$ [2106.11827]. This establishes the standard benchmark against which reduced-capacity linear parameterizations are measured.

## 2. Margin-sensitive bounds and the exact $\Theta$-bound

For hyperplane classifiers trained on $\{(x^i,y_i)\}_{i=1}^M$ with $x^i\in\mathbb R^n$ and $y_i\in\{+1,-1\}$, the geometric margin is
$$
d=\min_i y_i(w^\top x^i+b)\,\|w\|^{-1},
$$
and if $R=\max_i\|x^i\|$, then Vapnik’s classical fat-margin bound gives
$$
\gamma \le 1+\min\{R^2/d^2,\ n\}
$$
[1408.2803]. A small VC dimension implies better worst-case generalization; the same source states the probabilistic error bound
$$
P_{\mathrm{test}} \le P_{\mathrm{train}} + O\bigl(\sqrt{\gamma/M}\bigr)
$$
[1408.2803].

A more refined construction replaces this one-sided estimate by a two-sided, exact $\Theta$-bound. For a separating hyperplane $u^\top x+v=0$, define
$$
h=
\frac{\max_i y_i(u^\top x^i+v)}
{\min_i y_i(u^\top x^i+v)}.
$$
Then there exist constants $\alpha,\beta>0$ such that
$$
\alpha h^2 \le \gamma \le \beta h^2,
$$
so that $\gamma=\Theta(h^2)$ [1408.2803] [1410.7372] [1410.4573]. In this formulation, $h^2$ is an exact bound on the VC dimension in the sense that it controls $\gamma$ from above and below up to multiplicative constants.

The significance of this bound is conceptual as well as technical. The classical margin bound is one-sided; by contrast, the $h^2$ characterization is presented as a tight capacity descriptor for separating hyperplanes. This suggests that linear VC dimension is not exhausted by ambient dimension alone: once margin geometry is incorporated, different linear separators over the same feature space can have substantially different effective statistical capacity.

## 3. Minimal Complexity Machine and direct VC-dimension minimization

The exact $\Theta$-bound leads to an optimization principle: minimize $h$ in order to minimize the VC-dimension bound. In the hard-margin case, the resulting Linear Minimum Complexity Machine formulation is
$$
\min_{w,b,h}\ h
$$
subject to
$$
y^i(w^\top x^i+b)\ge 1,\qquad i=1,\dots,M,
$$
and
$$
y^i(w^\top x^i+b)\le h,\qquad i=1,\dots,M
$$
[1410.7372]. For non-separable data, slack variables $q^i\ge 0$ and a regularization weight $C>0$ yield
$$
\min_{w,b,h,q}\ h+C\sum_i q^i
$$
subject to
$$
y^i(w^\top x^i+b)+q^i\ge 1,\qquad
y^i(w^\top x^i+b)+q^i\le h,\qquad
q^i\ge 0
$$
[1410.7372]. Each of these is a linear program in the variables $\{w,b,h,q^i\}$, so no quadratic or second-order cone programming is needed [1410.7372].

The same idea extends to regression. Using the mapping of $\varepsilon$-regression in $\mathbb R^n$ to classification in $\mathbb R^{n+1}$, a separating hyperplane
$$
w^\top x+\eta y+b=0
$$
induces the regressor
$$
y(x)=-(w^\top x+b)/\eta,
$$
and the MCM regressor again minimizes the exact bound $h$ through a linear program [1410.4573]. The paper states that the resulting LP finds a hyperplane that is provably of minimal capacity among all that fit the data, or trade off error via $C$ [1410.4573].

A common misconception is that large-margin optimization automatically yields direct VC-dimension control. The comparison drawn in the source material is narrower. SVMs minimize $\|w\|^2$ or maximize minimum margin and thereby obtain a one-sided VC bound, but the VC dimension of SVMs can be very large or unbounded [1410.4573] [1408.2803]. By contrast, MCM explicitly minimizes a quantity whose square is an exact $\Theta$-bound on VC dimension.

This capacity-minimization viewpoint was also used for feature selection. Because for any separating hyperplane in $\mathbb R^n$ one has the classical bound $\gamma\le n+1$, minimizing the exact bound $h^2$ was used to drive many weight coordinates to zero, with the support of $w$ taken as the selected feature subset [1410.7372]. On ten gene-expression and artificial datasets, the linear MCM selected typically $32$–$132$ features out of thousands, approximately $0.6\%$–$10\%$ of the original dimension, while competing filters such as ReliefF and FCBF retained $500$–$9\,000$ features on the same problems [1410.7372].

## 4. Structured linear models: tensor-network VC dimension

A second major development concerns linear models whose weight tensors are constrained by tensor-network parameterizations. Let $G=(V,E,\dim)$ be a tensor-network graph whose dangling edges $\delta_G$ correspond to tensor modes of sizes $d_1,\dots,d_p$, and define
$$
\|G\|\coloneqq \sum_{v\in V}\prod_{e\ni v}\dim(e),
$$
the total number of free parameters in the core tensors [2106.11827]. For weight tensors $W\in T(G)$, the paper studies three linear hypothesis classes:
- completion: $h(i_1,\dots,i_p)=W_{i_1\dots i_p}$,
- regression: $h(X)=\langle W,X\rangle$,
- classification: $h(X)=\operatorname{sign}\langle W,X\rangle$ [2106.11827].

The general upper bound is
$$
\mathrm{VC}(H_G^{\mathrm{cla}}),\quad
\mathrm{Pdim}(H_G^{\mathrm{reg}}),\quad
\mathrm{Pdim}(H_G^{\mathrm{comp}})
\le 2\,\|G\|\,\log(12\,|V|)
$$
[2106.11827]. The key proof idea is that any $h\in H_G^{\mathrm{reg}}$ is a degree-$|V|$ polynomial in the $\|G\|$ entries of the core tensors, after which Warren’s lemma bounds the number of sign patterns on $n$ samples [2106.11827].

The same work gives lower bounds for common tensor decompositions, showing that the upper bound is tight up to a factor $O(\log |V|)$ [2106.11827].

| Model | Stated lower bound on VC, Pdim | Assumption |
|---|---:|---|
| Rank-one CP | $(d-1)p$ | — |
| CP of rank $r$ | $rd$ | $r\le d^{p-1}$ |
| Tucker of multilinear rank $r$ | $r^p$ | $r\le d$ |
| TT or TR of rank $r$ | $r^2 d$ | $r\le d^{\lfloor (p-1)/2\rfloor}$ |

For TT or TR, an additional bound is stated: if $r=d$ and $p$ is a multiple of $3$, then
$$
\mathrm{VC},\mathrm{Pdim}\ge (p/3)(d^3-1)=(p/3)(r^2d-1)
$$
[2106.11827].

These results place linear VC dimension in a parameterization-sensitive regime. Although the predictor remains linear in the ambient tensor input, its effective capacity is governed by $\|G\|$ and graph structure rather than the ambient product dimension $d_1\cdots d_p$. The source explicitly contrasts this with a full linear separator in $\mathbb R^{d^p}$, whose VC dimension is $d^p$ [2106.11827].

## 5. The matrix-product-state classifier and uniform generalization

For the Tensor-Train, or Matrix-Product-State, classifier with a $d\times\cdots\times d$ tensor of TT-rank $r$ and $p$ cores, the tensor-network parameters satisfy
$$
\|G\|=pdr^2,\qquad |V|=p.
$$
Hence
$$
\mathrm{VC}(H_G^{\mathrm{cla}})\le 2pdr^2\log(12p)
$$
[2106.11827]. Combined with the lower bound $\mathrm{VC}\ge r^2d$, the paper states
$$
\mathrm{VC}(H_G^{\mathrm{cla}})=\Theta(pdr^2\cdot \log p)
$$
up to the $\log p$ factor [2106.11827]. This is presented as resolving the open problem of Cirac–Garre-Rubio–Pérez-García on the statistical capacity of the MPS classifier of Stoudenmire–Schwab [2106.11827].

The same framework yields a uniform convergence bound. For a $[0,1]$-bounded loss $\ell$ and an i.i.d. sample $S=\{(x_i,y_i)\}_{i=1}^{(n)}$, with probability at least $1-\delta$ over $S$, every $h\in H_G^{\mathrm{cla}}$ satisfies
$$
R(h)<\hat R_S(h)
+2\sqrt{(2/n)\bigl[\|G\|\log(8en|V|/\|G\|)+\ln(4/\delta)\bigr]}
$$
[2106.11827]. By substitution, the corollaries include:
- low-rank matrix: generalization gap $\lesssim \sqrt{r(d_1+d_2)\log n/n}$,
- TT classifier: gap $\lesssim \sqrt{pdr^2\log n/n}$ [2106.11827].

In contrast, a full linear separator in $\mathbb R^D$ gives a gap $\lesssim \sqrt{D\log n/n}$ [2106.11827]. The significance is precise: bond dimensions act as explicit capacity-control parameters, so the linear predictor in the ambient space can have exponentially reduced VC dimension when the rank structure is restricted.

## 6. Zero sets, Littlestone dimension, and algebraic parameter spaces

Linear VC dimension also appears in classes defined by zero sets of linear combinations. Let $\mathbb F$ be any field and let $f_1,\dots,f_d:X\to\mathbb F$ be linearly independent. Define
$$
\mathcal Z=
\left\{
Z_{(a_1,\dots,a_d)}:
Z_{(a_1,\dots,a_d)}=
\left\{x\in X:\sum_{i=1}^d a_i f_i(x)=0\right\},
\ (a_1,\dots,a_d)\neq 0
\right\}.
$$
Then
$$
\mathrm{VCdim}(\mathcal Z)=\mathrm{Ldim}(\mathcal Z)=d-1
$$
[2109.04805]. The result identifies a setting in which VC dimension and Littlestone dimension coincide exactly, and the same paper characterizes when $\mathcal Z$ is VC-maximal of dimension $d-1$: this occurs iff $f(X)$ is not contained in a finite union of proper subspaces of $\mathbb F^d$ [2109.04805].

Low-dimensional examples make the pattern concrete. For $d=2$, two linearly independent functions yield $\mathrm{VCdim}=\mathrm{Ldim}=1$; for $d=3$, one gets $\mathrm{VCdim}=\mathrm{Ldim}=2$; and for conic sections in $\mathbb R^2$, where
$$
f(x,y)=(x^2,xy,y^2,x,y,1)\in\mathbb R^6,
$$
the class of all nontrivial real conics has
$$
\mathrm{VCdim}=\mathrm{Ldim}=5
$$
[2109.04805].

A more recent algebraic-geometric extension relates VC dimension to Krull dimension for parameterized constructible classifiers. For $V\subseteq \mathbb A^N(K)\times\mathbb A^n(K)$ and parameter variety $\mathbb Z\subseteq \mathbb A^N(K)$, writing
$$
d=\mathrm{VCdim}(\mathcal H(V,\mathbb Z)),\quad
r=\dim_{\mathrm{Krull}}(\mathbb Z),\quad
g=\mathrm{grad}(V),\quad
D=\deg_{\mathrm{lci}}(\mathbb Z),
$$
the key estimate is
$$
2^d \le D(1+dg)^r
$$
[2504.11345]. Equivalently,
$$
d\le \log_2 D + r\log_2(1+dg),
$$
and the source gives the explicit form
$$
\dim_{\rm Krull}(\mathbb Z)
\ge
\frac{d-\log_2(\deg_{\rm lci}\mathbb Z)}
{\log_2(d)+1+\log_2(\mathrm{grad}(V))}
$$
[2504.11345].

Applied to neural networks with rational activation function, if a network has depth $\ell$, total size $L$, space $S$, rational activation $\phi(t)=p(t)/q(t)$ of degree $d$, and parameter-variety $A\subset\mathbb A^M$ of Krull dimension $s$ and LCI-degree $\Delta$, then the induced classifier family satisfies
$$
G(\mathcal H(N,A),m)\le \Delta\Bigl(1+2m(dS)^{c\ell+1}\Bigr)^s,\qquad c>0\text{ universal},
$$
and in particular
$$
d_{\mathrm{vc}}=O\bigl(s\log(dS)\bigr)
$$
[2504.11345]. This suggests a broad organizing principle: for many linear or linearized classifier families, statistical capacity can be governed by intrinsic parameter-space dimension up to logarithmic factors, rather than by ambient representation size alone.

Source: https://www.emergentmind.com/topics/linear-vc-dimension