Linear VC Dimension in Classifier Design

• Linear VC Dimension is a measure of the capacity of linear classifiers, defined by the maximum number of points that can be shattered by affine hyperplanes.
• Margin-sensitive bounds refine this measure by linking the geometric margin to an exact Θ-bound, offering precise control over a model’s statistical capacity.
• Applications extend to structured models like tensor-network classifiers where reduced parameter counts and network structure lead to improved generalization.

Searching arXiv for relevant papers on linear VC dimension, hyperplane classifiers, and related linearized model classes. Linear VC dimension is the Vapnik–Chervonenkis dimension of hypothesis classes defined by linear decision rules, most classically affine hyperplanes in Euclidean space and, more generally, models whose predictions are linear in the input but whose parameterization imposes structural constraints. In its classical form, it measures the largest finite subset of $\mathbb R^n$ that can be shattered by classifiers of the form $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$; for the family of all oriented affine hyperplanes in $\mathbb R^n$, the VC dimension is $n+1$ (Jayadeva et al., 2014). More recent work studies how this capacity changes under margin constraints, exact $\Theta$-bounds, tensor-network parameterizations, zero sets of linear combinations, and algebraic parameter spaces (Jayadeva, 2014, Khavari et al., 2021, Guingona et al., 2021, Pardo et al., 15 Apr 2025).

1. Classical definition for linear hyperplane classes

Let $X=\mathbb R^n$ and let $\mathcal H$ denote the class of all oriented affine hyperplanes

$$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$

The VC dimension $\gamma(\mathcal H)$ is the cardinality of the largest finite set $S\subset X$ that can be shattered by $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$0, where shattering means that for every labeling $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$1 there exists a hyperplane $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$2 such that $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$3 for all $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$4 (Jayadeva et al., 2014).

For the family of all affine hyperplanes in $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$5,

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$6

(Jayadeva et al., 2014). In the same vein, a set of points in $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$7 can be shattered by an affine hyperplane provided $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$8 general position points (Jayadeva et al., 2015). This is the baseline notion of linear VC dimension: absent further restrictions, capacity is determined by ambient feature dimension.

The classical viewpoint also underlies the comparison classes used in later structured models. In particular, for a full linear separator in $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$9, one has $\mathbb R^n$0 (Khavari et al., 2021). This establishes the standard benchmark against which reduced-capacity linear parameterizations are measured.

2. Margin-sensitive bounds and the exact $\mathbb R^n$1-bound

For hyperplane classifiers trained on $\mathbb R^n$2 with $\mathbb R^n$3 and $\mathbb R^n$4, the geometric margin is

$\mathbb R^n$5

and if $\mathbb R^n$6, then Vapnik’s classical fat-margin bound gives

$\mathbb R^n$7

(Jayadeva, 2014). A small VC dimension implies better worst-case generalization; the same source states the probabilistic error bound

$\mathbb R^n$8

(Jayadeva, 2014).

A more refined construction replaces this one-sided estimate by a two-sided, exact $\mathbb R^n$9-bound. For a separating hyperplane $n+1$0, define

$n+1$1

Then there exist constants $n+1$2 such that

$n+1$3

so that $n+1$4 (Jayadeva, 2014, Jayadeva et al., 2014, Jayadeva et al., 2014). In this formulation, $n+1$5 is an exact bound on the VC dimension in the sense that it controls $n+1$6 from above and below up to multiplicative constants.

The significance of this bound is conceptual as well as technical. The classical margin bound is one-sided; by contrast, the $n+1$7 characterization is presented as a tight capacity descriptor for separating hyperplanes. This suggests that linear VC dimension is not exhausted by ambient dimension alone: once margin geometry is incorporated, different linear separators over the same feature space can have substantially different effective statistical capacity.

3. Minimal Complexity Machine and direct VC-dimension minimization

The exact $n+1$8-bound leads to an optimization principle: minimize $n+1$9 in order to minimize the VC-dimension bound. In the hard-margin case, the resulting Linear Minimum Complexity Machine formulation is

$\Theta$0

subject to

$\Theta$1

and

$\Theta$2

(Jayadeva et al., 2014). For non-separable data, slack variables $\Theta$3 and a regularization weight $\Theta$4 yield

$\Theta$5

subject to

$\Theta$6

(Jayadeva et al., 2014). Each of these is a linear program in the variables $\Theta$7, so no quadratic or second-order cone programming is needed (Jayadeva et al., 2014).

The same idea extends to regression. Using the mapping of $\Theta$8-regression in $\Theta$9 to classification in $X=\mathbb R^n$0, a separating hyperplane

$X=\mathbb R^n$1

induces the regressor

$X=\mathbb R^n$2

and the MCM regressor again minimizes the exact bound $X=\mathbb R^n$3 through a linear program (Jayadeva et al., 2014). The paper states that the resulting LP finds a hyperplane that is provably of minimal capacity among all that fit the data, or trade off error via $X=\mathbb R^n$4 (Jayadeva et al., 2014).

A common misconception is that large-margin optimization automatically yields direct VC-dimension control. The comparison drawn in the source material is narrower. SVMs minimize $X=\mathbb R^n$5 or maximize minimum margin and thereby obtain a one-sided VC bound, but the VC dimension of SVMs can be very large or unbounded (Jayadeva et al., 2014, Jayadeva, 2014). By contrast, MCM explicitly minimizes a quantity whose square is an exact $X=\mathbb R^n$6-bound on VC dimension.

This capacity-minimization viewpoint was also used for feature selection. Because for any separating hyperplane in $X=\mathbb R^n$7 one has the classical bound $X=\mathbb R^n$8, minimizing the exact bound $X=\mathbb R^n$9 was used to drive many weight coordinates to zero, with the support of $\mathcal H$0 taken as the selected feature subset (Jayadeva et al., 2014). On ten gene-expression and artificial datasets, the linear MCM selected typically $\mathcal H$1–$\mathcal H$2 features out of thousands, approximately $\mathcal H$3–$\mathcal H$4 of the original dimension, while competing filters such as ReliefF and FCBF retained $\mathcal H$5–$\mathcal H$6 features on the same problems (Jayadeva et al., 2014).

4. Structured linear models: tensor-network VC dimension

A second major development concerns linear models whose weight tensors are constrained by tensor-network parameterizations. Let $\mathcal H$7 be a tensor-network graph whose dangling edges $\mathcal H$8 correspond to tensor modes of sizes $\mathcal H$9, and define

$$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$0

the total number of free parameters in the core tensors (Khavari et al., 2021). For weight tensors $$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$1, the paper studies three linear hypothesis classes:

• completion: $$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$2,
• regression: $$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$3,
• classification: $$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$4 (Khavari et al., 2021).

The general upper bound is

$$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$5

(Khavari et al., 2021). The key proof idea is that any $$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$6 is a degree-$$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$7 polynomial in the $$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$8 entries of the core tensors, after which Warren’s lemma bounds the number of sign patterns on $$h_{w,b}(x)=\operatorname{sign}(w^\top x+b),\qquad w\in\mathbb R^n,\ b\in\mathbb R.$$9 samples (Khavari et al., 2021).

The same work gives lower bounds for common tensor decompositions, showing that the upper bound is tight up to a factor $\gamma(\mathcal H)$0 (Khavari et al., 2021).

Model	Stated lower bound on VC, Pdim	Assumption
Rank-one CP	$\gamma(\mathcal H)$1	—
CP of rank $\gamma(\mathcal H)$2	$\gamma(\mathcal H)$3	$\gamma(\mathcal H)$4
Tucker of multilinear rank $\gamma(\mathcal H)$5	$\gamma(\mathcal H)$6	$\gamma(\mathcal H)$7
TT or TR of rank $\gamma(\mathcal H)$8	$\gamma(\mathcal H)$9	$S\subset X$0

For TT or TR, an additional bound is stated: if $S\subset X$1 and $S\subset X$2 is a multiple of $S\subset X$3, then

$S\subset X$4

(Khavari et al., 2021).

These results place linear VC dimension in a parameterization-sensitive regime. Although the predictor remains linear in the ambient tensor input, its effective capacity is governed by $S\subset X$5 and graph structure rather than the ambient product dimension $S\subset X$6. The source explicitly contrasts this with a full linear separator in $S\subset X$7, whose VC dimension is $S\subset X$8 (Khavari et al., 2021).

5. The matrix-product-state classifier and uniform generalization

For the Tensor-Train, or Matrix-Product-State, classifier with a $S\subset X$9 tensor of TT-rank $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$00 and $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$01 cores, the tensor-network parameters satisfy

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$02

Hence

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$03

(Khavari et al., 2021). Combined with the lower bound $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$04, the paper states

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$05

up to the $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$06 factor (Khavari et al., 2021). This is presented as resolving the open problem of Cirac–Garre-Rubio–Pérez-García on the statistical capacity of the MPS classifier of Stoudenmire–Schwab (Khavari et al., 2021).

The same framework yields a uniform convergence bound. For a $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$07-bounded loss $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$08 and an i.i.d. sample $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$09, with probability at least $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$10 over $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$11, every $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$12 satisfies

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$13

(Khavari et al., 2021). By substitution, the corollaries include:

• low-rank matrix: generalization gap $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$14,
• TT classifier: gap $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$15 (Khavari et al., 2021).

In contrast, a full linear separator in $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$16 gives a gap $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$17 (Khavari et al., 2021). The significance is precise: bond dimensions act as explicit capacity-control parameters, so the linear predictor in the ambient space can have exponentially reduced VC dimension when the rank structure is restricted.

6. Zero sets, Littlestone dimension, and algebraic parameter spaces

Linear VC dimension also appears in classes defined by zero sets of linear combinations. Let $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$18 be any field and let $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$19 be linearly independent. Define

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$20

Then

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$21

(Guingona et al., 2021). The result identifies a setting in which VC dimension and Littlestone dimension coincide exactly, and the same paper characterizes when $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$22 is VC-maximal of dimension $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$23: this occurs iff $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$24 is not contained in a finite union of proper subspaces of $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$25 (Guingona et al., 2021).

Low-dimensional examples make the pattern concrete. For $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$26, two linearly independent functions yield $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$27; for $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$28, one gets $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$29; and for conic sections in $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$30, where

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$31

the class of all nontrivial real conics has

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$32

(Guingona et al., 2021).

A more recent algebraic-geometric extension relates VC dimension to Krull dimension for parameterized constructible classifiers. For $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$33 and parameter variety $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$34, writing

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$35

the key estimate is

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$36

(Pardo et al., 15 Apr 2025). Equivalently,

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$37

and the source gives the explicit form

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$38

(Pardo et al., 15 Apr 2025).

Applied to neural networks with rational activation function, if a network has depth $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$39, total size $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$40, space $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$41, rational activation $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$42 of degree $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$43, and parameter-variety $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$44 of Krull dimension $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$45 and LCI-degree $h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$46, then the induced classifier family satisfies

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$47

and in particular

$h_{w,b}(x)=\operatorname{sign}(w^\top x+b)$48

(Pardo et al., 15 Apr 2025). This suggests a broad organizing principle: for many linear or linearized classifier families, statistical capacity can be governed by intrinsic parameter-space dimension up to logarithmic factors, rather than by ambient representation size alone.