Linear SEParability (LSEP)

• Linear SEParability (LSEP) is defined as the existence of a hyperplane that assigns all positive and negative examples to opposite sides, satisfying y_i(w^T x_i + b) > 0 for every i.
• Equivalent formulations, including convex-hull separation and pointwise criteria, underpin analyses of convergence rates (e.g., O(1/σ²)) and probabilistic thresholds in high-dimensional settings.
• Modern applications leverage LSEP in neural network diagnostic probes, multimodal reasoning, and diffusion model regularization to improve training performance and mitigate overfitting.

Linear SEParability (LSEP) is the property that labeled data admit a hyperplane that assigns all positive and negative examples to opposite sides. In its standard finite-sample form, for $S=\{(x_i,y_i)\}_{i=1}^n$ with $x_i\in\mathbb R^d$ and $y_i\in\{+1,-1\}$, LSEP means that there exist $w\in\mathbb R^d$ and $b\in\mathbb R$ such that $y_i(w^T x_i+b)>0$ for all $i$. Recent literature also uses the acronym for operational constructs built around this property, including probe-based diagnostics, margin-based formulations, and training regularizers in multimodal and diffusion models (Hajnal, 2023, Vompa et al., 10 Jul 2025, Yun et al., 25 Sep 2025).

1. Core definitions and equivalent formulations

The most direct definition is affine hyperplane separation. For binary data, separability without intercept requires a vector $\beta\in\mathbb R^p$ such that

$y_i(x_i^T\beta)>0 \quad \text{for all } i,$

whereas allowing an intercept $\beta_0\in\mathbb R$ replaces this by

$x_i\in\mathbb R^d$0

In either case, the sign of the linear score agrees with the label on every sample (Kuchelmeister, 2024).

For two disjoint finite subsets $x_i\in\mathbb R^d$1, an equivalent convex-geometric statement is

$x_i\in\mathbb R^d$2

where $x_i\in\mathbb R^d$3 denotes the convex hull. In planar computational geometry, the same equivalence appears as

$x_i\in\mathbb R^d$4

for bichromatic point sets $x_i\in\mathbb R^d$5 and $x_i\in\mathbb R^d$6 (Troost et al., 2018, Gutiérrez et al., 2016).

A pointwise version is also standard. A point $x_i\in\mathbb R^d$7 is linearly separable from a finite set $x_i\in\mathbb R^d$8 if there exists $x_i\in\mathbb R^d$9 such that

$y_i\in\{+1,-1\}$0

Equivalently, $y_i\in\{+1,-1\}$1. A finite sample $y_i\in\{+1,-1\}$2 is called $y_i\in\{+1,-1\}$3-convex, or linearly separable, when every $y_i\in\{+1,-1\}$4 is separable from the remainder; equivalently, all sample points are vertices of the convex hull (Sidorov et al., 2020).

A distinct but exact characterization appears in one-class semi-supervised learning. There, a measurable set with indicator $y_i\in\{+1,-1\}$5 is linearly separable iff it is maximal by probability among all sets with the same mean. In that framework, if

$y_i\in\{+1,-1\}$6

then $y_i\in\{+1,-1\}$7 is maximal by probability iff $y_i\in\{+1,-1\}$8 is linearly separable (Bauman et al., 2017). This suggests that LSEP can be characterized either by hyperplane geometry or by extremality in a constrained moment space.

2. Hyperplanes, margins, and dynamical formulations

Hajnal’s perceptron-based fine approximation technique rewrites binary separation as a one-class problem through three preprocessing steps: dimension extension,

$y_i\in\{+1,-1\}$9

negative inversion,

$w\in\mathbb R^d$0

and unit normalization,

$w\in\mathbb R^d$1

After these steps, the task is to find $w\in\mathbb R^d$2 such that $w\in\mathbb R^d$3 for all $w\in\mathbb R^d$4, eliminating explicit labels and the bias term from the online updates (Hajnal, 2023).

The same paper replaces the classical full perceptron correction by a “just enough” update. For a mistaken transformed sample $w\in\mathbb R^d$5, the geometric update is

$w\in\mathbb R^d$6

which forces $w\in\mathbb R^d$7. A numerically stable variant uses a small $w\in\mathbb R^d$8: $w\in\mathbb R^d$9 Under an ideal separator $b\in\mathbb R$0 with $b\in\mathbb R$1 and positive margin $b\in\mathbb R$2, each update increases alignment with $b\in\mathbb R$3, decreases $b\in\mathbb R$4, and monotonically decreases the angle to the true separator. The paper states convergence in direction to $b\in\mathbb R$5; with a strictly positive margin $b\in\mathbb R$6, a finite termination bound of order $b\in\mathbb R$7 can be shown (Hajnal, 2023).

A max-margin matrix-game formulation expresses LSEP as

$b\in\mathbb R$8

where the rows of $b\in\mathbb R$9 are normalized examples and $y_i(w^T x_i+b)>0$0 is the normalized margin of the best separator (Kornowski et al., 2024).

Margin preservation under linear compression has also been formalized. If a data-generating distribution is linearly separable with margin $y_i(w^T x_i+b)>0$1, and a linear map $y_i(w^T x_i+b)>0$2 preserves inner products on the support up to distortion

$y_i(w^T x_i+b)>0$3

then separability is maintained whenever

$y_i(w^T x_i+b)>0$4

The proof proceeds via hard-SVM geometry on the convex hulls of the two classes (McVay et al., 2022).

A related dynamical result concerns logistic gradient flow at the edge of separability. When data are separable from the origin, the direction of the iterate converges to the hard-margin SVM solution, and the asymptotic form is

$y_i(w^T x_i+b)>0$5

When data are not separable from the origin, the asymptotic regime differs, and generalization can eventually become perfect despite early overfitting (Beck et al., 2024).

3. Probabilistic geometry and high-dimensional thresholds

For points sampled i.i.d. uniformly from the spherical layer

$y_i(w^T x_i+b)>0$6

stochastic separation theorems quantify how likely LSEP is in high dimension. If $y_i(w^T x_i+b)>0$7 denotes the probability that a fresh point is linearly separable from an $y_i(w^T x_i+b)>0$8-point sample, then

$y_i(w^T x_i+b)>0$9

If $i$0 denotes the probability that every point in the sample is separable from the rest, then

$i$1

Hence point-to-set separability persists for $i$2, whereas full-sample separability persists for $i$3 at fixed confidence. The lower bounds do not depend on the layer-thickness parameter $i$4 (Sidorov et al., 2020).

For Gaussian features and labels depending on one marginal of the features, the separability probability can be expressed through the intrinsic volumes of the polyhedral cone

$i$5

where $i$6. In that setting,

$i$7

admits both a kinematic-formula representation as a sum of intrinsic volumes $i$8 and an explicit mixture over the number $i$9 of correctly signed margins. The same framework yields finite-sample upper bounds, including

$\beta\in\mathbb R^p$0

under the stated condition involving $\beta\in\mathbb R^p$1, $\beta\in\mathbb R^p$2, $\beta\in\mathbb R^p$3, and $\beta\in\mathbb R^p$4, and the sharp sign-flip-noise bounds

$\beta\in\mathbb R^p$5

At $\beta\in\mathbb R^p$6, these recover Cover’s identity (Kuchelmeister, 2024).

A separate threshold result uses Wendel’s theorem for Gaussian points and separability from the origin. If $\beta\in\mathbb R^p$7 with $\beta\in\mathbb R^p$8, then the probability of separability undergoes a sharp transition at

$\beta\in\mathbb R^p$9

Below $y_i(x_i^T\beta)>0 \quad \text{for all } i,$0, the origin lies in the convex hull with high probability; above $y_i(x_i^T\beta)>0 \quad \text{for all } i,$1, the sample is separable from the origin. Near $y_i(x_i^T\beta)>0 \quad \text{for all } i,$2, delayed generalization and non-monotonic test loss appear, and the time gap between memorization and generalization can diverge (Beck et al., 2024).

These results support two different but compatible themes. First, high dimension can be a “blessing of dimensionality” for random data (Sidorov et al., 2020). Second, separability thresholds can also mark regimes of unstable optimization and prolonged overfitting (Beck et al., 2024).

4. Algorithms and complexity regimes

In online learning, the classical perceptron and Hajnal’s fine-approximation variant both process one sample at a time, each update costing $y_i(x_i^T\beta)>0 \quad \text{for all } i,$3 for the dot product and $y_i(x_i^T\beta)>0 \quad \text{for all } i,$4 for the weight adjustment. The proposed variant often needs fewer updates when $y_i(x_i^T\beta)>0 \quad \text{for all } i,$5, because each correction is “just enough” to remove the current mistake. Empirically, Hajnal reported three synthetic cases: when $y_i(x_i^T\beta)>0 \quad \text{for all } i,$6, the perceptron converged in about $y_i(x_i^T\beta)>0 \quad \text{for all } i,$7 epochs versus about $y_i(x_i^T\beta)>0 \quad \text{for all } i,$8 for the approximation method; when $y_i(x_i^T\beta)>0 \quad \text{for all } i,$9, the perceptron hit the $\beta_0\in\mathbb R$0-epoch limit while the approximation converged in about $\beta_0\in\mathbb R$1 epochs; and when $\beta_0\in\mathbb R$2, the approximation again reached full accuracy in about $\beta_0\in\mathbb R$3 epochs while the perceptron did not converge by $\beta_0\in\mathbb R$4 epochs (Hajnal, 2023).

For planar spatial databases, an R-tree–based algorithm decides whether two point sets are linearly separable while loading only relevant data into main memory. The method alternates optimistic and pessimistic convex-hull tests over sets of minimum bounding rectangles, and it runs in

$\beta_0\in\mathbb R$5

time in the worst case. A companion algorithm computes the convex hull of $\beta_0\in\mathbb R$6 planar points given in an R-tree in $\beta_0\in\mathbb R$7 time (Gutiérrez et al., 2016).

For stochastic bichromatic data, Xue, Li, and Janardan define the separable-probability

$\beta_0\in\mathbb R$8

and the expected separation-margin

$\beta_0\in\mathbb R$9

Their witness-based framework yields exact algorithms and lower bounds (Xue et al., 2016).

Setting	Task	Complexity
Planar point sets in separate R-trees	Decide separability	$x_i\in\mathbb R^d$00
Stochastic bichromatic points, $x_i\in\mathbb R^d$01	Compute $x_i\in\mathbb R^d$02	$x_i\in\mathbb R^d$03
Stochastic bichromatic points, $x_i\in\mathbb R^d$04	Compute $x_i\in\mathbb R^d$05	$x_i\in\mathbb R^d$06
Stochastic bichromatic points, $x_i\in\mathbb R^d$07	Compute $x_i\in\mathbb R^d$08	$x_i\in\mathbb R^d$09
Stochastic polytopes or balls	Generalized $x_i\in\mathbb R^d$10, $x_i\in\mathbb R^d$11	$x_i\in\mathbb R^d$12, $x_i\in\mathbb R^d$13

A different algorithmic route appears in one-class semi-supervised learning. Given a target mean $x_i\in\mathbb R^d$14, the maximal-probability formulation reduces to the linear program

$x_i\in\mathbb R^d$15

When the true class is linearly separable, an optimal extreme solution recovers the desired subset (Bauman et al., 2017).

At the oracle-complexity level, the matrix-game formulation separates access models. Any deterministic algorithm using the one-sided oracle $x_i\in\mathbb R^d$16 requires

$x_i\in\mathbb R^d$17

iterations, matching the perceptron rate. Under the stronger two-sided oracle $x_i\in\mathbb R^d$18, any deterministic algorithm still requires

$x_i\in\mathbb R^d$19

iterations up to logarithmic factors (Kornowski et al., 2024).

5. Separability as a representation principle in neural networks

For finite multi-class data decomposed into disjoint convex-hull pieces,

$x_i\in\mathbb R^d$20

a constructive two-hidden-layer network can realize linear separation once the pieces are separated pairwise. In the two-class case with decomposition sizes $x_i\in\mathbb R^d$21 and $x_i\in\mathbb R^d$22, the required hidden-layer widths are $x_i\in\mathbb R^d$23 and $x_i\in\mathbb R^d$24, so the total number of hidden nodes is

$x_i\in\mathbb R^d$25

The result assumes an activation function that is non-constant, strictly increasing, and has a finite left asymptote; a separate argument covers leaky ReLU under a slope condition (Troost et al., 2018).

A more diagnostic line of work studies separability of hidden representations directly. For two finite sets $x_i\in\mathbb R^d$26, the Minkowski difference

$x_i\in\mathbb R^d$27

satisfies

$x_i\in\mathbb R^d$28

From this, Minkowski-difference based linear separability measures (MD-LSMs) define continuous degrees of separation. One of them is

$x_i\in\mathbb R^d$29

where $x_i\in\mathbb R^d$30 and $x_i\in\mathbb R^d$31 for the difference matrix $x_i\in\mathbb R^d$32. The paper states that $x_i\in\mathbb R^d$33, $x_i\in\mathbb R^d$34, and $x_i\in\mathbb R^d$35 each lie in $x_i\in\mathbb R^d$36 and equal $x_i\in\mathbb R^d$37 exactly when $x_i\in\mathbb R^d$38 and $x_i\in\mathbb R^d$39 are linearly separable. Across MLP, CNN, DBN, ResNet, VGGNet, AlexNet, vision transformer, and GoogLeNet, hidden-layer separability rises in lock-step with training performance, layers closer to the output end show higher linear separability, and increasing width or depth improves the chance of higher $x_i\in\mathbb R^d$40 (Zhang et al., 2023).

In continual learning, Backward Feature Projection (BFP) preserves old-class LSEP by allowing new features $x_i\in\mathbb R^d$41 to match old features $x_i\in\mathbb R^d$42 up to a trainable linear map $x_i\in\mathbb R^d$43. The feature loss is

$x_i\in\mathbb R^d$44

If old classes were separated by $x_i\in\mathbb R^d$45 and $x_i\in\mathbb R^d$46, then the new space preserves separation through $x_i\in\mathbb R^d$47. Empirically, BFP yields higher linear-probe accuracies and better stability–plasticity behavior than rigid feature matching (Gu et al., 2023).

6. Probe ceilings, multimodal reasoning, and diffusion regularization

In visual-LLMs, the Linear Separability Ceiling (LSC) is defined as the accuracy of a simple nearest-centroid linear classifier on visual embeddings. For a Bongard-style task with positive images $x_i\in\mathbb R^d$48, negative images $x_i\in\mathbb R^d$49, and query $x_i\in\mathbb R^d$50, the model first mean-pools token embeddings into normalized vectors $x_i\in\mathbb R^d$51, forms normalized class centroids

$x_i\in\mathbb R^d$52

and predicts by cosine similarity. The LSC is then

$x_i\in\mathbb R^d$53

The paper emphasizes that the LSC is not an absolute performance bound but an “observational ceiling” for methods relying solely on linear decision-making over raw embeddings (Vompa et al., 10 Jul 2025).

The same study reports that most baseline VLMs lie on or below the diagonal $x_i\in\mathbb R^d$54, indicating a “linear reasoning bottleneck,” while some models exceed it. It further distinguishes two success pathways: non-linear representation refinement, where $x_i\in\mathbb R^d$55, and post-representation reasoning on less separable embeddings, where $x_i\in\mathbb R^d$56. The intervention needed is task-dependent: postfix tuning provides evidence for dormant reasoning pathways on semantic concepts, whereas more complex relational reasoning requires adapting core model weights. The paper also notes that improving representation quality can still reduce out-of-distribution prompt robustness (Vompa et al., 10 Jul 2025). This suggests that linear separability of embeddings and end-to-end reasoning competence need not coincide.

In diffusion models, LSEP names a training regularizer that embeds a linear probe into the denoising network. If $x_i\in\mathbb R^d$57 is a pooled intermediate representation and $x_i\in\mathbb R^d$58, then the classification loss is

$x_i\in\mathbb R^d$59

and the total objective is

$x_i\in\mathbb R^d$60

Because the backbone is not frozen, gradients from $x_i\in\mathbb R^d$61 reshape the geometry of intermediate features toward class-wise linear separability (Yun et al., 25 Sep 2025).

Quantitatively, on ImageNet $x_i\in\mathbb R^d$62, the paper reports linear probing gains at $x_i\in\mathbb R^d$63K iterations and $x_i\in\mathbb R^d$64: layer $x_i\in\mathbb R^d$65 improves from $x_i\in\mathbb R^d$66 to $x_i\in\mathbb R^d$67, layer $x_i\in\mathbb R^d$68 from $x_i\in\mathbb R^d$69 to $x_i\in\mathbb R^d$70, and layer $x_i\in\mathbb R^d$71 from $x_i\in\mathbb R^d$72 to $x_i\in\mathbb R^d$73. For SiT-XL/2 without classifier-free guidance at $x_i\in\mathbb R^d$74K iterations, the baseline FID is $x_i\in\mathbb R^d$75, $x_i\in\mathbb R^d$76LSEP gives $x_i\in\mathbb R^d$77, $x_i\in\mathbb R^d$78REPA gives $x_i\in\mathbb R^d$79, and $x_i\in\mathbb R^d$80REPA+LSEP gives $x_i\in\mathbb R^d$81 (Yun et al., 25 Sep 2025).

Across these modern uses, the acronym remains anchored to the same geometric idea—a preference for hyperplane-separable representations—but the operational meaning differs. In some papers LSEP is the property to be analyzed; in others it is a ceiling, a probe, or a regularizer. A plausible implication is that contemporary research treats linear separability not only as a binary predicate on datasets, but also as a measurable resource for optimization, diagnostics, and architectural intervention.