Overspecified Two-Component Mixed Linear Regression

• Overspecified 2MLR is defined by fitting a redundant two-component linear mixture model to data, enabling analysis of latent structures beyond the true model.
• Key methodologies include per-observation lifting via convex programming, EM-based optimization with symmetry-breaking, and semiparametric approaches that handle known/unknown components.
• The literature shows that overspecification can yield exact recovery under strict geometric and regularization conditions, or lead to slower convergence and statistical limitations in high-dimensional settings.

Searching arXiv for recent and foundational papers on overspecified two-component mixed linear regression and related convex/EM formulations. Overspecified two-component mixed linear regression (2MLR) denotes settings in which a two-component mixture of linear regressions is fitted with more latent structure than the data-generating mechanism contains, or in which the optimization is made deliberately redundant. In the literature summarized here, this includes fitting a two-component mixture to data generated by a single linear regressor, introducing one vector variable per observation even though only two regression vectors exist, and semiparametric two-component formulations in which one component is treated as known (Kwon et al., 2018, Luo et al., 13 Aug 2025, Hand et al., 2016, Bordes et al., 2013). The resulting theory is heterogeneous: some overspecified formulations admit exact recovery, some exhibit global EM convergence, and some reveal sharp slowdowns or statistical floors that depend on symmetry, initialization, or high-dimensional sparsity.

1. Definitions and forms of overspecification

A standard 2MLR model observes $(x_i,y_i)$ and latent labels $z_i\in\{0,1\}$ such that

$$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$

with two unknown regressors $\beta_1^*,\beta_2^*\in\mathbb R^p$. Estimation is naturally measured up to label swap through

$$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$

which isolates the intrinsic non-identifiability of component labels (Chen et al., 2013).

Within this baseline, the literature uses “overspecified” in several distinct ways. One is optimization-level overspecification: a convex program introduces one vector $z_i\in\mathbb R^d$ per data point and then uses an $\ell_1$-style fusion penalty to force these variables to collapse to the true regressors (Hand et al., 2016). A second is model misspecification: the data are generated by a single linear model, but EM is asked to fit a two-component mixture anyway (Kwon et al., 2018, Luo et al., 13 Aug 2025). A third is semiparametric overspecification, where one component is fully known and the other is unknown, so the two-component model contains a built-in asymmetry that changes identifiability and estimation strategy (Bordes et al., 2013).

Form of overspecification	Representative formulation	Main guarantee
Per-observation lifting	one vector variable $z_i\in\mathbb R^d$ for each data point	unique SOCP solution $z_i=\beta_{\ell_i}$ under well-separation, balance, and full span
Symmetric overfitting of a single regressor	fit $\pm \beta$ with equal weights	EM converges from random initialization
Overfitting with unknown weights	fit $z_i\in\{0,1\}$0 to a single-component model	linear convergence if initial weights are unbalanced; sublinear if perfectly balanced
Known/unknown semiparametric mixture	one component known, the other unknown	method-of-moments estimation in $z_i\in\{0,1\}$1 time

A recurring theme is that overspecification is not synonymous with non-identifiability. In several regimes, redundancy is absorbed either by geometric conditions, by symmetry-breaking in initialization, or by regularization that collapses superfluous degrees of freedom.

2. Convex overspecification by per-observation lifting

In the noiseless formulation of mixed linear regression over $z_i\in\{0,1\}$2 features, one observes $z_i\in\{0,1\}$3, $z_i\in\{0,1\}$4, with two unknown regression vectors $z_i\in\{0,1\}$5 and an unknown partition $z_i\in\{0,1\}$6 such that

$z_i\in\{0,1\}$7

The convex construction introduces one vector variable $z_i\in\{0,1\}$8 per observation and solves the SOCP

$z_i\in\{0,1\}$9

equivalently,

$$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$0

This is an extreme overspecification of the two-component model, because the parameterization contains $$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$1 vector variables rather than two regressors (Hand et al., 2016).

Exact recovery for $$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$2 is proved under three structural conditions. First, the measurements in each class must satisfy a well-separation inequality: $$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$3 where $$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$4, $$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$5, $$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$6, and $$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$7. Second, the data must satisfy the balance condition

$$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$8

Third, each class span must be full: $$y_i = z_i\cdot\langle x_i,\beta_1^*\rangle + (1-z_i)\cdot\langle x_i,\beta_2^*\rangle + e_i,\qquad i=1,\dots,n,$$9

Under these assumptions, the unique solution of the SOCP is

$\beta_1^*,\beta_2^*\in\mathbb R^p$0

The sample requirement is correspondingly geometric: each class must have at least $\beta_1^*,\beta_2^*\in\mathbb R^p$1 independent measurements, hence $\beta_1^*,\beta_2^*\in\mathbb R^p$2, and in total one needs at least $\beta_1^*,\beta_2^*\in\mathbb R^p$3 measurements. The overspecified variable set does not increase this bound, because the convex program still forces only two distinct $\beta_1^*,\beta_2^*\in\mathbb R^p$4’s. After solving the SOCP, one may run $\beta_1^*,\beta_2^*\in\mathbb R^p$5-means with any $\beta_1^*,\beta_2^*\in\mathbb R^p$6; the empirical $\beta_1^*,\beta_2^*\in\mathbb R^p$7’s lie in just two locations, so the true two clusters are recovered automatically.

The implementation route is iteratively reweighted least squares (IRLS). Since $\beta_1^*,\beta_2^*\in\mathbb R^p$8 is nonsmooth, iteration $\beta_1^*,\beta_2^*\in\mathbb R^p$9 forms

$$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$0

and solves

$$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$1

This makes the overspecified formulation computationally tractable while preserving the exact-recovery statement in the noiseless regime.

3. EM and global convergence in the symmetric overspecified model

A second overspecified regime arises when one fits a symmetric two-component mixture even though “one might have hoped only for a single linear fit.” In the analyzed model,

$$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$2

with isotropic Gaussian covariates $$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$3, Gaussian noise $$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$4, and signal-to-noise ratio

$$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$5

The fitted parameter is a single vector $$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$6, corresponding to symmetric components $$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$7. The paper emphasizes that EM behaves very differently in mixed linear regression from its behavior in Gaussian mixture models, and that it may fail to converge for three or more components, whereas the two-component case admits a global theory (Kwon et al., 2018).

The EM iteration has closed form. The E-step computes

$$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$8

and the M-step becomes

$$\rho((\hat\beta_1,\hat\beta_2),(\beta_1^*,\beta_2^*))= \min\{\|\hat\beta_1-\beta_1^*\|_2+\|\hat\beta_2-\beta_2^*\|_2,\; \|\hat\beta_1-\beta_2^*\|_2+\|\hat\beta_2-\beta_1^*\|_2\},$$9

An “easy-EM” variant replaces $z_i\in\mathbb R^d$0 by the identity.

The population dynamics have a particularly rigid geometry. All iterates remain in the plane spanned by $z_i\in\mathbb R^d$1 and $z_i\in\mathbb R^d$2, and the population map has only five critical points: aside from the trivial zero, the only local maxima are $z_i\in\mathbb R^d$3; the origin is a local minimum, and there are two saddles orthogonal to $z_i\in\mathbb R^d$4. A common concern is that overspecification necessarily creates spurious local maxima. In this symmetric 2MLR model, that concern is contradicted by the critical-point structure: there is no spurious local maximum to trap EM.

The proof strategy tracks the angle between $z_i\in\mathbb R^d$5 and $z_i\in\mathbb R^d$6. The angle strictly decreases at every step unless the iterate is already perfectly aligned or orthogonal. Starting from random $z_i\in\mathbb R^d$7, in $z_i\in\mathbb R^d$8 population-EM steps one reaches angle $z_i\in\mathbb R^d$9, then in $\ell_1$0 more steps one reaches angle $\ell_1$1, after which classical local $\ell_1$2-contraction yields exponential convergence in norm. Overall,

$\ell_1$3

suffices for $\ell_1$4.

The finite-sample analysis uses sample splitting. With fresh $\ell_1$5 samples per iteration, sample-splitting EM achieves $\ell_1$6 in the same number of iterations, with high probability. In the high-SNR regime $\ell_1$7, one obtains $\ell_1$8-error $\ell_1$9, independent of $z_i\in\mathbb R^d$0. The practical conclusion drawn in the paper is explicit: overspecification is not fatal.

4. Balance-sensitive overspecification with unknown mixing weights

A later overspecified analysis considers the true single-component model

$z_i\in\mathbb R^d$1

but fits the full two-component Gaussian 2MLR model with parameters $z_i\in\mathbb R^d$2 and mixing weight $z_i\in\mathbb R^d$3. The observed log-likelihood is

$z_i\in\mathbb R^d$4

and EM updates the responsibilities

$z_i\in\mathbb R^d$5

followed by the closed-form M-step

$z_i\in\mathbb R^d$6

$z_i\in\mathbb R^d$7

$z_i\in\mathbb R^d$8

The paper studies the overspecified-no-separation regime by normalizing by $z_i\in\mathbb R^d$9 and setting $z_i=\beta_{\ell_i}$0 (Luo et al., 13 Aug 2025).

The population theorem exhibits a sharp dependence on initialization through the imbalance variable $z_i=\beta_{\ell_i}$1. If the initial mixture is unbalanced, $z_i=\beta_{\ell_i}$2, then after $z_i=\beta_{\ell_i}$3 EM iterations one has $z_i=\beta_{\ell_i}$4, where $z_i=\beta_{\ell_i}$5. If the initial guess is perfectly balanced, $z_i=\beta_{\ell_i}$6, then EM converges sublinearly, with

$z_i=\beta_{\ell_i}$7

required to drive $z_i=\beta_{\ell_i}$8. The proof sketch isolates two different recurrences: a linear-rate contraction $z_i=\beta_{\ell_i}$9 in the unbalanced case, and a cubic drop $\pm \beta$0 in the balanced case, which leads to $\pm \beta$1.

The finite-sample theorem fixes the mixing weights $\pm \beta$2 and again splits the theory by balance. For

$\pm \beta$3

the sufficiently unbalanced regime

$\pm \beta$4

yields

$\pm \beta$5

and final accuracy

$\pm \beta$6

In the sufficiently balanced regime

$\pm \beta$7

the algorithm takes

$\pm \beta$8

steps and reaches accuracy

$\pm \beta$9

This is one of the clearest statements in the overspecified literature that balance is not a benign symmetry. A sufficiently unbalanced initialization preserves the usual $z_i\in\{0,1\}$00-type statistical scale, whereas exact balance slows the optimization to sublinear convergence and lowers the finite-sample accuracy to a $z_i\in\{0,1\}$01 floor. The same paper also extends the population analysis to low SNR by perturbing the EM updates around $z_i\in\{0,1\}$02.

5. Noisy convex relaxations and minimax rates for 2MLR

Overspecified 2MLR is most naturally interpreted against the broader theory of noisy two-component mixed regression. A foundational convex approach lifts the two unknown regressors into

$z_i\in\{0,1\}$03

and then estimates $z_i\in\{0,1\}$04 rather than $z_i\in\{0,1\}$05 directly. Under arbitrary noise, the estimator solves

$z_i\in\{0,1\}$06

while under stochastic sub-Gaussian noise with known $z_i\in\{0,1\}$07, it solves

$z_i\in\{0,1\}$08

The regression vectors are then reconstructed from

$z_i\in\{0,1\}$09

(Chen et al., 2013).

The guarantees are sharp. Let

$z_i\in\{0,1\}$10

and assume $z_i\in\{0,1\}$11 with $z_i\in\{0,1\}$12. Under arbitrary noise, if $z_i\in\{0,1\}$13 and $z_i\in\{0,1\}$14, then with probability $z_i\in\{0,1\}$15,

$z_i\in\{0,1\}$16

and consequently

$z_i\in\{0,1\}$17

Exact recovery in the noiseless case follows by setting $z_i\in\{0,1\}$18, $z_i\in\{0,1\}$19.

The same paper supplies minimax lower bounds up to constants or log factors. If $z_i\in\{0,1\}$20, then for any estimator and any true $z_i\in\{0,1\}$21 separated by at least $z_i\in\{0,1\}$22, with arbitrary noise satisfying $z_i\in\{0,1\}$23,

$z_i\in\{0,1\}$24

Under stochastic Gaussian designs and noise, the upper bounds display three SNR-driven phases, and the lower bounds match these phases:

1. if $z_i\in\{0,1\}$25, any estimator has risk $z_i\in\{0,1\}$26;
2. if $z_i\in\{0,1\}$27, the risk is $z_i\in\{0,1\}$28;
3. if $z_i\in\{0,1\}$29, the risk is $z_i\in\{0,1\}$30.

These results are not an overspecification theorem in the same sense as the EM papers, but they provide the principal noisy 2MLR benchmark. They also identify several limitations and open directions: if $z_i\in\{0,1\}$31, identifiability requires additional structure such as sparsity; extension to $z_i\in\{0,1\}$32 is open; direct convex extensions may blow up in dimensionality; and heavy-tailed designs would require robustification such as truncation or $z_i\in\{0,1\}$33-estimation.

6. High-dimensional inference, sparse regimes, and alternative objectives

In high-dimensional 2MLR, Zhang–Ma–Cai–Li study the model

$z_i\in\{0,1\}$34

with $z_i\in\{0,1\}$35, $z_i\in\{0,1\}$36, Gaussian noise of variance $z_i\in\{0,1\}$37, unknown nondegenerate $z_i\in\{0,1\}$38, and both $z_i\in\{0,1\}$39 $z_i\in\{0,1\}$40-sparse. Because direct likelihood maximization is nonconvex and ill-posed when $z_i\in\{0,1\}$41, the method uses an $z_i\in\{0,1\}$42-penalized EM algorithm: $z_i\in\{0,1\}$43 with $z_i\in\{0,1\}$44. Under good initialization, a separation condition $z_i\in\{0,1\}$45, bounded eigenvalues of $z_i\in\{0,1\}$46, and sparsity $z_i\in\{0,1\}$47, after $z_i\in\{0,1\}$48 iterations one has, with probability at least $z_i\in\{0,1\}$49,

$z_i\in\{0,1\}$50

The same work constructs debiased estimators, proves asymptotic normality for individual coordinates, gives confidence intervals, and proposes a large-scale testing procedure for $z_i\in\{0,1\}$51 with asymptotic FDR control (Zhang et al., 2020).

A complementary sparse-regime analysis examines two $z_i\in\{0,1\}$52-sparse signals $z_i\in\{0,1\}$53, $z_i\in\{0,1\}$54, under Gaussian design, Gaussian noise, and latent Bernoulli labels. The paper identifies a large statistical-computational gap in the symmetric balanced regime

$z_i\in\{0,1\}$55

denoted SB-MLR. Information-theoretically, exact support recovery requires

$z_i\in\{0,1\}$56

and the scaling $z_i\in\{0,1\}$57 appears as the minimax threshold up to log factors. Algorithmically, however, low-degree-polynomial hardness shows that if

$z_i\in\{0,1\}$58

then any polynomial-time algorithm fails, and slightly below that threshold super-polynomial runtime $z_i\in\{0,1\}$59 is required. Outside SB-MLR, a simple thresholding algorithm,

$z_i\in\{0,1\}$60

recovers the joint support $z_i\in\{0,1\}$61 with high probability whenever

$z_i\in\{0,1\}$62

in $z_i\in\{0,1\}$63 time; after support recovery, one can use any dense two-component mixed-regression solver to recover $z_i\in\{0,1\}$64 (Arpino et al., 2023). This suggests that the hardest overspecified and symmetric cases are concentrated in a narrow regime rather than being generic across mixed regression.

An alternative objective comes from optimal transport. Lin et al. formulate Wasserstein Mixed Linear Regression (WMLR) as a nonconvex–strongly-concave saddle-point problem,

$z_i\in\{0,1\}$65

where the discriminator family is

$z_i\in\{0,1\}$66

and a quadratic penalty regularizes the $z_i\in\{0,1\}$67-variables. Gradient descent ascent with

$z_i\in\{0,1\}$68

finds an $z_i\in\{0,1\}$69-minimax-stationary point in

$z_i\in\{0,1\}$70

iterations. In the symmetric ground-truth case $z_i\in\{0,1\}$71, any stationary minimax solution whose reference vector $z_i\in\{0,1\}$72 is sufficiently correlated with $z_i\in\{0,1\}$73 is actually global, so $z_i\in\{0,1\}$74. The sample complexity is linear in dimension, with $z_i\in\{0,1\}$75 for gradient-uniform error $z_i\in\{0,1\}$76. The same framework also permits over-parameterizing beyond the true $z_i\in\{0,1\}$77: extra $z_i\in\{0,1\}$78-blocks are shrunk back to the reference $z_i\in\{0,1\}$79 unless a true component is activated by the data (Diamandis et al., 2021).

A different semiparametric overspecified variant assumes one component is fully known. After recentering, the model is

$z_i\in\{0,1\}$80

where $z_i\in\{0,1\}$81, $z_i\in\{0,1\}$82, and the support of $z_i\in\{0,1\}$83 contains at least three points spanning $z_i\in\{0,1\}$84. Identifiability follows from regressions for $z_i\in\{0,1\}$85 and $z_i\in\{0,1\}$86, and estimation is based on empirical moments

$z_i\in\{0,1\}$87

The resulting estimators $z_i\in\{0,1\}$88 are strongly consistent and asymptotically normal, all required sample moments are computable in $z_i\in\{0,1\}$89, and the unknown error c.d.f. admits a plug-in estimator $z_i\in\{0,1\}$90 with $z_i\in\{0,1\}$91-consistency in $z_i\in\{0,1\}$92. A weighted bootstrap then yields asymptotically valid uniform confidence bands (Bordes et al., 2013).

Taken together, these strands show that overspecified 2MLR is not a single phenomenon but a family of regimes. In some, redundancy can be neutralized by convex fusion, by symmetry-breaking, or by shrinkage in a minimax objective. In others, especially balanced sparse mixtures, the same symmetry produces genuine statistical-computational barriers. The unresolved issues identified across the literature include extension to more than two components, robustification beyond sub-Gaussian or Gaussian designs, and a sharper understanding of misspecification in the low-SNR and high-dimensional settings.