Demixing Sparse Signals from Nonlinear Observations using Generalized Non-convex Regularization

Abstract: We consider the recovery of a pair of sparse vectors from a limited number of nonlinear observations of their superposition: $y_i=g(\inner{\ba_i}{\bPhi\bw<sup>\ast+\bPsi\bz<sup>\ast})+e_i$, $i=1,\dots,m$, with $m\ll n$, incoherent orthonormal bases $\bPhi,\bPsi$, a scalar link $g$, and noise $e_i$ that may be heavy-tailed or contaminated. We propose a regularization-based framework combining a Huberized data fidelity with generalized folded-concave penalties (SCAD, MCP), and a two-block proximal alternating algorithm with backtracking (NLD-PALM) whose whole iterate sequence provably converges to critical points under the Kurdyka--Łojasiewicz property, with local linear rates. On the statistical side we establish restricted strong convexity of the Huberized nonlinear loss through an exact sign-definite decomposition, and derive estimation error bounds of order $σ\sqrt{s\log(n)/m}$ that hold at \emph{every} localized stationary point, an oracle rate $σ\sqrt{s/m}$ free of $\log n$ and shrinkage bias under a beta-min condition, and a co-equal recovery theorem for \emph{unknown} monotone links via a linear surrogate and a clipped Plan--Vershynin decoupling. The estimator requires no knowledge of the sparsity levels, and its guarantees hold under symmetric noise with only finite variance. Experiments at $n=512$ under a frozen data-driven regularization rule show an earlier phase transition than convex $\ell_1$ demixing and greedy hard-thresholding baselines, a $35\times$ accuracy advantage over squared-loss estimation under $5\%$ gross outliers, and successful demixing of spike-plus-background signals observed through a saturating amplifier.

• The paper introduces a Huberized, folded-concave estimator that robustly demixes sparse signals from nonlinear and contaminated observations.
• It proposes the NLD-PALM algorithm, which guarantees convergence under the Kurdyka–Łojasiewicz property and exhibits local R-linear convergence.
• The study establishes statistical recovery bounds and demonstrates significant error improvements over traditional convex methods in the presence of outliers.

Demixing Sparse Signals from Nonlinear Observations using Generalized Non-convex Regularization

Problem Formulation and Motivation

The paper addresses the problem of sparse signal demixing from nonlinear observations, where the measurements are of the form $y_i = g({a_i}{\bPhi w^* + \bPsi z^*}) + e_i$ with $m \ll n$. Here, $\bPhi$ and $\bPsi$ are incoherent orthonormal bases, $g$ is a link function often corresponding to a nonlinear front-end (e.g., saturation, clipping, companding), and the noise term $e_i$ may be heavy-tailed or contaminated. The task is to recover both sparse components from significantly undersampled and potentially nonlinear data.

Traditional approaches are limited in this setting. Greedy algorithms such as Demixing Hard Thresholding (DHT) require exact sparsity knowledge and offer no robustness guarantees, while convex $\ell_1$ demixing suffers from shrinkage bias and fails to attain oracle performance, especially under strong noise or gross outliers.

This work proposes a Huberized, folded-concave regularized estimator for robust, bias-minimized demixing in both known- and unknown-link regimes, extending non-convex regularization theory (SCAD, MCP) to handle nonlinear and contaminated measurements.

Regularization Framework and Estimator

The estimator minimizes a Huberized, non-convex objective: $F(w, z) = \frac{1}{m} \sum_{i=1}^m h_H\left(y_i - g(a_i^\top (\bPhi w + \bPsi z))\right) + P_\lambda(w) + P_\lambda(z)$ where $h_H$ is the Huber loss (threshold $H$), and $m \ll n$0 belongs to the folded-concave penalty class (e.g., SCAD, MCP), ensuring unbiased recovery for large coefficients and superior oracle behavior compared to $m \ll n$1.

Huberization obviates compatibility conditions required by squared-loss analysis, ensuring concentration of empirical processes and robust behavior to only finite-variance symmetric noise. This is crucial in applications where gross outliers or heavy-tailed contamination are present.

Algorithmic Contributions: NLD-PALM

A core contribution is the Nonlinear Demixing Proximal Alternating Linearized Minimization (NLD-PALM) algorithm, a two-block proximal alternating scheme with per-block backtracking and over-relaxation parameter $m \ll n$2. The algorithm guarantees whole-iterate convergence to critical points under the Kurdyka–Łojasiewicz property, with local $m \ll n$3-linear convergence when RSC holds. The requirement $m \ll n$4 is necessary for sufficient decrease with non-convex penalties, distinguishing the method from fixed-step convex proximal algorithms.

Proximal mappings for SCAD, MCP, and $m \ll n$5 are closed-form, and the methodology ensures boundedness of iterates even under non-coercive penalties and loss functions.

Statistical Recovery Theory

Restricted strong convexity (RSC) of the Huberized loss is established via an exact sign-definite decomposition, ensuring curvature at every sample level. The key result is: $m \ll n$6 where $m \ll n$7 and $m \ll n$8 aggregates problem and penalty constants.

The main error bounds at stationary points demonstrate that for an appropriate choice of $m \ll n$9, the estimator achieves: $\bPhi$0 with $\bPhi$1 total sparsity, $\bPhi$2 ambient dimension, and $\bPhi$3 sample count. The oracle regime is established under a beta-min condition ($\bPhi$4 large enough), yielding $\bPhi$5 rates without $\bPhi$6 or shrinkage bias—unattainable by convex $\bPhi$7 demixing.

The estimator requires no knowledge of the sparsity levels, and its guarantees apply under symmetric noise with only finite variance.

Robustness and Unknown-Link Recovery

Huberization confers strong robustness to gross outliers and heavy tails. Experiments with $\bPhi$8 gross outlier contamination show $\bPhi$9 improvement in median relative error over squared-loss methods. The estimator's guarantees extend to unknown monotone links via a surrogate linear objective and clipped Plan–Vershynin decoupling, achieving comparable rates modulo a $\bPsi$0 irreducible floor induced by the nonlinearity.

Figure 1: Phase transition and median error versus sample size $\bPsi$1; SCAD achieves earlier transition, significantly lower error, and avoids shrinkage bias compared to $\bPsi$2 and DHT.

Numerical Experiments

Comprehensive experiments at $\bPsi$3 and various sparsity and noise regimes validate the theoretical findings. SCAD/MCP estimators attain phase transition approximately $\bPsi$4--$\bPsi$5 earlier than DHT, with SCAD's median error $\bPsi$6--$\bPsi$7 versus $\bPsi$8--$\bPsi$9 for DHT in scarce sampling regimes. $g$0 fails to attain exact recovery, instead plateauing at its shrinkage bias as asserted by theory.

Performance is robust across noise regimes: median relative error as low as $g$1 under $g$2 gross outliers and parity with squared-loss under clean noise. Measured error scales as $g$3 and $g$4 up to the information-theoretic floor.

A saturation-demixing application demonstrates the method's ability to successfully separate spikes and smooth backgrounds even under saturating amplifier nonlinearity and contamination.

Figure 2: Saturation demixing of spike-plus-smooth signals under nonlinear $g$5 observations and gross outliers; Huberized SCAD achieves relative error $g$6, outperforming squared-loss SCAD at $g$7.

Practical and Theoretical Implications

The findings substantiate that Huberized, folded-concave regularization is capable of robust, bias-minimized sparse demixing from highly noisy, nonlinear, and undersampled measurements. The theoretical guarantees hold at every localized stationary point, independent of sparsity knowledge, challenging the necessity of convexity in high-dimensional regression and signal recovery.

Practically, these methods enable recovery in compressed sensing, imaging, and source separation applications with nonlinear sensing front-ends. This is relevant for real-world acquisition and demixing tasks where saturation, quantization, and contamination are inherent.

Theoretically, the work opens new avenues for extending non-convex statistical theory to broader penalty classes ($g$8 quasi-norms), periodic and quantized links, and more generalized superposition models. Future developments may include deep-unfolded NLD-PALM architectures with trainable thresholding, fully removing dimensional restrictions for support recovery via primal–dual witness methods.

Conclusion

This paper establishes a robust, non-convex regularization approach for nonlinear sparse demixing, with provable convergence and statistical recovery guarantees. Huberized SCAD/MCP estimators provide earlier phase transitions and oracle rates, outperforming traditional DHT and $g$9 methods, especially in the presence of gross contamination and nonlinear front-ends. This framework is practically relevant and lays the foundation for deeper statistical and algorithmic exploration in nonlinear observation models and non-convex regularization.