ReLU Regression: Theory & Practical Insights

• ReLU regression is defined as a family of regression problems that incorporate the rectified linear unit nonlinearity in the hypothesis, loss, or target formulation.
• Key methodologies include using convex surrogate losses, radial-isotropic transformations, and streaming stochastic updates to achieve robust parameter recovery.
• The approach spans applications from deep network-based nonparametric quantile estimation to structured models in system identification and econometric analysis.

Searching arXiv for recent and foundational papers on ReLU regression to ground the article in the literature. Tool unavailable in this environment, so proceeding with the arXiv papers specified in the provided corpus and citing them directly. Rectified Linear Unit (ReLU) regression denotes a family of regression problems in which the rectified linear unit nonlinearity enters either the hypothesis class, the loss geometry, or the target functional itself. In the canonical single-neuron formulation, the target is a homogeneous ReLU

$$\mathrm{ReLU}_{w}(x)=\mathrm{ReLU}(w\cdot x)=\max\{0,w\cdot x\},$$

and the objective is to recover or approximate the optimal parameter from labeled data under square loss, absolute loss, or streaming updates (Diakonikolas et al., 2020, Diakonikolas et al., 2021, Jeong et al., 2024). In broader usage, the term also covers deep ReLU networks used as nonparametric regression sieves for quantile estimation (Padilla et al., 2020), and a threshold-indexed least-squares construction that regresses $(y-Y)_+$ on covariates to estimate integrated conditional distribution and quantile functionals (Oka, 28 May 2026). The resulting literature spans computational hardness, robust estimation under semi-random corruption, approximation schemes, expressive-capacity analysis, system identification, Bayesian geometric representations, and distributional treatment-effect inference.

1. Formal problem classes

A central line of work studies supervised fitting of a single ReLU predictor. In the agnostic squared-loss setting, the hypothesis class is

$$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$

with isotropic input distribution $D_X$, risk

$$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$

and benchmark

$$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$

In the robust exact-recovery setting, the learner receives i.i.d. samples $(x_i,y_i)$ with labels generated by $f(x)=\mathrm{ReLU}(w^*\cdot x)$ and then corrupted according to a real-valued Massart model; the empirical objectives emphasized are the $\ell_1$ loss

$\hat L(w)=\frac1m\sum_{i=1}^m |y_i-f(w\cdot x_i)|$

and the ideal $(y-Y)_+$0 fitting criterion (Diakonikolas et al., 2020, Diakonikolas et al., 2021).

A streaming formulation considers the single-neuron rectified linear system

$(y-Y)_+$1

where the covariates arrive sequentially and each sample is processed once. Here the focus is not batch ERM but a one-sample-at-a-time stochastic method for robust ReLU regression under adversarial semi-random corruption (Jeong et al., 2024).

A separate nonparametric regression line uses deep feedforward ReLU networks as estimator classes for conditional quantiles. For a quantile level $(y-Y)_+$2, the target is

$(y-Y)_+$3

and the estimator minimizes the check loss

$(y-Y)_+$4

over constrained ReLU architectures (Padilla et al., 2020).

The phrase “ReLU regression” is also used in a distinct distributional sense. For scalar outcome $(y-Y)_+$5, covariates $(y-Y)_+$6, and threshold $(y-Y)_+$7, one regresses the transformed outcome

$(y-Y)_+$8

on $(y-Y)_+$9. The population parameter $$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$0 minimizes

$$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$1

and the fitted regression function coincides with the integrated conditional distribution function when the linear specification is correct (Oka, 28 May 2026).

2. Identifiability, noise, and computational barriers

The single-ReLU literature distinguishes sharply between realizable, semi-random, and fully adversarial regimes. ReLU regression is efficiently solvable in the realizable setting, whereas under adversarial label noise even approximate learning is computationally hard in broad settings. The Massart model occupies an intermediate regime: each label is corrupted with probability at most $$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$2, but the corrupted value itself may be chosen adversarially after the sample is seen. Under mild anti-concentration assumptions on the input distribution, exact recovery of the true parameter becomes possible again; those assumptions are explicitly argued to be information-theoretically necessary rather than proof artifacts (Diakonikolas et al., 2021).

The agnostic squared-loss setting exhibits a different barrier. Exact optimization of the ReLU objective is hard, and prior hardness results rule out achieving $$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$3 even under standard Gaussian inputs. This hardness motivates approximation schemes rather than exact ERM, and it explains why later positive results emphasize either constant-factor guarantees, improper PTAS constructions, or robust recovery only under stronger structural conditions on the input distribution (Diakonikolas et al., 2020).

These results collectively show that ReLU regression is not distribution-free in any strong algorithmic sense. Exact recovery under Massart corruption relies on anti-concentration and lower bounds on halfspace mass; the streaming theory assumes isotropic Gaussian-like measurements together with symmetry conditions; the agnostic PTAS requires subgaussianity; and the proper constant-factor algorithm is proved under isotropic log-concavity (Diakonikolas et al., 2021, Jeong et al., 2024, Diakonikolas et al., 2020). A related misconception is that the same regularity assumptions are needed in all variants. The econometric ReLU-regression framework is substantially different: for its integrated-distribution target, the key assumptions are finite moments and positive definiteness of $$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$4, and the theory explicitly accommodates non-continuous outcomes without requiring conditional density smoothness (Oka, 28 May 2026).

3. Core algorithmic paradigms

In the agnostic single-ReLU setting, a central idea is to avoid direct nonconvex optimization of squared loss and instead minimize the convex surrogate

$$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$5

whose gradient is the mismatch between model and label Chow parameters: $$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$6 Under isotropic log-concavity, the distribution induces strong convexity of the surrogate for ReLU, yielding the first efficient proper constant-factor approximation algorithm for ReLU regression. Under stronger subgaussian assumptions, a PTAS is obtained by combining a constant-factor initializer with a three-region decomposition: zero on one region, a linear predictor on another, and polynomial regression on a narrow middle band (Diakonikolas et al., 2020).

For exact recovery under Massart noise, the main technical device is a radial-isotropic transformation. In the linear warm-up, this transformation rescales and reorients the sample so that no direction is underrepresented and no small set of corrupted high-leverage points can dominate $$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$7 regression. The ReLU algorithm then uses a separation-oracle formulation: for a query $$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$8, it restricts attention to the active set $$\mathcal C_{\mathrm{ReLU}}=\{x\mapsto \mathrm{ReLU}(\langle w,x\rangle): w\in\mathbb R^d\},$$9, exploits the fact that ReLU is linear on that region, computes a generalized Forster or radial-isotropic transform, and returns a separator to the ellipsoid method unless the current query already fits at least half the samples. Under the stated anti-concentration conditions, the sample complexity is

$D_X$0

and the algorithm outputs the exact target parameter with probability at least $D_X$1 (Diakonikolas et al., 2021).

The streaming setting replaces batch optimization by the update

$D_X$2

initialized at $D_X$3. This SGD-exp procedure is derived from the $D_X$4 loss with an exponentially decaying step size. Under Massart corruption $D_X$5, isotropic Gaussian-like measurements, and the ReLU symmetry condition, the method achieves a nearly linear convergence guarantee to the true parameter, with error decaying essentially like

$D_X$6

up to the prefactor stated in the theorem; under symmetric oblivious corruption, the factor $D_X$7 is replaced by $D_X$8, extending the guarantee to all $D_X$9 (Jeong et al., 2024).

A further optimization-theoretic development restricts gradient flow to a smooth normalization manifold in parameter space. For ReLU networks, positive homogeneity creates scaling symmetries that can generate degeneracy and possible parameter blow-up. The proposed normalized gradient flow keeps each hidden neuron’s incoming parameter block at unit norm, preserves the set of realizable functions, monotonically decreases the regression risk, and, in the shallow one-dimensional one-hidden-neuron case, yields globally bounded trajectories for every Lipschitz target function (Eberle et al., 2022).

4. Structured architectures and specialized regression models

Several papers study ReLU regression through structured architectures rather than generic one-neuron or fully connected models. One such model is the two-layer residual unit

$$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$0

with full-rank $$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$1 and $$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$2. The learning strategy is layer-wise and convex: for each layer, the empirical risk minimization problem is formulated as a positive semidefinite quadratic program, and in the noiseless case the solution space is equivalently characterized by linear inequalities, yielding a linear program. The paper proves exact population characterization of the layer-wise objectives and strong statistical consistency of the resulting estimator, with

$$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$3

for the $$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$4 output loss (Wang et al., 2020).

A different structured proposal is the univariate ReLU network. Instead of standard hidden units $$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$5, it first maps inputs to latent coordinates $$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$6, then applies dimension-wise hinge functions

$$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$7

and finally combines them linearly. The resulting predictor has the form

$$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$8

Training exploits variable projection: the model is nonlinear in $$L_D(h)=\mathbb E_{(x,y)\sim D}\big[(h(x)-y)^2\big],$$9 but linear in $$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$0, so $$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$1 is solved by least squares for fixed $$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$2. The method was proposed for nonlinear system identification and validated on the Bouc–Wen benchmark (Liang et al., 2020).

Bayesian formulations reparameterize the hidden layer geometrically. In the Poisson-hyperplane-process construction, each hidden ReLU unit corresponds to a hyperplane

$$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$3

and a realization of the process induces the regression function

$$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$4

With Gaussian priors on the output weights and Gaussian observation noise, this yields an alternative probabilistic representation of a two-layer ReLU network and motivates annealed sequential Monte Carlo for posterior inference (Ge et al., 9 Jan 2026).

5. Expressivity, approximation, and solution-space geometry

Deep ReLU regression networks are also analyzed as nonparametric estimators. For quantile regression, empirical minimization of the pinball loss over constrained ReLU sieves yields an oracle inequality in which the error decomposes into a complexity term and a best-approximation term. For compositional Hölder classes, suitably scaled deep sparse architectures achieve

$$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$5

and if $$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$6,

$$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$7

For Besov targets, the achieved rate is

$$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$8

up to logarithmic factors. These results are stated under minimal assumptions that allow heavy-tailed response distributions and do not require Gaussian or sub-Gaussian errors (Padilla et al., 2020).

Expressivity can also be measured by the number of linear regions. For a standard ReLU regression network of depth $$\mathrm{opt}_D(\mathcal C_{\mathrm{ReLU}}):=\min_{w\in\mathbb R^d}L_D(\mathrm{ReLU}_w).$$9 and width $(x_i,y_i)$0, the Montúfar-type lower bound recalled in the ternary-network paper is

$(x_i,y_i)$1

For a ternary architecture with weights in $(x_i,y_i)$2, odd layers using the identity activation, and even layers using ReLU, the main lower bound is

$(x_i,y_i)$3

The paper emphasizes that these are lower bounds on the maximum number of linear regions, not exact counts, and concludes that region counts still grow polynomially in width and exponentially in depth under ternary constraints (Nakahara et al., 21 Jul 2025).

A separate statistical-mechanics analysis, although framed as pattern storage rather than regression, studies the geometry of two-layer ReLU solution spaces. In a tree-like committee machine with spherical weights, the critical storage capacity remains finite as hidden width $(x_i,y_i)$4, with the 1RSB estimate

$(x_i,y_i)$5

The same analysis finds that typical exact-fit solutions are isolated, but rare dense regions of solutions exist and are broader for ReLU than for threshold activations at sufficiently low load. The paper interprets these dense clusters as robust to weight and input perturbations, suggesting that ReLU’s advantage in this stylized setting lies more in favorable geometry than in diverging raw capacity (Baldassi et al., 2019).

6. Applications, empirical evidence, and scope of the term

Empirical work on robust single-ReLU regression supports the theoretical distinction between naive loss minimization and geometry-aware methods. Under Massart corruption on a 30-dimensional mixture-of-Gaussians design, the radial-isotropic algorithm substantially outperforms naive $(x_i,y_i)$6 regression and $(x_i,y_i)$7 regression with simple normalization in exact parameter recovery rate. On synthetic ReLU experiments with noise level $(x_i,y_i)$8, transformed methods, especially radial-isotropic preprocessing on the active set, converge much better in $(x_i,y_i)$9-distance to the optimum. On a drug-discovery dataset with 3084 training points, 1000 test points, and dimension 410, radial-isotropic $f(x)=\mathrm{ReLU}(w^*\cdot x)$0 regression is slightly worse at very low noise but significantly better at moderate and high noise (Diakonikolas et al., 2021).

The streaming literature reports analogous robustness phenomena. On synthetic Gaussian ReLU data with sign-flip or large additive corruption, SGD-exp converges approximately linearly, whereas a square-root-decay analogue fails to recover the true parameter. On the Red Wine Quality dataset with 1599 samples, 10 numerical features, and corruption rate $f(x)=\mathrm{ReLU}(w^*\cdot x)$1, SGD-exp attains low loss on the uncorrupted objective while stochastic GLM-Tron variants are more affected by corruption (Jeong et al., 2024).

System-identification applications highlight the value of structured ReLU regressors. On the Bouc–Wen hysteretic benchmark, the UReLU network uses 201 parameters, compared with 1550 for the reported standard ReLU baseline, and achieves the best reported validation performance among the listed methods, with $f(x)=\mathrm{ReLU}(w^*\cdot x)$2 dB on multi-sine data and $f(x)=\mathrm{ReLU}(w^*\cdot x)$3 dB on swept-sine data (Liang et al., 2020). Residual-unit learning was also evaluated on benchmark regression datasets, where the convex layer-wise solver often outperformed SGD from random initialization and was additionally useful as an initializer for SGD (Wang et al., 2020).

The distributional meaning of ReLU regression has yet another applied domain. In treatment-effect analysis, regressing $f(x)=\mathrm{ReLU}(w^*\cdot x)$4 on $f(x)=\mathrm{ReLU}(w^*\cdot x)$5 yields estimates of integrated potential-outcome distributions, whose Legendre-Fenchel conjugates identify integrated quantile functionals. This supports inference for average quantile treatment effects over arbitrary probability subintervals,

$f(x)=\mathrm{ReLU}(w^*\cdot x)$6

and the empirical illustration on the Oregon Health Insurance Experiment uses a count outcome with many zeros and a sparse upper tail (Oka, 28 May 2026).

Taken together, these strands show that “ReLU regression” is not a single method but a cluster of related theories centered on the ReLU nonlinearity. In one-neuron learning, the main themes are hardness, exact recovery under semi-random noise, surrogate-based approximation, and streaming robustness. In deep-network regression, the emphasis shifts to approximation rates, expressive complexity, and solution geometry. In structured models, residual connections, univariate decompositions, and Poisson hyperplane processes provide alternative parameterizations with different computational and statistical tradeoffs. And in distributional econometrics, the ReLU transform itself becomes the regression target, linking least-squares projection to integrated distribution and quantile functionals (Diakonikolas et al., 2020, Diakonikolas et al., 2021, Padilla et al., 2020, Oka, 28 May 2026).