Rectified Linear Unit Regression

Abstract: This paper develops a regression framework for the direct estimation of integrated functionals of conditional outcome distributions. The proposed method, termed rectified linear unit (ReLU) regression, projects the ReLU-transformed outcome onto covariates and admits a closed-form estimator. Its population regression function coincides with the integrated conditional distribution function of the outcome, and its convex conjugate, obtained via the Legendre-Fenchel transformation, recovers the integrated conditional quantile function. Both the regression and its conjugate require only mild distributional assumptions and accommodate non-continuous outcomes. We establish the uniform asymptotic distribution of the estimator and develop inference for the conjugate functional via the delta method for Hadamard directionally differentiable maps. Building on these results, we establish identification and inference for average quantile treatment effects over arbitrary subintervals of probability levels. This broadens the set of distributional parameters available to empirical work.

• The paper introduces a novel ReLU regression method that projects a ReLU-transformed outcome onto covariates to yield closed-form estimators of integrated distribution functionals.
• It employs convex duality and the Legendre-Fenchel transformation to derive robust estimators, accommodating discrete and mixed outcomes with minimal smoothness assumptions.
• The approach provides a unified framework for estimating distributional treatment effects, demonstrated through empirical analysis of the Oregon Health Insurance Experiment.

Rectified Linear Unit Regression: A Technical Analysis

Introduction and Motivation

"Rectified Linear Unit Regression" (2605.30609) develops a novel regression framework for direct estimation of integrated functionals of conditional outcome distributions. The central methodological innovation—ReLU regression—projects the ReLU-transformed outcome onto covariates, yielding a closed-form estimator for the integrated conditional distribution function. Through application of convex duality, particularly the Legendre-Fenchel transformation, this framework also enables estimation of the integrated conditional quantile function. The author emphasizes that this construction significantly relaxes requirements on the continuity of the outcome distribution and computational tractability compared to quantile regression, thereby addressing longstanding challenges in the analysis of discrete, count, and mixed outcomes.

Methodological Contributions

The primary modeling device is to use the ReLU transformation, $(y-Y)_{+}$, as the dependent variable in a linear regression indexed by $y$. The population minimizer $\beta_0(y)$ for covariates $X$ is derived as an $L_2$ projection, admitting a closed-form expression:

$$\beta_0(y) = \big(\mathbb{E}[XX^\top]\big)^{-1} \mathbb{E}\big[ X (y-Y)_+ \big].$$

Notably, the linear predictor $X^\top \beta_0(y)$ equals $\mathbb{E}\big[ (y-Y)_+\,|\,X \big]$, representing the integrated conditional distribution function.

By convex duality, applying the Legendre-Fenchel transformation to the estimated conditional distribution yields the integrated conditional quantile function:

$$G_{Y|X}^*(\tau\,|\,x) = \int_{0}^{\tau} F_{Y|X}^{-1}(u\,|\,x) \, du,$$

which is robust to the non-uniqueness of quantile functions in the presence of discontinuities, in contrast to traditional quantile regression.

The methodology is highly general, requiring only existence of low-order moments and essentially no smoothness conditions on $Y$. This robustness makes it amenable for broad empirical applications, especially where outcome distributions are discrete or exhibit mass points.

Statistical Theory and Asymptotics

The ReLU regression estimator is shown to be uniformly asymptotically normal in function space under mild conditions:

$y$0

where $y$1 is a zero-mean Gaussian process.

Estimation of the integrated quantile function involves an empirical Legendre-Fenchel transformation. As this functional is not fully Hadamard differentiable, the paper invokes modern results on functional delta methods for directionally differentiable functionals, leveraging recent developments (e.g., [fang2019inference]). Inference for the conjugate functional is handled via the delta-method bootstrap specifically adapted for directionally differentiable settings, avoiding the inconsistency pitfalls of the standard nonparametric bootstrap at non-smooth points.

Treatment Effects and Causal Inference

A key theoretical contribution is the unified treatment of distributional treatment effects, including AQTE (average quantile treatment effects) over arbitrary subintervals of the probability scale. This parameterization nests both the average treatment effect (ATE) and the quantile treatment effect (QTE) as special cases:

• For $y$2, AQTE reduces to the ATE.
• As $y$3, AQTE converges to the QTE at $y$4, even for discrete or mixed outcomes.

Identification of AQTE is established under random assignment with minimal regularity provisions, and estimation is operationalized by ReLU regressions in treatment and control arms, followed by taking differences of integrated quantile functionals across subintervals.

Empirical Application: Oregon Health Insurance Experiment

In an application to the Oregon Health Insurance Experiment, the ReLU regression approach is used to analyze outpatient healthcare visits, a discrete outcome with strong zero-inflation and overdispersion. Estimation aligns with the randomization structure of the experiment, including cluster-adjusted bootstrap inference.

The empirical findings reveal:

• Substantial mass at zero in both control and treatment arms, with visible reduction in zero-mass under treatment (see Figure 1).

  Figure 1: Empirical distribution of outpatient visits by treatment status.

• The estimated integrated quantile functions are flat at lower quantiles (reflecting many zeros), then rise sharply above a threshold (see Figure 2).

  Figure 2: Estimated integrated quantile functions.

• The AQTE is approximately zero at lower quantiles, then becomes significantly positive at higher quantiles, indicating heterogeneous treatment effects concentrated among those with higher baseline utilization (see Figure 3).

  Figure 3: Average quantile treatment effect.

These results would have been obscured by conventional mean-based or quantile regression analyses, especially due to the discrete nature of the outcome.

Implications and Future Directions

The ReLU regression offers a computationally efficient, non-iterative alternative for distributional inference in causal and predictive modeling. Its closed-form nature simplifies theory and practice, and the convex-analytic framing unifies estimation of both distribution and quantile functionals, accommodating outcome types far beyond the continuous case assumed in traditional approaches.

The AQTE framework is especially valuable for policy evaluation, as it provides a granular description of distributional effects rather than a single summary measure.

Potential future directions include:

• Extension to high-dimensional or nonparametric covariate representations (e.g., via sieve or neural network bases).
• Applications to conditional risk and welfare analysis frameworks.
• Generalization to non-binary or longitudinal treatment settings.
• Integration with modern causal inference pipelines (e.g., double machine learning) for robustness to confounding in observational studies.

Conclusion

Rectified Linear Unit Regression constructs an integrated perspective on the conditional distribution and its quantile functional via a convex-analytic lens, offering computational tractability and rigorous inference in complex outcome settings. The method presents a unified approach for the estimation and inference of heterogeneous treatment effects, particularly accommodating discrete and mixed outcomes—a limitation for much of the existing semiparametric literature. Through a blend of analytic tractability, theoretical rigor, and empirical application, this approach broadens the scope of distributional analysis for econometric and statistical research.

---

Reference:

"Rectified Linear Unit Regression" (2605.30609)