ConquerNet: Convolution-Smoothed Quantile ReLU Neural Networks with Minimax Guarantees

Abstract: Quantile regression is a fundamental tool for distributional learning but poses significant optimization challenges for deep models due to the non-smoothness of the pinball loss. We propose ConquerNet, a class of \textbf{con}volution-smoothed \textbf{qu}antil\textbf{e} \textbf{R}eLU neural \textbf{net}works, which yield smooth objectives while preserving the underlying quantile structure. We establish general nonasymptotic risk bounds for ConquerNet under mild conditions, providing minimax guarantees over Besov function classes. In numerical studies, we demonstrate that the proposed approach outperforms standard quantile neural networks at multiple quantile levels, showing improved estimation accuracy and training efficiency across the board, with particularly pronounced advantages at high and low quantiles.

• The paper introduces convolutional smoothing for the quantile loss, enhancing differentiability and facilitating efficient gradient-based optimization.
• It establishes minimax optimal convergence rates over Besov spaces through nonasymptotic risk bounds, bridging theory with deep neural network applications.
• Empirical results demonstrate improved training speed and robust tail quantile estimation, outperforming traditional quantile regression techniques.

Convolution-Smoothed Quantile ReLU Neural Networks with Minimax Guarantees: An Analysis of ConquerNet

Introduction and Motivation

Quantile regression is indispensable in statistical learning for robust modeling, offering insights into distributional heterogeneity and tail behavior. However, direct optimization of the classical pinball loss, $\rho_\tau(u)$, is notoriously challenging in deep neural networks due to its non-differentiability, resulting in sharp minima and unstable convergence—especially as the model complexity scales. Practical training of deep quantile regression networks frequently suffers from compromised generalization and inefficient optimization.

ConquerNet, as formalized in "ConquerNet: Convolution-Smoothed Quantile ReLU Neural Networks with Minimax Guarantees" (2605.06265), advances the field by systematically integrating convolution-type smoothing of the quantile loss within modern deep ReLU architectures. The methodology leverages the theoretical robustness of smoothed loss approaches, previously prominent only in linear settings, democratizing their statistical benefits within expressive, nonlinear neural environments. The architecture is further shown to yield minimax-optimal convergence rates within Besov spaces, supported by nonasymptotic risk guarantees that extend well beyond classical smoothness regimes.

Smoothing Principle and Loss Construction

The pivotal innovation is the replacement of the non-smooth quantile loss $\rho_\tau(u)$ with its convolutional smoothed counterpart $\ell_h(u)$: $\ell_h(u) = \int \rho_\tau(v) K_h(v-u) dv,$ where $K_h$ is a scaled kernel (e.g., Gaussian, uniform, Epanechnikov) with bandwidth $h$. This smoothing not only ensures differentiability but also maintains the loss's robust statistical properties, as the smoothed and original loss closely align for small $h$, while larger $h$ offers better gradient propagation and mitigates the sharpness of minima.

Figure 1: Relationship between the original (solid) quantile loss $\rho_\tau(u)$ and its convolutional smoothing $\ell_h(u)$ (dashed) for various bandwidths $\rho_\tau(u)$0.

The action of $\rho_\tau(u)$1 is crucial: smaller $\rho_\tau(u)$2 limits the approximation region but begins to replicate the instability of the pinball loss; larger $\rho_\tau(u)$3 improves smoothness but may introduce bias. ConquerNet accommodates this tradeoff and provides guidance based on minimax theory for selecting $\rho_\tau(u)$4 in relation to sample size and function class complexity.

Theoretical Guarantees and Minimax Analysis

ConquerNet's principal theoretical achievement is the demonstration that its estimator achieves the minimax optimal convergence rate over Besov spaces (which generalize and contain Sobolev and Hölder classes), up to a logarithmic factor, under mild regularity conditions on both the data distribution and the neural network architecture: $\rho_\tau(u)$5 This aligns with the sharp rates established by deep ReLU networks under nonparametric settings, rigorously extending the minimax properties to the domain of smoothed quantile regression objectives. The theory relaxes previous requirements by allowing the target to be only in a Besov space (even discontinuous) and by avoiding stringent lower bounds on the covariate density outside the region of interest.

Moreover, general nonasymptotic upper bounds are provided for arbitrary neural architectures, even beyond ReLU MLPs, decoupling statistical risk from rigid architectural choices and focusing on the network class's complexity and empirical approximation error. This opens a flexible foundation for future extensions to residual and composite architectures as empirical process-based bounds are compatible with richer network topologies.

Empirical Studies

Extensive simulation and real-data experiments underpin the practical merits. Across several challenging regression scenarios (including non-smooth and high-dimensional targets, heavy-tailed noise, and discontinuities), ConquerNet—across various kernels—demonstrates substantial improvements in both mean squared error (MSE) and training efficiency compared to baseline quantile networks optimized on the standard pinball loss. Especially at extreme quantiles ($\rho_\tau(u)$6), gains are pronounced, consistent with theory that sharper minima impede robust learning in the tails.

Figure 2: Average training time across 50 trials at $\rho_\tau(u)$7 shows that ConquerNet reduces computation compared to baseline quantile networks.

Figure 4: Log-log plot of MSE against sample size for scenario 2 (model A), where ConquerNet with Gaussian kernel outperforms baseline, with fitted lines indicating superior convergence rates for the smoothed model.

ConquerNet also yields a smoother loss landscape during optimization, as shown by loss surface visualizations, resulting in a single well-behaved minimum and supporting improved training dynamics.

The model's practical stability with respect to bandwidth selection is demonstrated, with a broad range of $\rho_\tau(u)$8 yielding competitive results. K-fold cross-validation is proposed and validated as robust for tuning $\rho_\tau(u)$9 in practice.

On benchmark datasets—such as BMI prediction—ConquerNet achieves lower pinball loss than baseline quantile neural networks, validating its applicability beyond synthetic domains.

Implications and Theoretical Impact

The successful unification of convolutional smoothing and deep nonparametric quantile estimation is consequential at both foundational and applied levels. Theoretically, it resolves the tension between the statistical efficiency of quantile estimators and the optimization tractability demanded by deep networks in high dimensions. ConquerNet's minimax guarantees in Besov spaces close longstanding gaps in the literature, while its general risk bounds pave the way toward new methodologies for robust distributional learning.

Practically, the reduction in training time and improvement in tail quantile estimation have immediate relevance for risk-sensitive applications—finance, survival analysis, and uncertainty quantification in decision-making pipelines.

The general risk framework invites adaptation to more sophisticated/dynamic architectures, including deep ResNets or joint quantile processes, and opens inquiry into smoothing for other distributional losses such as CRPS or Wasserstein distances. Further, smoothed objectives may enhance reinforcement learning paradigms (see, e.g., distributional RL), where quantile and higher-moment estimation are central.

Conclusion

ConquerNet exemplifies a principled pathway toward statistically efficient, optimizable, and theoretically grounded quantile regression in deep learning. By leveraging convolutional smoothing within ReLU architectures, the estimator attains minimax optimality in function-rich settings and exhibits superior empirical performance, particularly in high-variance or heavy-tailed regimes. The methodology—marked by scalable optimization, robust statistical guarantees, and flexible theoretical underpinnings—provides a solid baseline and springboard for future exploration in distributional deep learning and nonparametric estimation.