Online Quantile Regression for Nonparametric Additive Models

Abstract: This paper introduces a projected functional gradient descent algorithm (P-FGD) for training nonparametric additive quantile regression models in online settings. This algorithm extends the functional stochastic gradient descent framework to the pinball loss. An advantage of P-FGD is that it does not need to store historical data while maintaining $O(J_t\ln J_t)$ computational complexity per step where $J_t$ denotes the number of basis functions. Besides, we only need $O(J_t)$ computational time for quantile function prediction at time $t$. These properties show that P-FGD is much better than the commonly used RKHS in online learning. By leveraging a novel Hilbert space projection identity, we also prove that the proposed online quantile function estimator (P-FGD) achieves the minimax optimal consistency rate $O(t^{-\frac{2s}{2s+1}})$ where $t$ is the current time and $s$ denotes the smoothness degree of the quantile function. Extensions to mini-batch learning are also established.

• The paper presents a projected functional gradient descent approach that enables online quantile regression in high-dimensional nonparametric additive models.
• It leverages Hilbert space projections with ℓ1 norm constraints to ensure statistical consistency and memory efficiency in streaming data environments.
• The method achieves minimax optimal rates for Sobolev-type additive models and adapts seamlessly to mini-batch and ensemble learning settings.

Projected Functional Gradient Descent for Online Quantile Regression in Nonparametric Additive Models

Overview

The paper "Online Quantile Regression for Nonparametric Additive Models" (2604.08969) presents a projected functional gradient descent (P-FGD) methodology for online estimation of conditional quantiles within high-dimensional, nonparametric additive models. The approach addresses significant algorithmic and theoretical obstacles associated with the classic pinball loss in quantile regression, notably under nonparametric model classes, while ensuring computational efficiency and memory scalability in streaming data setups. The work advances stochastic approximation for quantile regression, achieving minimax optimal rates and providing tractable procedures that leverage Hilbert space projections.

Context and Problem Formulation

Quantile regression is essential for robust modeling under heteroskedasticity and heavy-tailed noise, outperforming least squares (mean regression) in such cases. Classical online quantile regression has predominantly been restricted to linear settings. Nonparametric additive models offer a key extension: they generalize linearity while circumventing the curse of dimensionality through structural function decomposition $q_\tau(x) = \alpha + \sum_{k=1}^p f_k(x^{(k)})$. The main challenges in this framework are as follows:

• The pinball loss is only piecewise linear, lacking global strong convexity, and its local curvature is dictated by the unknown conditional density.
• Existing functional gradient descent (FGD) or Reproducing Kernel Hilbert Space (RKHS) approaches have prohibitive memory/storage requirements or fail to scale online.
• Statistical consistency for nonparametric online quantile regression cannot be established through naive stochastic gradient descent due to coefficient divergence.

The paper resolves these challenges via a computational strategy that is provably minimax optimal, does not revisit or store historical data, and generalizes to mini-batch and ensemble procedures.

Algorithmic Framework

The proposed Projected Functional Gradient Descent (P-FGD) works as follows:

1. Basis Expansion: The conditional quantile $q_\tau$ is approximated using an additive representation in terms of orthonormal basis functions (e.g., trigonometric, wavelet, or Legendre). For each covariate and time $t$, only a finite number $J_t$ of basis functions are used, with $J_t$ growing over time to control approximation error.
2. Online Update: Upon receiving $(X_t, Y_t)$ at time $t$, compute the subgradient with respect to the pinball loss and update the coefficient vector accordingly.
3. Projection: After each gradient step, project the updated coefficients onto an $\ell_1$ ball of radius $R$. This crucially enforces uniform boundedness (and thus, uniform boundedness of the estimated function), and guarantees statistical consistency. The projection leverages efficient algorithms with $O(J_t \log J_t)$ complexity.
4. Incremental Dimension: As data accumulate, $q_\tau$0 increases, and coefficients for new higher-order basis functions are initialized to zero.
5. Prediction: The quantile estimate at any $q_\tau$1 is given by the dot product of the current coefficient vector with the evaluated basis function vector at $q_\tau$2.
6. Mini-batch Extension: In the case of arriving data batches, the procedure averages gradients over the batch, applies the same projected update, and achieves minimax-optimal rates with respect to the total sample size.
7. Ensemble Extension: Inspired by random forest methodology, the method can randomly subselect coordinates for updates at each iteration, and average multiple such online estimators for further robustness.

Theoretical Guarantees

Minimax Consistency and Optimality

The P-FGD estimator achieves the minimax optimal rate over Sobolev-type additive models, i.e.,

$q_\tau$3

where $q_\tau$4 is the smoothness parameter of the underlying function and $q_\tau$5 is the sample size. This convergence matches the known lower bounds for nonparametric estimation in Sobolev ellipsoids and is obtained under the minimal assumptions of boundedness and positivity of the marginal and conditional densities.

The proof employs:

• A custom Hilbert space projection analysis that decomposes the error dynamics into an approximation error component (due to truncated basis expansion) and an estimation error component (due to stochastic gradients).
• Novel use of the curvature of the pinball loss in expectation, leveraging lower bounds on the conditional density.
• Careful control of the coefficient norm through projection, preventing the divergence issues faced by unprojected stochastic gradient methods.

For the mini-batch setting with cumulative batch size $q_\tau$6, the same minimax rate holds upon rescaling $q_\tau$7. The ensemble extension preserves these rates through standard averaging arguments.

Computational Properties

• Per-iteration Complexity: Both online update and prediction incur $q_\tau$8 and $q_\tau$9 costs, respectively, matching the best known for high-dimensional online methods and substantially outperforming kernel-based approaches (which require revisiting or storing past data).
• Memory Efficiency: The procedure does not require storing any of the history of observed data, making it suitable for streaming or distributed architectures.

Practical and Theoretical Implications

The method provides a tractable, theoretically-sound approach to online quantile regression in nonparametric additive models. It is suitable for big data applications, including financial risk management, signal processing, and any setting demanding distributional robustness and streaming inference.

The theoretical analysis introduces a general template for controlling the stochastic dynamics of nonparametric function approximation with non-strongly convex losses and expands the use of $t$0 projection in infinite (growing) dimensional spaces for convergence and regularization.

Directions for Future Work

Open theoretical and practical avenues include:

• Automated selection or adaptation of the order and type of basis functions under data-driven streaming.
• Extension to time-varying or dependent data, as well as high-dimensional settings with $t$1 large relative to $t$2.
• Integration with advanced model averaging, structured regularization, or adaptive sparsity.
• Exploration of quantile regression under adversarial robustness in the online regime.

Conclusion

This work addresses the previously unresolved problem of nonparametric online quantile regression by formulating a functional stochastic gradient procedure with judicious coefficient norm projection, achieving optimal statistical rates and scaling efficiently for large-scale and streaming applications. The approach generalizes to several settings of practical interest and provides a rigorous foundation for future developments in robust, high-dimensional online inference.