--- title: Distributionally Robust Quantile Regression url: https://www.emergentmind.com/topics/distributionally-robust-quantile-regression type: topic --- # Distributionally Robust Quantile Regression Searching arXiv for the specified papers and closely related work on distributionally robust quantile regression. Distributionally robust quantile regression (DR-QR) studies quantile regression under ambiguity about the data-generating law. In the Wasserstein formulation, one minimizes the worst-case expected check loss over all distributions in a type-$p$ Wasserstein ball around the empirical law, and this yields a tractable convex reformulation with an additive penalty $c_{\tau,p}\epsilon \|\overline\beta\|_*$ and, for $p>1$, a radius-dependent intercept correction [2603.14991]. A related robust line based on $\beta$-divergence motivates a distributionally robust interpretation through a $\beta$-divergence ball around the empirical distribution, but implements a tractable surrogate loss that down-weights high-loss samples and is used for robust uncertainty estimation in deep neural networks in the presence of outliers [2309.07374]. ## 1. Problem setup and core objects In DR-QR with type-$p$ Wasserstein ambiguity sets, one observes i.i.d. samples $\{(X_i,Y_i)\}_{i=1}^N$ of a random pair $(X,Y)\in \mathbb{R}^d\times \mathbb{R}$ with unknown law $P$. For $\tau\in(0,1)$, quantile regression uses the check loss $$ \rho_\tau(u)=u(\tau-\mathbb{1}\{u<0\})=\tau u_+ + (1-\tau)u_-, $$ where $u_+=\max\{u,0\}$ and $u_-=\max\{-u,0\}$. Let $\|\cdot\|$ be a norm on $\mathbb{R}^{d+1}$ and $p\in[1,\infty]$. For distributions $P,Q$ on $\mathbb{R}^{d+1}$, the type-$p$ Wasserstein distance is $$ W_{p,\|\cdot\|}(P,Q)=\left(\inf_{\pi\in\Pi(P,Q)}\int \|z-z'\|^p\,d\pi(z,z')\right)^{1/p}, \qquad p\in[1,\infty), $$ and $$ W_{\infty,\|\cdot\|}(P,Q)=\inf_{\pi\in\Pi(P,Q)}\operatorname*{ess\,sup}_{\pi}\|z-z'\|. $$ Here $\Pi(P,Q)$ is the set of couplings. With the empirical distribution $$ P_N=\frac1N\sum_{i=1}^N \delta_{(X_i,Y_i)}, $$ the ambiguity set is $$ \mathcal{U}_\epsilon(P_N)=\{Q:W_{p,\|\cdot\|}(Q,P_N)\le \epsilon\}. $$ The dual norm associated with $\|\cdot\|$ is $$ \|v\|_*=\sup_{\|u\|\le 1}u^\top v. $$ A central quantity is $$ \overline\beta:=(-\beta,1)\in\mathbb{R}^{d+1}, $$ with dual norm $\|\overline\beta\|_*=\|(-\beta,1)\|_*$. The DR-QR objective is $$ \min_{\beta_0,\beta}\sup_{Q\in \mathcal{U}_\epsilon(P_N)}\mathbb{E}_Q\big[\rho_\tau(Y-\beta_0-X^\top\beta)\big]. $$ This setup makes the geometry of the ambiguity set, the order $p$, and the transport norm jointly determine the robustification. ## 2. Wasserstein DR-QR and the tractable reformulation A key reduction projects the Wasserstein ball onto the one-dimensional residual variable. For any $(\beta,s)$ and $F_0$, $$ \sup_{F\in \mathcal{B}_p(F_0,\epsilon)} \mathbb{E}^{F}\big[\rho_\tau(Y-X^\top\beta-s)\big] = \sup_{G\in \mathcal{B}_p(G_0,\epsilon \|\overline\beta\|_*)}\mathbb{E}^{G}\big[\rho_\tau(Z-s)\big], $$ where $G_0$ is the law of $Z=Y-X^\top\beta$ under $F_0$ and $\mathcal{B}_p$ denotes the type-$p$ ball with the same transport norm [2603.14991]. This one-dimensional reduction is the basis for the closed-form analysis of the worst-case loss. Let $q$ be the Hölder conjugate of $p$, so that $1/p+1/q=1$. Define $$ c_{\tau,p}= \begin{cases} \max\{\tau,1-\tau\}, & p=1,\\[4pt] \big(\tau^q(1-\tau)+\tau(1-\tau)^q\big)^{1/q}, & 10$, $$ \sup_{G\in \mathcal{B}_1(G_0,\epsilon)}\mathbb{E}^{G}[\rho_\tau(Z)] = \mathbb{E}^{G_0}[\rho_\tau(Z)] + \epsilon \max\{\tau,1-\tau\}, $$ $$ \sup_{G\in \mathcal{B}_p(G_0,\epsilon)}\mathbb{E}^{G}[\rho_\tau(Z)] = \inf_{\lambda>0}\left\{ \mathbb{E}^{G_0}\big[\rho_\tau(Z+k_2\lambda^{1-q})\big] +\lambda\epsilon^p+k_1\lambda^{1-q} \right\}, \qquad 11$. However, after optimizing over the intercept, an exact identity holds: $$ \inf_{s\in\mathbb{R}}\sup_{F\in \mathcal{B}_p(F_0,\epsilon)} \mathbb{E}^{F}\big[\rho_\tau(Y-X^\top\beta-s)\big] = \inf_{s\in\mathbb{R}}\mathbb{E}^{F_0}\big[\rho_\tau(Y-X^\top\beta-s)\big] +c_{\tau,p}\epsilon \|\overline\beta\|_*. $$ This leads to the tractable reformulation $$ \inf_{\beta\in\mathbb{R}^d,s\in\mathbb{R}} \sup_{F\in \mathcal{B}_p(F_0,\epsilon)} \mathbb{E}^{F}\big[\rho_\tau(Y-X^\top\beta-s)\big] = \min_{\beta\in\mathbb{R}^d,\bar s\in\mathbb{R}} \left\{ \mathbb{E}^{F_0}\big[\rho_\tau(Y-X^\top\beta-\bar s)\big] +c_{\tau,p}\epsilon\|\overline\beta\|_* \right\}. $$ Moreover, if $(\beta^*,\bar s^*)$ solves the right-hand program, then an optimizer $(\beta^*,s^*)$ for the left-hand DRO problem is obtained by $$ s^*= \begin{cases} \bar s^*, & p=1,\\[4pt] \bar s^*+\dfrac{\epsilon}{q}\big(\tau^q-(1-\tau)^q\big)c_{\tau,p}^{1-q}\|\overline\beta^*\|_*, & p\in(1,\infty]. \end{cases} $$ In particular, the slope vector $\beta^*$ is the minimizer of the regularized empirical risk with penalty $c_{\tau,p}\epsilon\|\overline\beta\|_*$, and the intercept is adjusted by a radius- and $p$-dependent correction when $p>1$. ## 3. Regimes of Wasserstein order and transport geometry The distinction between $p=1$ and $p>1$ is qualitative rather than merely notational. For $p>1$, the optimizer $\beta^*$ from DR-QR coincides with the optimizer of the regularized problem $$ \min_{\beta,\bar s}\ \mathbb{E}^{P_N}\big[\rho_\tau(Y-X^\top\beta-\bar s)\big] + c_{\tau,p}\epsilon\|\overline\beta\|_*. $$ Conditional on $\lambda=c_{\tau,p}\epsilon$, DR-QR and the regularized formulation yield the same slopes $\beta^*$. The intercept differs: $$ s^*-\bar s^*= \frac{\epsilon}{q}\big(\tau^q-(1-\tau)^q\big)c_{\tau,p}^{1-q}\|\overline\beta^*\|_*. $$ Thus, the Wasserstein order $p$ affects only this correction through $q$ and $c_{\tau,p}$, while the slopes already solve the regularized objective [2603.14991]. For $p=1$, the inner worst-case expectation equals $\mathbb{E}[\rho_\tau(\cdot)]$ plus the Lipschitz coefficient times $\epsilon\|\overline\beta\|_*$, so the DRO and regularized problems coincide without any intercept correction: $$ s^*=\bar s^*, \qquad c_{\tau,1}=\max\{\tau,1-\tau\}. $$ In certain edge cases, such as no mass on one side of the residual, the worst-case distribution may not be attained, reflecting the extremal geometry of the $p=1$ ball. The transport norm determines the dual norm $\|\cdot\|_*$ that shapes both the slope regularizer $\|\overline\beta\|_*$ and the intercept correction magnitude via $\|\overline\beta^*\|_*$. The following examples summarize the norm geometry. | Transport norm on $\mathbb{R}^{d+1}$ | Dual norm | $\|\overline\beta\|_*$ | |---|---|---| | $\|\cdot\|_2$ | $\|\cdot\|_2$ | $\sqrt{\|\beta\|_2^2+1}$ | | $\|\cdot\|_1$ | $\|\cdot\|_\infty$ | $\max\{\|\beta\|_\infty,1\}$ | | $\|\cdot\|_\infty$ | $\|\cdot\|_1$ | $\|\beta\|_1+1$ | This geometry also appears in the characterization of worst-case distributions. For optimal $(\beta^*,s^*)$, adversarial perturbations align with the dual norm direction of $\overline\beta^*$. The constructions differ with $p$: extreme-tail concentration for $p=1$, uniform shifts for $p=\infty$, and quantile-weighted shifts for $11/2$ and $\mathbb{P}(Y-X^\top\tilde\beta-\tilde{\bar s}\le y)<\tau$ on $[0,\Delta s]$, then DR-QR yields strictly lower out-of-sample loss. ## 4. Uniqueness of the check loss and finite-sample guarantees For $p>1$, the check loss has a special status. Let $\ell:\mathbb{R}\to\mathbb{R}$ be convex and define, for any distribution $G$, $$ \rho^G(X):=\inf_{s\in\mathbb{R}}\mathbb{E}^{G}[\ell(X-s)]. $$ There exists $c>0$ such that, for all $G_0$ on $\mathbb{R}$ and $\epsilon>0$, $$ \sup_{G\in \mathcal{B}_p(G_0,\epsilon)} \rho^G(X)=\rho^{G_0}(X)+c\epsilon $$ holds if and only if $\ell$ is an affine transform of the check loss: $$ \ell(x)=a\,\rho_\tau(x-b)+d $$ for some $\tau\in(0,1)$, $a>0$, and $b,d\in\mathbb{R}$ [2603.14991]. Thus, for $p>1$, quantile loss is the unique convex loss yielding an exact additive Wasserstein regularization once the intercept is optimized out. The finite-sample out-of-sample guarantee is dimension-free. Fix $\tau\in(0,1)$ and $p\ge 1$. Let $F^*$ be the data-generating law of $(Y,X)$ with finite $s$-th moment $$ \Gamma:=\mathbb{E}^{F^*}[\|(Y,X)\|^s]<\infty $$ for some $s>2$. For any $\eta\in(0,1)$, with probability at least $1-\eta$ over the sample, $$ \mathbb{E}^{F^*}\big[\rho_\tau(Y-X^\top\beta-s)\big] \le \sup_{F\in \mathcal{B}_p(P_N,\epsilon_N(\eta))} \mathbb{E}^{F}\big[\rho_\tau(Y-X^\top\beta-s)\big], $$ simultaneously for all $(\beta,s)$, where $$ \epsilon_N(\eta)=\frac{c_\tau \log(2N+1)^{1/s}}{\sqrt{N}}, $$ with $$ c_\tau= \left(\frac{\tau\vee(1-\tau)}{\tau\wedge(1-\tau)}\right) \left( 360\sqrt{d+2}+2\sqrt{2\log(3/\eta)} +\sqrt{\left(\frac{3\Gamma}{\eta}\right)\left(\frac{32}{s-2}\right)\sqrt{\log(24/\eta)}+2(d+2)} \right). $$ Consequently, the DRO in-sample risk upper-bounds the true out-of-sample risk with $O(N^{-1/2})$ rates, without a curse of dimensionality. The same result yields a finite-sample, model-agnostic guarantee for regularized quantile regression. If $(\tilde\beta,\tilde{\bar s})$ minimizes $$ \min_{\beta,\bar s}\ \mathbb{E}^{P_N}\big[\rho_\tau(Y-X^\top\beta-\bar s)\big]+\lambda\|\overline\beta\|_*, $$ then for $\lambda=c_{\tau,p}\epsilon_N(\eta)$ the in-sample DRO risk equals the regularized in-sample risk. Under the same moment conditions, $$ \mathbb{P}^N\!\left( \mathbb{E}^{F^*}\big[\rho_\tau(Y-X^\top\tilde\beta-\tilde{\bar s})\big] \le \mathbb{E}^{P_N}\big[\rho_\tau(Y-X^\top\tilde\beta-\tilde{\bar s})\big] +\lambda \rho(\tilde\beta) \right)\ge 1-\eta, $$ where $$ \rho(\beta)=(\tau\vee(1-\tau))\|\overline\beta\|_*. $$ ## 5. Optimization, computation, and practical use In the empirical version, DR-QR reduces to the convex program $$ \min_{\beta\in\mathbb{R}^d,\bar s\in\mathbb{R}} \frac1N\sum_{i=1}^N \rho_\tau(Y_i-X_i^\top\beta-\bar s)+c_{\tau,p}\epsilon\|\overline\beta\|_*. $$ The optimality conditions reflect the standard quantile structure together with norm-based regularization. At optimum, $\bar s$ is any empirical $\tau$-quantile of the residuals $r_i=Y_i-X_i^\top\beta$, corresponding to the subgradient condition $$ 0\in \frac1N\sum_i \partial_{\bar s}\rho_\tau(r_i). $$ For the slope, $$ 0\in \frac1N\sum_{i=1}^N (\tau-\mathbb{1}\{r_i<0\})X_i + c_{\tau,p}\epsilon\,\partial_\beta\|\overline\beta\|_*. $$ Here $\partial_\beta\|\overline\beta\|_*$ is the subdifferential of the dual norm with respect to $\beta$ [2603.14991]. The computational options stated for this program are proximal subgradient or proximal coordinate descent, conic or LP formulations for common norms via epigraph tricks, and a second-order cone formulation for $\ell_2$. Each proximal iteration is $O(Nd)$, and convergence follows standard rates for nonsmooth convex optimization; strong convexity may arise with appropriate smoothing or additional ridge terms. Practical calibration follows the same structure. Theory suggests $\epsilon\propto N^{-1/2}$ with explicit calibration $\epsilon_N(\eta)$, while in practice cross-validation over a grid, often logarithmic, works well; empirical studies show that the cross-validated $\epsilon$ tracks $N^{-1/2}$. For $p>1$, DR-QR performs an intercept correction governed by $p$ through $q$ and $c_{\tau,p}$. Empirically, $p\in\{2,\infty\}$ often yields stable performance; $p=1$ can be overly conservative and may suffer from non-attainment in edge cases. The transport norm sets the regularizer geometry through $\|\overline\beta\|_*$ and the magnitude of the intercept correction, so feature and response scaling are important because $\|\overline\beta\|_*$ includes the intercept coordinate. Reporting fitted quantiles requires distinguishing the regularized intercept from the DRO-adjusted intercept. One first solves the regularized program to obtain $(\beta^*,\bar s^*)$ with $\lambda=c_{\tau,p}\epsilon$. One then reports $s^*=\bar s^*$ for $p=1$, and $$ s^*=\bar s^*+\frac{\epsilon}{q}\big(\tau^q-(1-\tau)^q\big)c_{\tau,p}^{1-q}\|\overline\beta^*\|_* $$ for $p>1$. This single parameter $\epsilon$ jointly calibrates slope regularization and intercept robustness. Relative to standard QR, DR-QR shrinks slopes and adjusts intercepts toward conservative quantiles, for example upward for upper quantiles, reducing under-coverage bias common in small $N$ and extreme $\tau$. By guarding against Wasserstein perturbations, it improves out-of-sample stability relative to standard QR and plain regularization that lacks intercept correction. ## 6. Related robust formulations, empirical behavior, and limitations A distinct robust approach is "Beta quantile regression for robust estimation of uncertainty in the presence of outliers" [2309.07374]. Standard quantile regression minimizes $$ \min_\theta \sum_{i=1}^n \rho_\tau(y_i-f_\theta(x_i)), $$ with the pinball loss written equivalently as $$ \rho_\alpha(y-f_\theta(x))= \begin{cases} (y-f_\theta(x))\alpha, & y\ge f_\theta(x),\\ (f_\theta(x)-y)(1-\alpha), & y1$ formulations to high-dimensional sparsity-inducing penalties is described as a natural direction. Heteroscedasticity, covariate shift, and fixed-design settings are identified as structured cases in which targeted ambiguity sets or companion fixed-design analyses may be useful. For $\beta$-QR, the paper states that formal guarantees such as finite-sample risk bounds or breakdown-point analysis are not provided, that $\sigma$ is fixed to $1$, and that very large $\beta$ can over-suppress informative hard examples while very small $\beta$ approaches standard QR. The paper also notes that, in severe distribution shift beyond outliers, pure $f$-divergence balls might be less effective than Wasserstein-based uncertainty sets. Taken together, these results delineate two technically different notions of robust quantile regression. Wasserstein DR-QR admits an exact convex reformulation, a radius-dependent intercept correction for $p>1$, and finite-sample out-of-sample guarantees of order $O(N^{-1/2})$ under mild moments. The $\beta$-divergence approach provides a tractable robust surrogate with exponential down-weighting that is straightforward to integrate into SGD- and Adam-based deep learning workflows. Source: https://www.emergentmind.com/topics/distributionally-robust-quantile-regression