Expected Shortfall Regression Overview

• Expected Shortfall Regression is a statistical framework that jointly models Value-at-Risk (VaR) and Expected Shortfall (ES) to capture tail risk in finance and other domains.
• It employs joint estimation techniques, two-step procedures, and penalized methods to overcome non-elicitability and computational challenges.
• The approach enhances backtesting, inference, and dynamic risk forecasting, aiding regulatory compliance and effective risk management.

Expected Shortfall Regression

Expected Shortfall (ES) regression refers to a class of statistical frameworks for modeling, estimating, and forecasting the conditional expected shortfall of a response variable, typically with respect to covariates or time-series dynamics. Expected shortfall, also known as Conditional Value-at-Risk (CVaR), is the mean of the response conditional on exceeding (for the upper tail) or falling below (for the lower tail) a specified quantile—such as the Value-at-Risk (VaR). ES regression addresses the demand for tail risk modeling in fields such as financial risk management, insurance, econometrics, and increasingly, in scientific domains concerned with tail-dependent phenomena. Unlike quantile regression, traditional M-estimation is infeasible for ES due to its lack of single-parameter elicitability, necessitating the use of joint estimation, two-step procedures, or compositional methods, and raising challenges in estimation, inference, and backtesting.

1. Theoretical Foundations: Elicitability and Joint Losses

The central challenge in ES regression arises from the non-elicitability of ES as a stand-alone risk measure: there does not exist a strictly consistent loss function whose minimizer is solely the ES functional. However, the seminal result of Fissler and Ziegel (2016) proves the joint elicitability of the (VaR, ES) pair: there exist strictly consistent scoring functions whose minimizer yields both the quantile (VaR) and the expected shortfall simultaneously. The general class of such loss functions for a response $Y$ and pair of forecasts $(q, e)$ is

$$\ell_\alpha(q,e;y) = (\mathbf{1}\{y \le q\} - \alpha)(q - y) + v_1(q,e) + v_2(q,e)(e-y)/\alpha,$$

where $v_1$ and $v_2$ are appropriately chosen specification functions, often homogeneous in $e$ (Dimitriadis et al., 2017). The strict consistency property guarantees that the population minimizer of $\ell_\alpha$ corresponds to the true (VaR, ES) pair under the underlying conditional distribution.

This foundational result underpins all subsequent ES regression methodologies. Joint M-estimation is possible but computationally nontrivial due to non-differentiability and possible non-convexity; thus, two-step procedures and Neyman-orthogonal scores have seen practical adoption for computational stability and scalability, especially in large-scale and high-dimensional settings (He et al., 2022).

2. Methodological Variants of Expected Shortfall Regression

The landscape of ES regression is characterized by several main approaches, each adapted to distinct modeling, computational, and inferential settings.

2.1 Joint M-Estimation of VaR–ES Regression

Direct joint regression specifies both conditional quantile and ES as linear (or smooth) functions of covariates: $$Q_\alpha(Y \mid X) = X^\top\beta, \quad ES_\alpha(Y \mid X) = X^\top\gamma.$$ Minimization of a strictly consistent joint loss function (as above) over $(\beta, \gamma)$ yields consistent and asymptotically normal estimators. The explicit form of the estimating equations depends on the choice of $(v_1, v_2)$, with positively homogeneous specifications preferred for numerical stability and mean squared error (Dimitriadis et al., 2017). This framework allows direct forecasting, backtesting, and inference for both VaR and ES.

2.2 Two-Step and Neyman-Orthogonal Procedures

To address the numerical instabilities of non-convex joint loss minimization, two-step procedures first estimate $\hat\beta$ via standard quantile regression (using the check loss), then regress ES on covariates adjusting for the estimated quantile: $$\hat\gamma \in \arg\min_\gamma \sum_{i=1}^n G_2(X_i^\top\gamma)\left\{X_i^\top(\gamma - \hat\beta) + \frac{(X_i^\top\hat\beta - Y_i)1\{Y_i \le X_i^\top\hat\beta\}}{\alpha}\right\} - \mathcal{G}_2(X_i^\top\gamma),$$ where $(G_2, \mathcal{G}_2)$ are as above (Peng et al., 2022, He et al., 2022). This decomposition is justified by the smoothness of the objective in $\gamma$ and enables Neyman-orthogonal estimation, which is robust to first-stage estimation error. Adaptive Huberization further yields finite-sample and non-asymptotic statistical guarantees under heavy-tailed error distributions (He et al., 2022, Yu et al., 11 Nov 2025).

2.3 High-Dimensional and Penalized ES Regression

In high-dimensional settings, where $p \gg n$, lasso-penalized regression is formulated for both the quantile stage and the ES stage: $\hat\beta = \arg\min_{\beta} \frac1n \sum_{i=1}^n \rho_\alpha(Y_i-X_i^\top\beta) + \lambda_q\|\beta\|_1, \quad \hat\theta = \arg\min_\theta \frac1n \sum_{i=1}^n \left\S_0(\hat\beta,\theta;Y_i,X_i)\right\S^2 + \lambda_e\|\theta\|_1.$ Non-asymptotic risk bounds of order $\sqrt{s\log p / n}$ under restricted eigenvalue and sub-Gaussianity conditions hold for the lasso ES estimator (Barendse, 2023, Zhang et al., 2023, He et al., 2022). Debiased (or desparsified) estimators are further developed for valid inference on $\theta_j$.

2.4 Deep and Nonparametric ES Regression

Deep neural expected shortfall regression replaces the function classes in the two-step procedure with deep ReLU networks, leveraging network capacity for hierarchical compositional functions. A robust variant with Huber loss addresses sensitivity to heavy-tailed errors: $$\hat g_{n,\tau} = \arg\min_g \frac1n\sum_{i=1}^n \ell_\tau(Z_i(\hat f_n) - \alpha\,g(X_i)),$$ where $Z_i(f) = \min\{Y_i-f(X_i), 0\} + \alpha f(X_i)$ and $\ell_\tau$ is the Huber loss. The resulting estimator achieves non-asymptotic minimax rates and is robust to error distributions with only $p$-th moments, suitable for high-dimensional and nonparametric conditional structures (Yu et al., 11 Nov 2025).

3. Time Series and Dynamic ES Regression

Expected Shortfall regression in time series necessitates additional structure to capture dynamics, autocorrelation, and feedback between past and current risk measures.

3.1 Dynamic Semiparametric VaR–ES Models

Patton, Ziegel, and Chen (2017) proposed dynamic two-factor and one-factor GAS-type models, joint CAViaR-ES models, and GARCH-ES models, estimated by minimizing joint Fissler–Ziegel losses (Patton et al., 2017). The vector recursion form

$$\begin{pmatrix} v_{t+1} \ e_{t+1} \end{pmatrix} = w + B \begin{pmatrix} v_t \ e_t \end{pmatrix} + A\lambda_t$$

enables rich autocorrelation and shock responses. Asymmetric Laplace-based maximum likelihood (or Bayesian) estimation provides an alternative to the pure scoring loss, often used with high-frequency realized-measure augmentation (Gerlach et al., 2018, Wang et al., 2018).

3.2 Conditional Autoregressive ES (CAESar) and Extensions

The CAESar model (Gatta et al., 9 Jul 2024) generalizes the CAViaR philosophy to coupled VaR–ES autoregressions without distributional assumptions, employing joint FZ-type losses with monotonicity constraints to prevent crossing (i.e., ES exceeding VaR). Estimation involves three phases: initial quantile autoregression (pinball loss), ES-residual autoregression (Barrera loss), and joint re-estimation using a penalized FZ-type loss (Patton form) with soft monotonicity penalties. CAESar demonstrates high forecasting performance and is robust to tail regime changes, outperforming neural network and score-driven GAS approaches.

4. Compositional and Multivariate ES Regression

Forecasting ES contributions (ESCs), or gradient allocations of ES under the Euler principle of capital allocation, requires multivariate compositional regression. Koike et al. (2024) (Koike et al., 22 Jan 2024) introduce a semiparametric compositional-regression model for the vector of ESCs, exploiting the simplex geometry via the isometric log-ratio (ilr) transform: $z_t = \mathrm{ilr}(w_t) = V^\top \ln(w_t),$ with $w_t$ ESC weights, and a dynamic linear autoregression in ilr-coordinates. The model is jointly backtestable and multi-objective elicitable, allowing full calibration via vector-valued scoring functions and Murphy diagrams. Empirical validation on stock returns demonstrates sharp outperformance over standard benchmarks in both tuplewise and componentwise backtests.

5. Backtesting, Inference, and Robustness

Backtesting of predicted ES (and ESCs) employs both traditional (identification function-based) and comparative (score or loss-based) frameworks. Score-based backtesting, grounded in joint scores for (VaR, ES), enables Diebold–Mariano tests, Murphy diagrams for dominance under classes of consistent scoring rules, and direct tests such as McNeil–Frey or Acerbi–Szekely Z-statistics (Koike et al., 22 Jan 2024, Gatta et al., 9 Jul 2024, Bayer et al., 2018).

For inference on regression parameters or functionals, score-type tests that avoid re-optimization—by evaluating derivatives of the partial loss—demonstrate higher power and better size control than Wald-type approaches, especially in high dimensions or for heteroskedastic data (Peng et al., 2022, He et al., 2022). Robust estimation is achieved via adaptive Huberization or Neyman-orthogonal constructions, characterized by strong non-asymptotic guarantees for both estimation and confidence interval coverage, including in high-dimensional settings (He et al., 2022, Barendse, 2023, Zhang et al., 2023).

6. Applications and Extensions

ES regression is central in financial risk management for tail risk forecasting and capital allocation under regulatory regimes such as Basel III. Empirical applications span daily and intraday returns of major indices, high-frequency trading, systemic risk (e.g., ΔCoES), and capital adequacy assessment. Beyond finance, ES regression is leveraged in scientific domains such as climate and health disparity analysis, where upper- or lower-tail responses are of primary interest (Yu et al., 11 Nov 2025, Zhang et al., 2023). Extensions to compositional models, multivariate risk allocation, lasso-penalized high-dimensional regression, and deep learning under heavy-tailed errors illustrate the rapid evolution and cross-disciplinary adoption of ES regression.