Optimizing Expected Shortfall under an $\ell_1$ constraint -- an analytic approach

Abstract: Expected Shortfall (ES), the average loss above a high quantile, is the current financial regulatory market risk measure. Its estimation and optimization are highly unstable against sample fluctuations and become impossible above a critical ratio $r=N/T$, where $N$ is the number of different assets in the portfolio, and $T$ is the length of the available time series. The critical ratio depends on the confidence level $\alpha$, which means we have a line of critical points on the $\alpha-r$ plane. The large fluctuations in the estimation of ES can be attenuated by the application of regularizers. In this paper, we calculate ES analytically under an $\ell_1$ regularizer by the method of replicas borrowed from the statistical physics of random systems. The ban on short selling, i.e. a constraint rendering all the portfolio weights non-negative, is a special case of an asymmetric $\ell_1$ regularizer. Results are presented for the out-of-sample and the in-sample estimator of the regularized ES, the estimation error, the distribution of the optimal portfolio weights and the density of the assets eliminated from the portfolio by the regularizer. It is shown that the no-short constraint acts as a high volatility cutoff, in the sense that it sets the weights of the high volatility elements to zero with higher probability than those of the low volatility items. This cutoff renormalizes the aspect ratio $r=N/T$, thereby extending the range of the feasibility of optimization. We find that there is a nontrivial mapping between the regularized and unregularized problems, corresponding to a renormalization of the order parameters.