CVaR-PPO: A Risk-Aware RL Approach

• CVaR-PPO is a risk-aware reinforcement learning method that constrains the conditional value at risk to enhance safety and robust performance under uncertainty.
• It integrates a Lagrangian relaxation approach with PPO’s clipped surrogate loss to maintain stable policy updates while enforcing risk constraints.
• Empirical results on continuous control tasks show that CVaR-PPO achieves higher returns and improved robustness compared to traditional policy gradient methods.

Conditional Value at Risk-Proximal Policy Optimization (CVaR-PPO), also referred to as CVaR-Proximal Policy Optimization (CPPO), is a risk-sensitive reinforcement learning algorithm that constrains the conditional value at risk (CVaR) of returns in deep policy optimization. The approach is designed to enhance both the robustness and safety of deep reinforcement learning (DRL) in environments subject to transition and observation uncertainties. By imposing explicit constraints on the lower tail of the return distribution, CPPO achieves improved performance and resilience compared to standard policy gradient and PPO methods in continuous control domains (Ying et al., 2022).

1. Formal Framework and Risk Metric

The CPPO methodology is formulated within the infinite-horizon discounted Markov Decision Process (MDP) $M = (\mathcal{S}, \mathcal{A}, P, R, \gamma)$, where states $\mathcal{S}$, actions $\mathcal{A}$, transition kernel $P(s'|s,a)$, reward function $R(s,a)\in[-R_{\mathrm{max}}, R_{\mathrm{max}}]$, and discount factor $\gamma\in[0,1)$ define the environment. Policies are parametrized as $\pi_{\theta}(a|s)$.

The return of a trajectory $\xi = (s_0, a_0, r_0, \dots, s_T)$ is

$D(\xi) = \sum_{t=0}^{T} \gamma^t r_t,$

with expected policy performance

$$J(\pi_\theta) = \mathbb{E}_{\xi \sim \pi_\theta}[D(\xi)].$$

Tail risk is quantified using the Conditional Value at Risk (CVaR) at confidence level $\alpha \in (0, 1)$, defined for a bounded-mean random variable $Z$ (here $Z = -D(\xi)$) as: \begin{align*} \mathrm{VaR}\alpha(Z) &= \inf{z \mid P(Z \leq z) \geq \alpha},\ \mathrm{CVaR}\alpha(Z) &= \mathbb{E}[Z \mid Z \geq \mathrm{VaR}\alpha(Z)],\ \mathrm{CVaR}\alpha(Z) &= \min_{\eta \in \mathbb{R}} \left{ \eta + \frac{1}{1-\alpha}\mathbb{E}[(Z - \eta)^+] \right}. \end{align*}

2. Risk-Constrained Optimization Objective

The central problem is to maximize expected return under a constraint that controls the CVaR of the negative return (i.e., the average of the worst-case outcomes):

$$\max_\theta ~ J(\pi_\theta),\qquad \textrm{s.t.}~ -\mathrm{CVaR}_\alpha(-D(\pi_\theta)) \geq \beta,$$

where $\beta$ is a user-specified threshold for acceptable tail risk. In practical form, the constraint is rewritten using the variational form of CVaR:

$$\min_{\theta, \eta} -J(\pi_\theta),\qquad \textrm{s.t.}~ \frac{1}{1-\alpha} \mathbb{E}_{\xi\sim\pi_\theta}[(\eta - D(\xi))^+] - \eta \leq -\beta.$$

3. Theoretical Justification: Value Function Range

CPPO’s robustness properties are established through the Value Function Range (VFR), defined as

$$V(s) = \mathbb{E}\left[ \sum_{t=0}^{\infty}\gamma^t r_t \mid s_0 = s \right], \qquad \mathrm{VFR}(\pi) = \max_s V(s) - \min_s V(s).$$

The performance degradation under model transition disturbance (changes in $P$) and observation disturbance (perturbations in the state or policy) is bounded by VFR:

• For transition disturbance with total variation distance $\epsilon_P$,

$$|J_P(\pi) - J_{\hat P}(\pi)| \leq \frac{2\gamma}{1-\gamma}\epsilon_P\cdot \mathrm{VFR}(\pi).$$

• For observation disturbance with TV-distance $\epsilon_\pi$ and $R_{\mathrm{max}}$-bounded rewards,

$$|J(\pi) - J(\pi \circ \nu)| \leq \frac{\gamma}{1-\gamma}\epsilon_\pi\cdot \mathrm{VFR}(\pi) + \frac{2}{1-\gamma} \epsilon_\pi R_{\mathrm{max}}.$$

VFR thus governs robustness, but minimizing VFR directly is conservative; CVaR-based control is proposed as a tractable surrogate, with the theoretical guarantee

$$-\mathrm{CVaR}_\alpha(-D(\xi)) \leq -\mathrm{CVaR}_\alpha(-V(s_0)),$$

allowing for a trajectory-level CVaR constraint (Ying et al., 2022).

4. Lagrangian Relaxation and Policy Update

The constraint optimization is handled via Lagrangian relaxation, with $\lambda \geq 0$ as the multiplier. The objective becomes a saddle-point problem:

$$L(\theta, \eta, \lambda) = -J(\pi_\theta) + \lambda\left[ \frac{1}{1-\alpha}\mathbb{E}_{\xi\sim\pi_\theta}[(\eta-D(\xi))^+] - \eta + \beta \right].$$

Optimization proceeds by descending $(\theta, \eta)$ and ascending $\lambda$, with the following gradients: \begin{align*} \nabla_\theta L &= -\mathbb{E}{\xi\sim\pi\theta}[\nabla_\theta \log P_\theta(\xi) (D(\xi) - \frac{\lambda}{1-\alpha}(\eta - D(\xi))^+)],\ \partial L/\partial \eta &= \frac{\lambda}{1-\alpha} P_{\xi\sim\pi_\theta}(D(\xi) \leq \eta) - \lambda,\ \partial L/\partial \lambda &= \frac{1}{1-\alpha}\mathbb{E}[(\eta-D(\xi))^+] - \eta + \beta. \end{align*} To preserve stable policy updates, the PPO clipped surrogate loss is retained for the standard policy gradient term, and the CVaR penalty is integrated directly.

5. CPPO Algorithmic Implementation

CPPO proceeds by batch-based approximation of expectations using $N$ sampled episodes per policy iteration. Key steps per iteration:

• Roll out $N$ trajectories under the current policy; compute discounted returns and advantages.
• Update $\eta$ via stochastic gradient descent.
• Update $\theta$ via policy gradient: combines standard PPO surrogate and a CVaR penalty term.
• Update $\lambda$ by gradient ascent.
• Fit value function parameters $\phi$ to empirical returns with MSE regression. An adaptive heuristic for $\beta$—setting it to the sample quantile of the bottom $\alpha$-fraction of recent returns—stabilizes optimization.

6. Empirical Evaluation and Robustness Analysis

Benchmarking is conducted on five MuJoCo v3 continuous control tasks: Ant, HalfCheetah, Walker2d, Swimmer, and Hopper. Baselines include VPG, TRPO, PPO, and PG-CMDP (Chow & Ghavamzadeh). Two disturbance regimes are considered:

• Transition disturbance: agent mass is scaled by a factor in $[0.5, 2.0]$.
• Observation disturbance: states are perturbed with additive Gaussian noise or adversarial FGSM perturbation.

Performance metrics are average return over training and robustness under disturbances.

Summary of Key Results:

Method	Ant	HalfCheetah	Walker2d	Swimmer	Hopper
VPG	13±0	897±531	629±229	48±11	888±209
TRPO	1625±356	2074±741	2006±399	101±29	2391±455
PPO	3372±301	3245±947	2946±944	122±8	2726±886
PG-CMDP	7±4	929±563	597±220	55±19	1039±21
CPPO	3515±247	3680±1121	3194±648	183±46	3145±158

CPPO achieves higher mean returns and enhanced robustness (maintaining approximately 10–20% higher returns than PPO) across transition-mass scaling. Under Gaussian noise levels up to $\sigma=0.5$, CPPO return degradation is limited to $\leq$15%, whereas PPO and TRPO degrade by 25–40%. Under FGSM adversarial attacks, CPPO also outperforms competing baselines. Ablation on $\alpha$ and $\beta$ finds that higher $\alpha$ increases risk aversion (lower CVaR, lower mean return); $\alpha\approx0.9–0.95$ is recommended for a suitable trade-off and adaptive $\beta$ stabilizes training (Ying et al., 2022).

7. Practical Implications and Recommendations

The CVaR constraint introduces two additional scalar variables ($\eta$, $\lambda$) with per-episode complexity $O(N)$ for $\max\{\eta-D_i, 0\}$ and indicator calculations. This overhead is negligible relative to typical network computations. CPPO inherits PPO’s stability properties due to the clipped surrogate. The Lagrange multiplier $\lambda$ only increases significantly when CVaR exceeds $\beta$, further supporting stable optimization behavior if learning rates are selected appropriately.

Hyperparameter tuning guidelines:

• $\alpha$ governs the risk sensitivity; recommend $[0.9,0.99]$ for practical applications.
• $\beta$ should be set below the unconstrained expectation and above the extremal minimum, e.g., using the mean of the worst-performing $\alpha$-fraction of recent episodes.

In summary, CPPO provides a theoretically grounded and practically tractable approach for risk-aware reinforcement learning in continuous domains, simultaneously improving expected return and lower-tail robustness by leveraging the CVaR criterion linked rigorously to the Value Function Range (Ying et al., 2022).