Deviation-based Objectives (SD3)

• Deviation-based Objectives are optimization formulations that integrate explicit deviation measures with canonical cost functions to achieve a tradeoff between optimality and robustness.
• They are applied across inverse optimization, risk-averse MDPs, and deep reinforcement learning, enabling tailored risk management and control strategies.
• Specialized algorithms, including combinatorial Newton-type procedures and softmax-based actor-critic methods, efficiently compute these objectives under various constraints.

Deviation-based objectives (often abbreviated as DB-Objectives or as in reinforcement learning, SD3 for "Softmax Deep Double Deterministic Policy Gradients") denote a class of optimization objectives and algorithmic frameworks in which the performance metric explicitly involves a deviation or dispersion measure of the output, policy, or cost vector. Such objectives provide fine-grained control over the tradeoffs between optimality—defined via canonical cost or reward functions—and robustness, risk sensitivity, or regularization, quantified through a deviation term. These formulations appear in domains as diverse as inverse combinatorial optimization, risk-averse Markov decision processes, quantitative risk management, and continuous control with function approximation.

1. Deviation-based Objectives in Inverse Optimization

Deviation-based objectives in inverse optimization emphasize modifications of cost vectors via deviation measures rather than standard summative norms. Let $S$ denote a ground set of $n$ elements, $F\subseteq 2^S$ a family of feasible solutions, $F^*\in F$ the target solution to be made optimal, and $c\in\mathbb{R}^S$ the original cost vector. The goal is to find a deviation vector $p\in\mathbb{R}^S$ such that $F^*$ becomes optimal under the modified cost $c' = c - p$, while penalizing the magnitude of $p$ by a deviation-based objective and enforcing componentwise bounds $\ell(s)\leq p(s)\leq u(s)$.

Two principal deviation-based objectives are studied in this context (Bérczi et al., 2023):

• Weighted Bottleneck Hamming Distance ($H_{\infty,w}$): The objective is $H_{\infty,w}(p)=\max\{w(s): p(s)\neq 0\}$ with $w\in\mathbb{R}^S_+$, penalizing only the maximal “weight” among changed coordinates. This measure is non-separable and “bottleneck” in structure.
• Weighted $\ell_\infty$-Norm ($\|p\|_{\infty,w}$): Defined as $$\|p\|_{\infty,w} = \max \{ w(s) \cdot |p(s)| : s\in S \}$$, this objective penalizes the maximum weighted amplitude among all deviations.

These objectives differ fundamentally from conventional $\ell_1$ or $\ell_2$ distances: they stress extremal or localized deviations, which is critical in scenarios where the largest change dominates cost or feasibility considerations.

2. Algorithmic Strategies for Deviation-based Objectives

Optimal adjustment of cost vectors under deviation-based constraints requires specialized combinatorial algorithms distinct from standard convex optimization or linear programming.

For weighted bottleneck Hamming distance, a combinatorial Newton-type iterative procedure is deployed (Bérczi et al., 2023). The algorithm incrementally lifts a bottleneck threshold $\delta$ across coordinates, updating the deviation vector and checking for the optimality of $F^*$ at each stage using a black-box oracle for the base optimization problem. Analysis (Lemma 2.1 et seq.) shows that only finitely many $\delta$ values—those matching weights in $w$—need be considered, guaranteeing at most $n$ iterations.

For the weighted $\ell_\infty$-norm, a min–max characterization expresses the optimal deviation in terms of the worst-case optimality gap divided by an effective weight over set differences (Theorem 3.10). The Newton-type algorithm (Algorithm 2) advances a deviation threshold $d_i$, adjusting $p$ until the target solution achieves minimal cost. Pseudo-polynomial complexity arises in the general weighted case; unit weights yield strongly polynomial running times, provided the underlying combinatorial oracle is efficient. Both algorithms extend naturally to the case of multiple cost functions by solving parallel inverse problems and selecting the maximum critical threshold required.

These combinatorial schemes require only primitive vector operations and repeated calls to canonical optimization oracles (e.g., shortest path, minimum spanning tree), eschewing the need for explicit LPs or binary search (Bérczi et al., 2023).

3. Deviation-based Objectives in Risk-averse Markovian Models

Deviation-based objectives find broad application in risk-averse planning in Markov decision processes (MDPs). The archetypal formulation is to maximize

$\mathbb{E}[R] - \lambda \cdot \text{Dev}(R)$

where $R$ is the random variable representing total accumulated reward, $\lambda \geq 0$ quantifies risk aversion, and $\text{Dev}$ is a deviation measure.

Several deviation measures and their algorithmic and structural implications are systematically analyzed (Baier et al., 9 Jul 2024):

• Variance-Penalized Expectation (VPE): Quadratic penalty on reward variance; causes optimal policies to become eventually reward-minimizing (ERMin), discouraging further accumulation of rewards in high-reward states.
• Semi-Variance: One-sided penalty for downside risk only; structurally behaves similarly to the variance, also enforcing eventual minimization of further rewards.
• Mean Absolute Deviation (MAD): Penalizes average distance from the mean. For $\lambda \leq 1/2$, it yields eventually reward-maximizing (ERMax) policies, ensuring that the policy never penalizes further increases in already high rewards—a desirable risk-averse property absent in VPE or semi-variance.
• Semi-MAD: Focuses only on below-mean deviations and relates directly to MAD via a constant factor.
• Threshold-based Penalty: Penalizes only for falling below a fixed threshold $t$, inducing finite-memory ERMax policies and allowing solution by pseudo-polynomial MDP unfolding.

Complexity and required scheduler structure depend sensitively on the choice of deviation measure. Mean absolute deviation and threshold-based penalties are unique among polynomially expressible objectives in providing intuitive, sound risk-averse control without penalizing high-reward tails. Decision problems for these objectives range from PP-hard for MDPs to pseudo-polynomial or EXPSPACE in the general case.

4. Shortfall Deviation Measures in Risk Management

Shortfall Deviation (SD) and Shortfall Deviation Risk (SDR), introduced by Righi and Ceretta, furnish alternative deviation-based risk measures built atop expected shortfall (ES) (Righi et al., 2015). For a random variable $X$ and confidence level $0<\alpha<1$, define ES and its associated mean: $$q_\alpha(X) = F_X^{-1}(\alpha), \quad e_\alpha(X) = \mathbb{E}[X|X\leq q_\alpha(X)] = -\text{ES}_\alpha(X).$$ The shortfall deviation of order $p$ is

$$\text{SD}_\alpha(X) = \| X - e_\alpha(X) \|_{L^p} = \left( \mathbb{E}[| X - e_\alpha(X) |^p] \right)^{1/p}.$$

The shortfall deviation risk integrates a penalty scaled by dispersion in the ES tail: $$\text{SDR}_\alpha(X) = \text{ES}_\alpha(X) + (1-\alpha)^\beta \text{SD}_\alpha(X).$$ SD is a generalized deviation measure; SDR is a coherent risk measure satisfying translation invariance, monotonicity, subadditivity, positive homogeneity, relevance, and law-invariance. SDR admits a dual representation as a supremum of expected losses over a convex set of absolutely continuous measures and a spectral (weighted-ES) representation via Kusuoka's theorem.

Compared to standard metrics (VaR, ES), SDR provides enhanced resilience in heavy-tailed settings, with a penalty term that dominates in extreme tails ($\alpha\downarrow 0$), thereby constructing a more robust capital buffer.

5. Softmax Deep Double Deterministic Policy Gradients (SD3) in RL

In deep reinforcement learning for continuous control, deviation-based objectives are instantiated via the Boltzmann softmax operator, which replaces the max in value estimation with a “weighted average” to incorporate function smoothness and bias control (Pan et al., 2020). For a Q-function $Q(s,\cdot)$ over action space $\mathcal{A}$: $$Q_\text{soft}(s) = \text{softmax}_\beta[Q(s,\cdot)] = \frac{ \int_{a \in \mathcal{A}} e^{\beta Q(s,a)} Q(s,a) da }{ \int_{a\in \mathcal{A}} e^{\beta Q(s,a)} da }$$ with inverse temperature $\beta > 0$. As $\beta\to\infty$, this operator approaches $\max_a Q(s,a)$; for $\beta \to 0$ it yields the mean. This operator has several effects:

• Theoretical guarantees bound its deviation from the true max and establish value iteration convergence to $V^*$ with error $O(1/((1-\gamma)\beta))$.
• In actor-critic algorithms, use of the softmax operator smooths the optimization landscape, empirically converting highly non-convex actor losses into smoother, nearly convex basins.
• SD2 (Softmax DDPG) and SD3 (Softmax Double DDPG) leverage this operator. SD3 in particular utilizes double critics and a softmax-over-min ensemble for target Q-value computation, mitigating both overestimation (as in DDPG) and underestimation bias (as in TD3).

Empirical studies on standard continuous-control tasks demonstrate that SD3 achieves higher sample efficiency and final rewards relative to state-of-the-art baselines (DDPG, TD3, SAC), while reducing outcome variance across seeds and improving policy stability (Pan et al., 2020).

6. Summary Table of Deviations in Algorithmic and Risk Contexts

The following table contrasts core deviation-based objectives discussed above by domain, structure, and main features:

Domain	Objective / Measure	Core Properties / Use
Inverse Opt	Bottleneck Hamming ($H_{\infty,w}$), $\ell_\infty$-norm	Localized/maximal deviation, combinatorial Newton-type algorithms (Bérczi et al., 2023)
MDPs	Variance, semi-variance, MAD, semi-MAD, TB-penalty	Scheduler structure (ERMin/ERMax), EXPSPACE/PP-hard, exact risk aversion (Baier et al., 9 Jul 2024)
Risk Mgmt	Shortfall Deviation (SD), SDR	Generalized deviation and coherent risk, spectral/dual forms (Righi et al., 2015)
Deep RL	Boltzmann softmax Q, SD3	Smooth value estimation, bias mitigation, sample efficiency (Pan et al., 2020)

7. Connections and Practical Implications

Deviation-based objectives unify the control of extremal versus average behavior across optimization, control, and risk applications. In inverse optimization, these objectives enable strong and efficiently computable guarantees for the minimum deviation needed to ensure solution optimality. In risk theory, deviation measures introduce sensitivity to tail behavior beyond classical moments, yielding measures such as SDR that combine tail expectation and dispersion for enhanced risk management. In MDPs, only deviation measures with carefully selected penalty regimes (e.g., MAD with $\lambda\leq 1/2$) avoid pathological scheduler structures, revealing subtle interactions between deviation penalties and policy optimality.

A notable aspect across domains is the frequent existence of efficiently computable Newton-type or combinatorial iterative algorithms, provided base oracle access is available. This architectural principle recurs from combinatorial inverse optimization (Bérczi et al., 2023) to deep RL ensembles (Pan et al., 2020), highlighting the algorithmic tractability of many non-separable deviation structures under mild assumptions.

The systematic paper of deviation-based objectives is critical for settings demanding robust, interpretable, or risk-aware behavior, providing an extensive spectrum of tractable yet expressive formulations.