Regularized Bellman Operators

• Regularized Bellman operators are modifications of the standard Bellman operator that incorporate structured penalties and incentives to enhance action-gap, robustness, and stability.
• They encompass gap-increasing, stochastic, entropy, and twice-regularized formulations with formal guarantees such as contraction, monotonicity, and optimality preservation.
• Empirical results demonstrate improved performance in tasks like Atari and control environments, evidencing faster convergence, increased action gaps, and reduced value overestimation.

A regularized Bellman operator is a modification of the standard Bellman operator for Markov Decision Processes (MDPs), designed to incorporate forms of regularization into the dynamic programming or reinforcement learning update. These operators introduce structured penalties or incentives—such as gap-increasing corrections, convex surrogates, randomized penalties, or robustification against model uncertainty—directly within the Bellman backup. This architectural modification enhances action-gap, robustness, stability, or bias-variance characteristics relative to the classical formulation, and it underlies contemporary advances in robust reinforcement learning, entropy-regularized algorithms, and robust MDP formulations.

1. Formal Definitions and Classes of Regularized Bellman Operators

Let $(S,A,P,r,\gamma)$ be a discounted MDP with finite state and action spaces. The standard Bellman optimality operator for $Q:S\times A\rightarrow\mathbb{R}$ is

$$(\mathcal{T}Q)(s,a) = r(s,a) + \gamma\,\mathbb{E}_{s'\sim P(\cdot|s,a)}\left[\max_{b\in A} Q(s',b)\right].$$

Regularized operators instantiate algorithmic variants by augmenting or modifying the backup, leading to several canonical classes:

1.1. Gap-Increasing ("Consistent" and $\alpha$-Parametrized) Operators

A family of gap-increasing operators characterized by

$$(\mathcal{T}_\alpha Q)(x,a) = (\mathcal{T}Q)(x,a) - \alpha\,[V(x) - Q(x,a)],\quad V(x) = \max_b Q(x,b), \quad\alpha\in[0,1).$$

This includes the Consistent Bellman operator, which can be written as

$$(\mathcal{T}_C Q)(x,a) = (\mathcal{T}Q)(x,a) - \gamma\,P(x|x,a)\,[V(x)-Q(x,a)]$$

and Baird’s advantage learning as the special case $\mathcal{T}_{\rm AL} = \mathcal{T}_\alpha$ (Bellemare et al., 2015).

1.2. Stochastic ("RSO") Gap-Regularized Operators

The robust stochastic operator (RSO) family generalizes $\mathcal{T}_\alpha$ to stochastic penalties: $$(\mathcal{T}_{\beta_k} Q)(x,a) = R(x,a) + \gamma\,\mathbb{E}_{x'\sim P(\cdot\mid x,a)}\left[\max_b Q(x',b)\right] - \beta_k \,[V_k(x)-Q(x,a)],$$ with $\{\beta_k\}$ a sequence of independent nonnegative random variables, $\mathbb{E}[\beta_k]=\bar{\beta}_k\in[0,1]$ (Lu et al., 2018).

1.3. Entropy and Convex Regularized Operators

For a state-conditional convex regularizer $\Omega:\Delta_A\rightarrow\mathbb{R}$,

$$[T_{*,\Omega} v](s) = \max_{\pi_s\in\Delta_A}\left\{\langle\pi_s,\,r_s+\gamma P_sv\rangle-\Omega(\pi_s)\right\},$$

with Legendre–Fenchel dual $$\Omega^*(q) = \max_{\pi}\left\{\langle\pi,q\rangle-\Omega(\pi)\right\}$$, so $T_{*,\Omega} v = \Omega^*(r_s+\gamma P_sv)$ (Geist et al., 2019).

1.4. Twice-Regularized Operators (Policy and Value Regularization)

Letting $\Omega_s$ (policy) and $\Phi_s$ (possibly value-dependent) be strongly-convex,

$$T^{*,\Omega,\Phi}[v](s) = \max_{\pi_s\in\Delta_A}\left\{r_0^\pi(s)+\gamma(P_0^\pi v)(s)-\Omega_s(\pi_s)-\Phi_s(\pi_s;v)\right\},$$

where $\Phi_s$ often encodes sensitivity to transition uncertainty (Derman et al., 2023, Derman et al., 2021).

2. Theoretical Properties: Gap, Contraction, and Robustness

Regularized Bellman operators possess several invariants and monotonicity properties central to stability and performance.

2.1. Contraction and Monotonicity

For strongly convex $\Omega$, both $T_{*,\Omega}$ and $T_{\pi,\Omega}$ are $\gamma$-contractions (in sup-norm). Generally, for twice-regularization, under bounded transition-uncertainty radii, $T^{*,\Omega,\Phi}$ is a strict contraction: $$\|T^{*,\Omega,\Phi} v_1 - T^{*,\Omega,\Phi} v_2\|_\infty \le (1-\epsilon)\|v_1-v_2\|_\infty$$ for some $\epsilon>0$ (Derman et al., 2023, Derman et al., 2021).

2.2. Optimality Preservation and Action Gap

For gap-increasing operators, Theorem 1 of (Bellemare et al., 2015) shows that if $\mathcal{T}'Q \le \mathcal{T}Q$ and $\mathcal{T}'Q \ge \mathcal{T}Q-\alpha[V(x)-Q(x,a)]$, then the sequence $V_k(x) = \max_a Q_k(x,a)$ converges to $V^*(x)$ and any suboptimal $a$ remains suboptimal in the limit. The action gap is provably nondecreasing: $$\liminf_{k\rightarrow\infty}[V_k(x)-Q_k(x,a)] \ge V^*(x)-Q^*(x,a).$$

RSO operators in (Lu et al., 2018) retain these guarantees almost surely, even under randomized penalties. The convex order over $\{\beta_k\}$ yields stochastic gap magnification.

2.3. Robustness and Regularization Duality

Twice-regularized operators, as in $R^2$-MDPs, exactly recover robust MDP values under rectangular uncertainty in rewards and transitions, mapping robust min-max constraints to convex dual regularizers (via the support function) (Derman et al., 2023, Derman et al., 2021).

3. Algorithmic Schemes and Planning Implications

Regularized Bellman operators support the following algorithms, typically with only minor overhead per update:

3.1. Value and Policy Iteration

Incorporate regularized (single or twice) Bellman operators into value iteration: $v_{t+1} \leftarrow T^{*,\Omega,\Phi}[v_t]$ or modified policy iteration: $$\pi_{k+1} \leftarrow \arg\max_{\pi_s} T^{\pi,\Omega,\Phi}[v_k](s),\quad v_{k+1}\leftarrow (T^{\pi_{k+1},\Omega,\Phi})^m v_k$$

3.2. Q-Learning and Stochastic Updates

For model-free temporal-difference learning,

• Gap-increasing/RSO update: sample $\beta_k$, adjust TD target as in

$$\text{target}_{\text{RSO}} = r + \gamma V_{k+1} - \beta_k(V_k - Q(x,a))$$

and update $Q(x,a)$ (Lu et al., 2018).

• Twice-regularized Q-learning: modify TD target to include both reward and value-dependent penalties (Derman et al., 2023).

3.3. Bi-level and Function Approximation Algorithms

With linear function approximation, regularized Bellman operator composition with a projection is not in general a contraction. A bi-level optimization approach, coupling a fast (Bellman equation) loop and a slower (projection) loop with carefully tuned learning rates, yields nonasymptotic convergence guarantees to stationary points (Xi et al., 2024).

4. Empirical Performance and Benchmarking Evidence

Regularized Bellman operators have demonstrated robust empirical superiority on an array of tasks.

Task/Env.	Algorithm	Action Gap (Acrobot)	Avg. Score (CartPole)	Lunar Lander Score
Classical Bellman	Q-learning	0.1436	—	-241.9
Consistent	(Bellemare+)	0.1263	—	-188.4
RSO (U[0,2))	(Lu et al., 2018)	0.9004	$>$195, faster, lower variance	-167.5

• In Atari-2600 (60 games), advantage learning and persistent variants yield median score increases of 8.4–9.1% (mean 27–32%), with systematically enlarged action gaps and suppressed value overestimation (Bellemare et al., 2015).
• Twice-regularized (R$^2$) MDPs empirically match or outperform robust dynamic programming at a fraction of computational overhead (Derman et al., 2023, Derman et al., 2021).

5. Interpretations, Extensions, and Unified Frameworks

Regularized Bellman operators subsume and extend gap-increasing operators, robust MDP frameworks, and entropy/KL-regularized RL under a unified lens.

• Margin-based penalties as in $\mathcal{T}_\alpha$ or RSO correspond to SVM-style regularization terms on Bellman residuals, robustifying greedy policies to value estimation noise (Bellemare et al., 2015, Lu et al., 2018).
• Convex duality construction used in (Geist et al., 2019) and (Derman et al., 2023) connects regularization to mirror descent and proximal optimization updates for the policy, resulting in a rigorous equivalence between regularization and robustness (with reward and/or transition uncertainty).
• Twice-regularized frameworks ($R^2$-MDPs) extend single regularization to handle model uncertainty in both reward and transition, with algorithmic complexity remaining comparable to standard DP (Derman et al., 2023, Derman et al., 2021).
• The shift property and monotonicity ensure standard dynamic programming convergence guarantees persist.

A unified perspective identifies regularized Bellman operators as a key modeling tool in bias-variance trade-offs, robustness control, and modern policy optimization (Geist et al., 2019, Bellemare et al., 2015, Lu et al., 2018).

6. Open Directions and Limitations

• Most gap-increasing and RSO theory and empirical results are established in the tabular setting; scalable application to deep RL remains an active direction (Lu et al., 2018).
• Choice of regularization strength or stochastic penalty variance requires balancing early-learning speed and late-stage convergence (Lu et al., 2018).
• While regularized Bellman operators often mitigate maximization bias and propagate action gaps, projection-based schemes with function approximation may break the contraction property, necessitating careful two-timescale or bi-level methods for stability (Xi et al., 2024).
• In practice, entropy- and convex-regularized operators are foundational for scalable soft RL algorithms (e.g., SAC, TRPO, proximal policy optimization).

Regularized Bellman operators encapsulate a broad algorithmic spectrum, enabling theoretical advances and practical robustness in sequential decision-making, and continue to unify and sharpen the theoretical underpinnings of modern reinforcement learning.