Entropy-Augmented Bellman Operator

• Entropy-Augmented Bellman Operator is a core tool in entropy-regularized reinforcement learning that adds an entropy bonus to encourage exploration.
• It replaces the standard max operator with a log-sum-exp function, ensuring smooth policy updates and strong contraction properties for convergence.
• Applicable in both discrete and continuous domains, it underpins algorithms like soft actor-critic, maximum entropy DDP, and linearly solvable MDPs.

The entropy-augmented Bellman operator, often referred to as the "soft" or "maximum-entropy" Bellman operator, plays a central role in entropy-regularized reinforcement learning (RL), optimal control, and stochastic planning. By augmenting the standard Bellman recursion with an entropy or temperature term, this operator induces stochasticity and improved exploration behavior in the resulting policy, while also inheriting strong contraction and convergence properties fundamental to dynamic programming. The entropy-augmented Bellman operator is a canonical instance of the broader class of non-linear Bellman operators and appears in both discrete-time and continuous-time control settings, including maximum-entropy value iteration, soft actor-critic methods, and maximum entropy differential dynamic programming (ME-DDP) (Hasselt et al., 2019, So et al., 2021, Kim et al., 2020).

1. General Non-Linear Bellman Operators

The foundational setting for entropy-augmented Bellman operators is the Banach space of bounded real-valued functions over a finite state space $S$, equipped with the sup-norm $\|v\|_\infty = \max_{s \in S}|v(s)|$. A general non-linear Bellman operator $T^f$ arises as:

$$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$

for a measurable function $f:\mathbb{R} \times \mathbb{R} \to \mathbb{R}$.

Under the following structural assumptions:

• Monotonicity: for fixed $r$, $v \mapsto f(r, v)$ is non-decreasing,
• Lipschitz continuity: $$\exists \kappa \in [0,1): ~|f(r, v) - f(r, u)| \leq \kappa |v-u|$$ for all $r, v, u$,
• Bounded rewards: $R_{min} \leq R(s,a) \leq R_{max} < \infty$,

the operator $\|v\|_\infty = \max_{s \in S}|v(s)|$0 is a $\|v\|_\infty = \max_{s \in S}|v(s)|$1-contraction with respect to the sup-norm, guaranteeing a unique fixed point $\|v\|_\infty = \max_{s \in S}|v(s)|$2 and geometric convergence of value iteration (Hasselt et al., 2019).

2. The Entropy-Augmented Bellman Operator: Discrete-Time Case

The archetypal entropy-augmented ("soft") Bellman operator in discrete time, as used in maximum-entropy RL (e.g., soft Q-learning, soft actor-critic), is defined as:

$\|v\|_\infty = \max_{s \in S}|v(s)|$3

where $\|v\|_\infty = \max_{s \in S}|v(s)|$4 is the temperature or entropy coefficient, and $\|v\|_\infty = \max_{s \in S}|v(s)|$5 is the discount factor. This operator replaces the hard max/min in the traditional Bellman update with a log-sum-exp soft maximum, encoding an entropy bonus.

The operator can equivalently be viewed as optimizing over stationary policies:

$\|v\|_\infty = \max_{s \in S}|v(s)|$6

where $\|v\|_\infty = \max_{s \in S}|v(s)|$7 is the Shannon entropy.

The contraction factor of $\|v\|_\infty = \max_{s \in S}|v(s)|$8 is $\|v\|_\infty = \max_{s \in S}|v(s)|$9, independent of $T^f$0:

$T^f$1

Consequently, value iteration using $T^f$2 converges geometrically to a unique entropy-regularized value function $T^f$3:

$T^f$4

These properties, including fixed-point uniqueness and contraction, guarantee stability and facilitate the extension of the method to sampled and approximate algorithms (Hasselt et al., 2019).

3. Maximum Entropy Bellman Operators in Continuous-Time and HJB Formulation

In continuous-time deterministic optimal control, the entropy-augmented (soft) Bellman operator is characterized via the Hamilton-Jacobi-Bellman (HJB) equation:

$T^f$5

where $T^f$6 is the system dynamics and $T^f$7 is the running reward. The optimal control distribution is a Boltzmann/Gibbs law:

$T^f$8

This leads to several key properties:

• The soft HJB defines a well-posed PDE whose unique viscosity solution coincides with the optimal entropy-augmented value function.
• The optimal policy is Gaussian in control-affine and linear-quadratic cases (Kim et al., 2020).
• In the zero-temperature limit $T^f$9, the entropy-augmented HJB recovers the standard deterministic HJB by Laplace principle.

A grid-free solution to the soft HJB can be constructed using a generalized Hopf-Lax formula, given suitable convexity (Kim et al., 2020).

4. Algorithmic Embeddings and Policy Classes

Entropy-augmented Bellman operators underpin a variety of algorithmic frameworks:

• Maximum Entropy Differential Dynamic Programming (ME-DDP): In discrete-time, DDP with entropy-regularized backups maintains the forward-backward sweep structure of classical DDP but computes local Gaussian policies with means and covariances derived from entropy-augmented Q-function approximations. The resulting policy has the form $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$0, where $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$1 is the mean update and $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$2 is the local Hessian. The soft backup introduces an additional entropy term into the quadratic value increment. A multimodal extension (MME-DDP) operates on multiple nominal trajectories and combines policies and value functions via log-sum-exp to escape local minima (So et al., 2021).
• Linearly Solvable MDPs: With desirability variables $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$3 and exponentiated reward densities, the entropy-augmented Bellman equation transforms into a linear integral equation, central to linearly solvable stochastic control.
• Model-Free RL via Adaptive Dynamic Programming: In continuous-time LQ settings, entropy-augmented Bellman equations enable adaptive estimation of value function parameters from data, yielding policies with closed-form Gaussian exploration (Kim et al., 2020).

5. Fixed Point and Contraction Properties

A defining property of the entropy-augmented Bellman operator in both discrete and continuous settings is geometric convergence to a unique fixed point under mild regularity. For discrete-time, the contraction rate is $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$4, implying logarithmic complexity in $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$5 to reach $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$6-accuracy via value iteration. This carries over to approximate algorithms (e.g., TD learning, DQN, soft actor-critic), where asynchronicity or sampling does not invalidate the contraction-based guarantees under common stochastic approximation conditions (Hasselt et al., 2019).

6. Zero-Temperature Limit and Policy Structure

In the limit $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$7, the entropy-augmented Bellman operator reduces to the classical Bellman operator:

$$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$8

This recovers deterministic optimal policies and standard dynamic programming, with entropy regularization interpreted as a smoothing of the hard maximum operation. For control-affine and LQ cases, the optimal policy transitions from Gaussian with covariance $$(T^f v)(s) = \mathbb{E}_{A \sim \pi(\cdot|s), S' \sim P(\cdot|s,A)}[f(R(s,A), v(S'))],$$9 to a Dirac measure concentrated at the argmax action or control (Kim et al., 2020).

7. Practical Implications and Connections

The entropy-augmented Bellman operator offers several practical benefits:

• Provable stability and uniqueness of the entropy-regularized value function.
• Implicit information-theoretic exploration, as entropy regularization promotes diversified policies.
• Compatibility with both tabular, parametric, and sampled RL algorithms.
• Direct connections to linearly solvable control, compositional planning via desirability summation, and exploration strategies in continuous-time model-free RL (So et al., 2021, Kim et al., 2020).

A plausible implication is that the entropy-augmented Bellman operator provides a mathematically robust and algorithmically advantageous foundation for scalable exploration in RL, extending to a broad range of discrete and continuous control tasks.