Entropy-Regularized Objectives

• Entropy-regularized objectives are optimization formulations that add an entropy term to the standard objective, promoting convexity and improved exploration in reinforcement learning.
• They transform optimization challenges by smoothing the Bellman operator into a softmax structure, enabling tractable dual representations and stable policy updates.
• Algorithms like TRPO and entropy-regularized policy gradients implement mirror descent and dual averaging methods, ensuring robust convergence properties.

Entropy-regularized objectives are a class of formulations in optimization, reinforcement learning, and related domains that augment a baseline objective (e.g., maximization of expected reward) with a convex regularization term encoding entropy or relative entropy (Kullback-Leibler divergence). The inclusion of entropy regularization has profound theoretical and algorithmic implications: it smooths and convexifies otherwise non-smooth problems, enables tractable dual representations closely related to dynamic programming, provides practical advantages for exploration and stability in learning, and often yields policies with desirable robustness properties.

1. Formulation and Mathematical Foundations

The entropy-regularized objective is constructed by adding a convex regularizer to the standard criterion in Markov decision processes (MDPs) and related frameworks. In the context of average-reward MDPs, the objective becomes

$$\max_{\mu \in \Delta} \left[ \sum_{x,a} \mu(x,a) r(x,a) - \frac{1}{\eta} R(\mu) \right], \tag{1}$$

where $\mu$ is the stationary joint state-action distribution, $r(x,a)$ is the reward, $R(\mu)$ is a convex regularizer (often negative entropy or negative conditional entropy), and $\eta > 0$ controls the regularization strength (Neu et al., 2017).

Two principal choices for $R(\mu)$ are:

• Negative Shannon entropy: $$R(\mu) = -\sum_{x,a} \mu(x,a) \log \frac{\mu(x,a)}{\mu'(x,a)}$$, providing a regularization relative to a reference distribution $\mu'$;
• Negative conditional entropy: $$R_C(\mu) = \sum_{x,a} \mu(x,a) \log \frac{\mu(x,a)}{\nu_\mu(x)}$$, where $\nu_\mu(x) = \sum_a \mu(x,a)$.

Entropy regularization transforms the original policy optimization into a strictly convex problem when $R(\mu)$ is strictly convex, and modifies the geometry of the feasible set in ways conducive to both optimization and statistical estimation.

The dual of this entropy-regularized formulation, particularly with the conditional entropy regularizer, yields a set of nonlinear equations closely related to the BeLLMan optimality equations, but with a log-sum-exp (softmax) structure: $$V^*_\eta(x) = \frac{1}{\eta} \log \sum_a \pi_{\mu'}(a|x) \exp\left[\eta \big(r(x,a)-\rho_\eta^* + \sum_y P(y|x,a) V^*_\eta(y)\big)\right], \tag{2}$$ where $V^*_\eta$ is the value function, $\rho_\eta^*$ is the optimal average reward under the regularized objective, and $\pi_{\mu'}(a|x)$ is the reference policy (Neu et al., 2017).

2. Algorithmic Interpretations: Convex Optimization, Mirror Descent, and Dual Averaging

Entropy-regularized objectives give rise to optimization algorithms with explicit connection to convex optimization schemes:

• Mirror Descent (MD): The update rule,

$$\mu_{k+1} = \arg\max_{\mu \in \Delta} \left[\rho(\mu) - \frac{1}{\eta} D_R(\mu \| \mu_k)\right],$$

uses the Bregman divergence $D_R$ induced by the regularizer $R$. For conditional entropy, this corresponds to a "soft" greedy policy improvement and yields closed-form updates closely related to the exponentiated gradient method (Neu et al., 2017).

• Dual Averaging (DA): The update,

$$\mu_{k+1} = \arg\max_{\mu \in \Delta} [\rho(\mu) - \tfrac{1}{\eta_k} R(\mu)],$$

(with $\eta_k$ increasing over time) formalizes follow-the-regularized-leader schemes and ensures convergence toward the optimal policy when the convex structure is exactly preserved.

These algorithmic frameworks clarify the relationship between popular RL algorithms and fundamental principles in convex optimization. Notably, the "exact" version of Trust-Region Policy Optimization (TRPO) corresponds to mirror descent with the conditional entropy Bregman divergence, providing convergence guarantees that are often lacking in standard policy gradient approaches with approximate surrogates.

3. Policy Optimization Algorithms: TRPO and Entropy-Regularized Policy Gradients

The duality and convexity analysis enable precise characterization of different entropy-regularized reinforcement learning algorithms:

• TRPO (Trust-Region Policy Optimization): When performed exactly, TRPO can be written as

$$\pi_{k+1}(a|x) \propto \pi_k(a|x) \exp\left[\eta A^\pi_\infty(x,a)\right],$$

where $A^\pi_\infty$ is the advantage function of the current policy (Neu et al., 2017). This update is equivalent to mirror descent in the space of state-action distributions and ensures convergence to the entropy-regularized optimum.

• Entropy-Regularized Policy Gradient (PG) Methods: Methods such as A3C with entropy bonus are viewed as approximate dual averaging on a surrogate objective,

$$L(\theta) = \sum_x \nu_{\pi_k}(x) \sum_a \pi_\theta(a|x) \left[ A^\pi_\infty(x,a) - \frac{1}{\eta} \log \pi_\theta(a|x) \right].$$

However, the objective is nonconvex in both $\theta$ and the occupancy $\mu$, and changes at each iteration, possibly leading to poor local optima or divergence. These methods, while effective in practice, are not characterized by the same strong global convergence guarantees as exact mirror descent/TRPO (Neu et al., 2017).

4. Trade-offs, Empirical Behaviour, and Tuning

The regularization parameter $\eta$ directly controls the smoothness of the optimal policy and the exploration-exploitation balance. Empirical investigations reveal:

• Low $\eta$ (strong regularization): Policies are overly stochastic, leading to under-exploitation and slow learning.
• High $\eta$ (weak/no regularization): Fast convergence, yet with over-commitment to potentially suboptimal actions due to premature exploitation.
• Intermediate $\eta$: Best performance, aligning with the optimum in convexity-preserving algorithms (TRPO, dual averaging).

Table: Effects of Regularization Strength (interpreted from (Neu et al., 2017))

$\eta$	Policy Behavior	Convergence	Exploration
Very small	Highly stochastic	Slow	High
Intermediate	Balanced stochasticity	Reliable	Adequate
Very large	Nearly deterministic	Rapid, premature	Low

The convexification effect of entropy regularization is essential: if algorithms break the convex structure (for instance, by using nonconvex surrogates or by disregarding occupancy corrections), then guarantees are lost and empirical performance may degrade or be inconsistent.

5. Theoretical Properties: Duality, Regularized BeLLMan Equations, and Convergence

The introduction of (conditional) entropy regularization allows for a duality-based analysis linking the primal LP and its dual, producing "regularized" BeLLMan equations and revealing structural parallels between policy optimization and dynamic programming. The central dual equation (as above) softens the maximization in the standard BeLLMan operator to log-sum-exp, preserving continuity and facilitating both numerical and theoretical analysis.

Key mathematical expressions, such as: $$V^*_\eta(x) = \frac{1}{\eta} \log \sum_{a} \pi_{\mu'}(a|x) \exp\left\{ \eta\left( r(x,a) - \rho^*_\eta + \sum_y P(y|x,a)V^*_\eta(y) \right) \right\},$$ establish a framework in which optimality criteria correspond to stationary points of convex variational principles, and in which solutions can be interpreted as fixed points of regularized, contractive operators. This underlies convergence guarantees for mirror descent and dual averaging algorithms under appropriate implementation conditions (Neu et al., 2017).

6. Broader Impact, Applications, and Design Principles

Entropy-regularized objectives unify concepts from convex optimization and classical reinforcement learning, clarifying the structure underlying successful algorithms and providing practical guidance:

• Exploration through stochasticity: Entropy terms incentivize randomization, mitigating premature convergence to suboptimal deterministic policies.
• Algorithmic stability: Entropic regularization renders the optimization problem strictly convex, improving numerical stability and sensitivity to estimation error.
• Unified analysis of policy iteration and policy gradient methods: The convex framework distinguishes between globally convergent methods (e.g., “exact” TRPO) and those with potential for instability (e.g., policy gradient with shifting surrogates).
• Guidance for algorithm design: Incorporate entropy regularization so that the convex structure is preserved throughout optimization steps, use appropriate parameter tuning for $\eta$, and recognize limits of performance and convergence when convexity is broken by approximations or heuristics.

Entropy-regularized objectives are increasingly central in modern reinforcement learning, both as a theoretical tool for bridging gaps between dynamic programming and convex optimization, and as a practical technique for ensuring robust, stable, and efficient learning in complex environments.