---
title: Entropy-Regularized Optimal Control
url: https://www.emergentmind.com/topics/entropy-regularized-optimal-control-problem
type: topic
---

# Entropy-Regularized Optimal Control

An entropy-regularized optimal control problem is an extension of classical optimal control in which the objective functional combines traditional state-action costs with a penalty term that measures the divergence between a candidate (controlled) probability law and a reference law, such as the uncontrolled dynamics or a prior. The regularization is typically given by the relative entropy (Kullback–Leibler divergence), its generalizations (e.g., Tsallis entropy), or entropy-like information measures (e.g., transfer entropy). This framework, motivated by both theoretical control and reinforcement learning, enables explicit trade-offs between exploitation (cost minimization) and exploration (policy stochasticity), leading to regularized, randomized controllers with advantageous robustness, tractability, and learning properties.

## 1. Problem Formulation and Theoretical Structure

Given a controlled stochastic process, the entropy-regularized optimal control problem is generally formulated as
\[
\min_Q\; \mathbb{E}^Q[C] + \lambda \mathcal{H}(Q;P)
\]
where $C$ is the cost random variable, $P$ is a reference (typically uncontrolled) probability law, $Q$ is the controlled measure (absolutely continuous w.r.t. $P$), $\lambda>0$ is a regularization weight, and $\mathcal{H}(Q;P) = \mathbb{E}^Q[\log(\frac{dQ}{dP})]$ is the relative entropy.

Under suitable integrability conditions—specifically, when
\[
\mathbb{E}^P[e^{-C}|C|] < \infty,
\]
an explicit solution exists: the optimal $Q^*$ has likelihood ratio
\[
Z^* = \frac{dQ^*}{dP} = \frac{e^{-C}}{\mathbb{E}^P[e^{-C}]}
\]
as shown in [1205.6946]. This solution captures the minimum of a trade-off between the original cost and the informational distance from the baseline law. In the continuous-time case with Wiener reference measure, every absolutely continuous change of measure is identified via Girsanov’s theorem with a drift control $U$, and the entropy cost acquires the form
\[
\mathcal{H}(Q;P) = \frac{1}{2} \mathbb{E}^P\left[\int_0^\infty |U_t|^2\,dt\right].
\]
The resulting control problem is equivalent to minimizing expected cost plus a quadratic control energy penalty, and the optimal feedback law admits explicit representation using Malliavin calculus or martingale techniques:
\[
U_t = -\frac{\mathbb{E}^P[e^{-C} D_t C \mid \mathcal{F}_t]}{\mathbb{E}^P[e^{-C} \mid \mathcal{F}_t]},
\]
where $D_t C$ is the Malliavin derivative of the cost [1205.6946]. This explicit structure generalizes across settings—discrete-time/dynamics, optimal transport, and mean-field games—by varying the entropy measure, the structure of $C$, or the form of $P$.

## 2. Regularization Types and Sparsity

While the canonical form uses Shannon (relative or differential) entropy, more general regularizers are possible:

- **Transfer-entropy regularization:** The objective penalizes information flow from states to controls, as in transfer-entropy-regularized MDPs (TERMDPs). This is formulated for Markov Decision Processes as
\[
\min_{\{q_t(\cdot)\}}\, \sum_t \mathbb{E}[c_t(X_t,U_t)] + \beta I_{m,n}(X^{(T)}\to U^{(T)}),
\]
where $I_{m,n}$ is a multi-step transfer-entropy and $\beta$ controls the information cost [1708.09096].

- **Tsallis entropy regularization:** Here, the classic entropy is replaced by Tsallis entropy, parametrized by $q$, yielding control policies with sparse support (i.e., assigning zero probability to many actions). For Tsallis $q<1$ and regularization parameter $\lambda$, the optimal policy becomes
\[
\pi^*(u|x) \propto \exp_q(-Q_k(x,u)/\lambda + C_k(x)),
\]
with bounded support, promoting sparsity compared to the dense softmax form produced by Shannon entropy [2403.01805, 1802.03501].

- **Relative entropy in robust control and games:** In robust control or order execution with market uncertainty, the problem’s min-max structure becomes
\[
\max_{v} \min_{\pi \ll \pi^0} \; \mathbb{E}[R(v,\pi) + \frac{1}{\beta} \mathrm{KL}(\pi||\pi^0)]
\]
where $\pi^0$ is a prior over adversarial/environment actions and $\beta$ quantifies market resilience [2311.06476].

The type and strength of entropy regularization directly influence the policy’s degree of exploration, the sparsity pattern, and safety-related margin.

## 3. Methods of Solution and Algorithmic Strategies

The variational structure of entropy-regularized problems allows explicit or semi-explicit characterization of optimal policies:

- **Feedback expressions via Malliavin calculus** when the cost is smooth [1205.6946].
- **Dynamic programming and variational inequalities** in continuous/discrete time, leading e.g. to “soft” HJB equations whose minimizer is always (when the reward and system are quadratic/affine) a Gaussian policy:
  \[
  g^*(t,x,u) \propto \exp\left(-\frac{1}{\alpha}(p\cdot f(x,u) + r(x,u))\right),
  \]
  where $\alpha$ is the entropy-regularization parameter [2009.13097].
- **Ricatti equations for LQ problems (with entropy term):**
  The value function remains quadratic but acquires extra linear terms, and the optimal policy is a Gaussian distribution whose mean and variance are explicitly determined by solution of a modified Riccati equation [2009.13097, 2309.10662, 2403.01805].
- **Iterative forward-backward or block coordinate algorithms:** In the TERMDP setting, a forward–backward recursion generalized from the Arimoto–Blahut algorithm is employed to find stationary points of the non-convex, entropy-regularized MDP objective [1708.09096].
- **Gradient flows in measure spaces:** The optimization may be recast as a gradient flow of probability measures in the Wasserstein metric space, with entropy regularization ensuring geometric convergence under convexity [2006.05956].

For trajectory optimization and path integral methods, entropy regularization lifts the problem from policy-space to trajectory distribution-space, allowing direct path-consistency learning, sample-based estimation (e.g., Monte Carlo, cross-entropy), or derivative-free evolutionary approaches [2007.03960, 2110.02647].

## 4. Applications and Explicit Examples

Entropy-regularized optimal control is applicable in numerous contexts:

- **Obstacle avoidance and rare-event steering:** Hard constraints are imposed by infinite costs, handled by the explicit Gibbs-type change of measure [1205.6946].
- **Mean-variance portfolio optimization with exploration:** A continuous-time mean-variance problem with Lévy jumps is regularized by differential entropy of the randomized control, leading to an optimal (Gaussian) distributional control and explicit SDE dynamics, directly linked to the trade-off between learning and performance [2312.13409].
- **Sample-efficient policy learning:** Algorithms such as Regularized Policy Gradient (RPG) or Sample-Based RPG deploy entropy regularization to ensure global convergence of policy-gradient optimization, even in non-convex LQ control with multiplicative noise and unknown parameters [2510.02896].
- **Optimal transport and large-scale control:** Large problems are decomposed into entropic optimal transport problems over overlapping subdomains, solved independently (e.g., via Sinkhorn), with entropy regularization guaranteeing linear convergence in KL divergence and enabling parallel/distributed algorithms [2001.10986].
- **Entropy-regularized stopping and mean field games:** Classical bang-bang stopping is smoothened by cumulative residual entropy or entropy of random stopping probabilities, reformulating the problem as a singular control with “finite fuel,” suitable for policy iteration or fictitious play learning in mean-field games [2408.09335, 2509.18821].

## 5. Regularization Parameters, Limits, and Trade-Off Analysis

The entropy regularization parameter fundamentally controls the trade-off between exploitation (cost minimization) and exploration (policy stochasticity):

- **Temperature parameter role:** As the regularization strength ($\lambda$, $\beta$, or $\alpha$) increases, the policy becomes more randomized (higher entropy, wider support); as it vanishes, the solution converges (often uniformly) to the original, unregularized optimal control, typically deterministic or bang-bang [1205.6946, 2408.09335, 2309.10662].
- **Exploration-exploitation trade-off:** Optimal scheduling or decay of the regularization parameter during learning can guarantee asymptotic optimality with optimal regret rates, e.g., $\mathcal{O}(\sqrt{N}\log N)$ in LQ RL [2208.04466].
- **Sparsity vs. robustness:** Tsallis entropy allows tuning between strictly sparse and fully exploratory laws depending on the deformation parameter $q$ [2403.01805].
- **Information-constrained planning:** Transfer-entropy or KL penalties formalize limitations of information transmission and are grounded in communication theory and thermodynamics, quantifying the “price of information” [1708.09096].

## 6. Computational and Algorithmic Implications

Entropy regularization supports computational tractability and scalable algorithm design:

- **Explicit feedback and sample-based methods:** Many problems admit closed-form feedback, facilitating efficient simulation and Monte Carlo approximations when integrals cannot be computed analytically [1205.6946, 2007.03960].
- **Grid-free and multiscale solvers:** For “soft” HJB equations, generalized Hopf–Lax formulas enable grid-free dynamic programming, circumventing the curse of dimensionality [2009.13097].
- **Adaptive sparsity and parallelization:** Domain decomposition algorithms with entropic regularization show linear convergence, and fast solvers via adaptive sparsity and multi-scale refinement are directly transferable to entropy-regularized control [2001.10986].
- **Convergence of RL algorithms:** The presence of entropy regularization guarantees global convergence of policy iteration and policy gradient methods under appropriate regularity or convexity, even for measure-valued controls or in backward stochastic settings [2209.07059, 2411.13219, 2510.02896].
- **Policy iteration and fictitious play:** In mean-field or multi-agent settings, entropy-regularized reformulations ensure uniqueness and stability of equilibria, underlying robust fictitious play and model-free learning schemes [2509.18821].

## 7. Broader Impact, Generalizations, and Connections

Entropy-regularized optimal control is central to contemporary reinforcement learning (e.g., Soft Actor–Critic), robust control, stochastic games, path integral methods, optimal transport, and mean-field game theory. The formulation justifies randomization/exploration in learning agents, smoothens non-convex objectives to facilitate optimization, and provides interpretable Bayesian connections (invariant measure is a Gibbs posterior). The framework unifies variational, dynamic programming, and probabilistic inference approaches and suggests principled regularizations (e.g., Tsallis or transfer-entropy) beyond Shannon entropy tailoring policies for safety, sparsity, or information constraints. Its theoretical tractability and practical benefits, such as explicit feedback laws, scalable algorithms, and convergence guarantees, make entropy-regularized optimal control both foundational and indispensable for modern stochastic control, RL, and large-scale decision-making problems.

Source: https://www.emergentmind.com/topics/entropy-regularized-optimal-control-problem