---
title: Entropy-Regularized Mean-Variance Objective
url: https://www.emergentmind.com/topics/entropy-regularized-mean-variance-objective
type: topic
---

# Entropy-Regularized Mean-Variance Objective

import requests, textwrap, re, xml.etree.ElementTree as ET, json, math
url='http://export.arxiv.org/api/query?search_query=all:%22entropy-regularized%20mean-variance%22&start=0&max_results=5'
try:
    r=requests.get(url, timeout=20)
    print(r.status_code)
    print(r.text[:500])
except Exception as e:
    print("ERR", e)



to=python code
to=python code
to=python code
to=python code
The entropy-regularized mean-variance objective is a continuous-time stochastic control criterion that augments a mean-variance terminal objective with an entropy term on randomized controls. In the portfolio formulations considered in "Continuous-Time Mean-Variance Portfolio Selection: A Reinforcement Learning Framework" [1904.11392], "Entropy Regularization as Robustness under Bayesian Drift Uncertainty" [2602.16862], and "Electricity Virtual Bidding Strategy Via Entropy-Regularized Stochastic Control Method" [2303.02303], the control at each time is not a point action but a probability density over holdings or allocations, and the objective combines terminal quadratic loss with either a Shannon-entropy penalty of the form $\int \pi \ln \pi$ or, equivalently, an entropy reward written through $H(\pi_t)=-\int \pi_t(u)\ln\pi_t(u)\,du$.

## 1. Canonical objective and control-theoretic formulation

In the basic continuous-time mean-variance formulation, discounted wealth is driven by a risky allocation and Brownian noise. For a single risky asset, the classical controlled wealth SDE is
$$
dX_t^u=\sigma\,u_t\,(\rho\,dt+dW_t),\qquad X_0^u=x_0,
$$
where $\rho=(\mu-r)/\sigma$ is the Sharpe ratio and $u_t$ is the discounted dollar amount invested in the risky asset. The exploratory version replaces the point control $u_t$ by a density $\pi_t(\cdot)$ on $\mathbb R$, with relaxed dynamics determined by the mean and variance of the sampled action [1904.11392].

The associated entropy-regularized mean-variance criterion is written with a Lagrange multiplier $w$ as
$$
J[\pi;w]
=
\E\bigl[(X_T^\pi-w)^2\bigr]
+
\lambda\,\E\!\Bigl[\int_0^T\!\!\int_{\mathbb R}\pi_t(u)\ln\pi_t(u)\,du\,dt\Bigr]
-(w-z)^2,
$$
where $z$ is the target expected terminal wealth and $\lambda>0$ is the exploration weight. The first term is the squared-error around $w$, while the entropy term rewards higher-entropy policies because $\int \pi\ln\pi$ is the negative of differential Shannon entropy [1904.11392].

A multidimensional version appears in electricity virtual bidding. There, wealth evolves as
$$
dX_t=\mathbf q_t^\top\bigl(\mathbf b_t\,dt+\boldsymbol\sigma_t\,d\mathbf W_t\bigr),
$$
where the drift vector $\mathbf b_t=\mathbf b_t(\boldsymbol\theta_t)$ and volatility matrix $\boldsymbol\sigma_t=\boldsymbol\sigma_t(\boldsymbol\theta_t)$ are functions of observed meteorological variables $\boldsymbol\theta_t$. The virtual bidding problem is then transformed into a mean-variance portfolio management problem and approached by the exploratory mean-variance portfolio management framework [2303.02303].

Under Bayesian drift uncertainty, the same objective is embedded in partial information. The discounted wealth process satisfies
$$
dX_t=\sigma\,u_t\bigl(m_t\,dt+d\widehat W_t\bigr),
$$
with posterior mean and variance $(m_t,P_t)$ obtained from the Kalman-Bucy filter. The entropy-regularized cost is
$$
V(t,x,m,P)=\inf_{\{\pi_s\}_{s\in[t,T]}}
\E\Bigl[(X_T-w)^2+\tau\int_t^T H(\pi_s)\,ds\mid X_t=x,m_t=m,P_t=P\Bigr],
$$
and this is stated to be dual to
$$
J^{\lambda,\gamma}(\pi)=\E[X_T]-\frac{\lambda}{2}\Var(X_T)+\gamma\,\E\!\Bigl[\int_0^T H(\pi_t)\,dt\Bigr]
$$
with suitable matching of $\lambda\leftrightarrow w$ and $\gamma\leftrightarrow\tau$ [2602.16862].

## 2. HJB structure and the Gaussian optimal policy

The defining analytical feature of the entropy-regularized mean-variance objective is that dynamic programming produces an HJB equation whose inner optimization is over probability densities rather than point controls. In the single-asset case, the value function
$$
V(t,x;w)=\inf_\pi
\E\Bigl[(X_T^\pi-w)^2+\lambda\int_t^T\!\!\int \pi_s(u)\ln\pi_s(u)\,du\,ds\mid X_t^\pi=x\Bigr]-(w-z)^2
$$
satisfies
$$
V_t+\min_{\pi(\cdot)\in\mathcal P(\mathbb R)}
\Biggl\{
\int_{\mathbb R}
\Bigl[
\tfrac12\sigma^2u^2V_{xx}+\rho\sigma uV_x+\lambda\ln\pi(u)
\Bigr]\pi(u)\,du
\Biggr\}=0,
$$
with terminal condition $V(T,x;w)=(x-w)^2-(w-z)^2$ [1904.11392].

The stationarity condition in $\pi$ yields a Gibbs density,
$$
\pi^*(u)\propto
\exp\!\bigl\{
-\tfrac1\lambda[
\tfrac12\sigma^2u^2V_{xx}+\rho\sigma uV_x]
\bigr\},
$$
and, because the exponent is quadratic in $u$, the optimizer is Gaussian. For the known-drift single-asset problem, the explicit solution is
$$
\pi^*(u;t,x,w)
=
\mathcal N\!\Bigl(
-\tfrac{\rho}{\sigma}(x-w),\;
\tfrac{\lambda}{2\sigma^2}e^{\rho^2(T-t)}
\Bigr),
$$
so the optimal feedback policy is Gaussian with time-decaying variance [1904.11392].

The same structure persists in multidimensional and partial-information settings. In the exploratory mean-variance portfolio problem used for virtual bidding, the optimizer is
$$
\pi^*(\mathbf q\mid t,x)
=
\mathcal N\!\Bigl(
-(\boldsymbol\sigma_t\boldsymbol\sigma_t^\top)^{-1}\mathbf b_t\,\frac{V_x}{V_{xx}},
\;
(\boldsymbol\sigma_t\boldsymbol\sigma_t^\top)^{-1}\frac{\gamma}{V_{xx}}
\Bigr),
$$
so both the mean and covariance inherit the market-price parameters $\mathbf b_t(\boldsymbol\theta_t)$ and $\boldsymbol\sigma_t(\boldsymbol\theta_t)$ [2303.02303].

With Bayesian drift uncertainty, the HJB is written in $(t,x,m,P)$ and the inner minimization depends on
$$
G=mV_x+PV_{xm}.
$$
The optimizer is again Gaussian,
$$
\pi^*(u)=\mathcal N(\bar u^*,\varsigma^{*2}),
\qquad
\bar u^*=-\frac{mV_x+PV_{xm}}{\sigma V_{xx}},
\qquad
\varsigma^{*2}=\frac{\tau}{\sigma^2V_{xx}},
$$
with the mixed derivative $V_{xm}$ coupling learning into the mean [2602.16862].

## 3. Explicit value functions and the link to classical mean-variance control

A central result in the literature is that entropy regularization preserves the tractability of continuous-time mean-variance control. In the single-asset RL formulation, the value function admits the quadratic representation
$$
V(t,x;w)
=
e^{-\rho^2(T-t)}(x-w)^2
+
\tfrac{\lambda\rho^2}{4}(T^2-t^2)
-
\tfrac{\lambda}{2}\Bigl(\rho^2T-\ln\tfrac{\sigma^2}{\pi\lambda}\Bigr)(T-t)
-(w-z)^2,
$$
which gives $V_{xx}=2e^{-\rho^2(T-t)}$ and directly produces the Gaussian optimizer [1904.11392].

The zero-entropy limit connects the exploratory and classical problems. As $\lambda\to0$, the optimal distribution $\pi^*(\cdot;t,x)$ converges weakly to the Dirac mass at the classical optimal control
$$
u^*(t,x)=-(\rho/\sigma)(x-w),
$$
and the value function converges pointwise to
$$
V^{\rm cl}(t,x;w)=(x-w)^2e^{-\rho^2(T-t)}-(w-z)^2.
$$
The exploratory problem with $\lambda>0$ and the classical problem with $\lambda=0$ are stated to share the same Lagrange multiplier $w$, the same drift in the optimal wealth SDE, and solvability equivalence [1904.11392].

The virtual bidding formulation yields the same qualitative link. The mean of the optimal Gaussian policy is exactly the classical mean-variance optimal holding, proportional to $(\boldsymbol\sigma\boldsymbol\sigma^\top)^{-1}\mathbf b$, but shrunk by the risk-adjusted term $V_x/V_{xx}$. Its covariance is proportional to $(\boldsymbol\sigma\boldsymbol\sigma^\top)^{-1}$ and acts as an exploration bonus of order $\gamma$. As $\gamma\downarrow0$, the policy collapses to the deterministic classical policy; for $\gamma>0$, the policy remains Gaussian and explores around the mean [2303.02303].

Under Bayesian drift uncertainty, the value function remains quadratic in wealth:
$$
V(t,x,m)=e^{\alpha(t)m^2+\gamma(t)}(x-w)^2+\eta(t)m^2+\zeta(t),
$$
with explicit closed-form coefficients $\alpha(t)$ and $\gamma(t)$ along the deterministic path of $P_t$. The paper emphasizes that $A(t,m,P)$ is independent of $w$ and $\tau$, so the entropy term enters through the additive entropy-premium term and the policy variance rather than through the quadratic wealth coefficient itself [2602.16862].

## 4. Exploration, robustness, and information masking

The interpretation of the entropy term depends on the informational structure of the model. In the reinforcement-learning formulation with known drift, the entropy-regularized mean-variance criterion is explicitly presented as a device for achieving the best tradeoff between exploration and exploitation. The policy variance
$$
\Var[u\mid t]=\frac{\lambda}{2\sigma^2}e^{\rho^2(T-t)}
$$
decays exponentially from an initially high exploration level, while the mean equals the classical exploitative control [1904.11392].

Under Bayesian drift uncertainty, the same formal regularizer acquires a different meaning. The mean control is exactly the deterministic Bayesian Markowitz feedback and is independent of $\tau$, whereas the policy variance depends only on $\tau$ and $V_{xx}$. Because filtering $(m_t,P_t)$ is independent of the policy, the extra variance does not speed up learning; instead it provides belief-dependent robustness. The optimal variance is symmetric in $m$, minimized at $m=0$, and strictly increasing in $|m|$, so action randomization increases when the certainty-equivalent mean position is most aggressive [2602.16862].

A third interpretation appears in mean-variance Stackelberg competition with asymmetric information. There the leader maximizes
$$
\E[Z_1(T)]-\tfrac{\gamma_1}{2}\Var(Z_1(T))+\lambda_0\int_0^T H(\Pi_t)\,dt,
$$
and the optimal randomized trading policy is Gaussian with variance
$$
\Sigma_t=\frac{\lambda_0}{\gamma_1\sigma^2\chi^2}.
$$
Larger $\lambda_0$ produces a wider Gaussian and more masking of the true information from the follower, whereas larger $\gamma_1$ or larger $\sigma^2$ reduce randomization [2509.03669].

These formulations rule out a common simplification: entropy regularization is not uniformly an exploration mechanism. In the RL setting it is tied to exploration; in the Bayesian partial-information setting it is explicitly interpreted as robustness; and in the Stackelberg setting it is used to prevent information leakage [1904.11392; 2602.16862; 2509.03669].

## 5. Variants and application domains

The objective has been embedded in several distinct stochastic environments. The common elements are randomized controls, mean-variance terminal criteria, and an entropy or relative-entropy regularizer; the differences lie in the state dynamics, the source of uncertainty, and the operational role of randomization.

| Setting | Dynamics and uncertainty | Role of entropy term |
|---|---|---|
| Electricity virtual bidding | Brownian motion with drift; drift and volatility depend on meteorological variables | Exploration around mean-variance-optimal holding |
| Portfolio optimization with jumps | Lévy-driven return process and exploratory wealth SDE with compensated jump measure | Gaussian control under jump-diffusion dynamics |
| Bayesian drift uncertainty | Kalman-Bucy filtering for unknown Sharpe ratio | Belief-dependent robustness |
| Stackelberg asymmetric information | Leader randomizes actions observed by follower | Information masking |
| Robust optimal order execution | KL penalty relative to prior market-rate distribution | Distributional robustness |

In electricity virtual bidding, price differences between the day-ahead market and the real-time market are modeled as Brownian motion with drift, where the drift rate and volatility are functions of meteorological variables. The paper states that one estimates $\mathbf b_t(\cdot)$, for example by regression on temperature, wind, and humidity, and $\boldsymbol\sigma_t(\cdot)$, for example via a rolling-window covariance from historical LMP-differences; these quantities then enter both the drift term of $dX_t$ and the shape of the exploration bonus [2303.02303].

In the jump setting, the discounted return process is Lévy-driven, the exploratory wealth SDE is obtained as the weak-convergence limit of a natural discrete-time randomization scheme, and the continuous-time HJB again yields a Gaussian optimal distribution. The coefficient fields $M_\alpha$ and $S_\alpha$ are defined through a backward PIDE, and the optimal covariance is $\Sigma_t=\lambda S_\alpha(t,Y_t)^{-1}$, while the optimal wealth process admits a closed-form representation via the Doléans-Dade exponential [2312.13409].

Relative-entropy regularization in optimal order execution is closely related but not identical. There the inner minimization is over distributions $\pi_t\ll\pi_t^0$ for the market trading rate, and the objective adds a Kullback-Leibler term
$$
\frac1\beta\ln\frac{\pi_t(a)}{\pi_t^0(a)}.
$$
Under the LQG ansatz and a Gaussian prior, the posterior $\pi_t^*(a)$ is Gaussian, the HJI reduces to coupled Riccati and linear ODEs, and finite $\beta$ forces the posterior to stay close to the prior, guarding against misspecified market-rate models [2311.06476].

## 6. Algorithmic consequences and recurrent points of confusion

The entropy-regularized mean-variance objective is not only analytically tractable; it also supports policy-improvement and learning procedures. In the RL formulation, if a smooth feedback policy $\pi$ has value $V^\pi$ with $V^\pi_{xx}>0$, then the update
$$
\tilde\pi(u;t,x)\propto
\exp\!\bigl\{
-\tfrac1\lambda[
\tfrac12\sigma^2u^2V^\pi_{xx}+\rho\sigma uV^\pi_x]
\bigr\}
$$
satisfies
$$
V^{\tilde\pi}(t,x)\le V^\pi(t,x).
$$
Iterating this improve-evaluate procedure is stated to converge, in finitely many steps for the mean-variance LQ structure, to the global optimum $V$ and $\pi^*$ [1904.11392].

The implementable EMV algorithm parameterizes $V^\theta(t,x)$ by the quadratic/exponential form implied by the HJB solution and $\pi^\phi(u\mid t,x)$ as a Gaussian family, computes the continuous-time TD error
$$
\delta_t=\dot V^\theta(t,X_t)+\lambda\int\pi^\phi\ln\pi^\phi,
$$
updates $\theta$ and $\phi$ by stochastic gradient descent to minimize squared TD error, and updates the Lagrange multiplier through the Robbins-Monro rule
$$
w_{n+1}=w_n-\alpha\,(X_T-z).
$$
The paper reports that this algorithm outperforms both an adaptive control based method and a deep neural networks based algorithm in simulations [1904.11392].

Several misconceptions are resolved directly by the cited formulations. One is that entropy regularization necessarily changes the deterministic feedback law; in the Bayesian partial-information model, the mean control is identical to deterministic Bayesian Markowitz feedback and only the policy variance changes [2602.16862]. Another is that entropy-regularized mean-variance control must be diffusion-only; the jump formulation shows that Gaussian randomized controls persist in a Lévy environment, although the coefficient fields are determined by a backward PIDE rather than an ODE system [2312.13409]. A further misconception is that entropy regularization always operates through the agent’s own action distribution; the order-execution model uses relative entropy with respect to a prior on the market trading-rate distribution, yielding a robust stochastic differential game rather than a purely exploratory control problem [2311.06476].

Across these formulations, the entropy-regularized mean-variance objective retains the mean-variance target-tracking structure, admits HJB-based closed forms in LQ and related settings, and replaces deterministic controls by Gaussian policies whose variance quantifies exploration, robustness, or strategic masking according to the surrounding information model [1904.11392].

Source: https://www.emergentmind.com/topics/entropy-regularized-mean-variance-objective