Memoryless Stochastic Optimal Control

• Memoryless Stochastic Optimal Control is a framework utilizing Markov state-feedback laws that depend solely on the current state and time for optimal decision making.
• It leverages methods such as HJB equations, linearly solvable formulations, and martingale-based solvers to derive efficient, memoryless controllers.
• This approach is applied in quantum control, differential games, and generative modeling, emphasizing the role of full observability and risk-consistent objectives.

Searching arXiv for recent and foundational work on memoryless stochastic optimal control. Memoryless stochastic optimal control (SOC) concerns the synthesis of a feedback law that depends only on the current state and time, typically $u_t=\pi(x_t,t)$, for a stochastic dynamical system with running and terminal costs. In fully observed Markov settings, the Bellman optimality principle implies that the optimal policy is memoryless; this contrasts with history-dependent controls $u_t=\pi(x_{0:t},t)$, and with partially observed problems in which the control depends on a belief state $b_t$ over latent states, so memory is unavoidable (Patil, 24 Apr 2025). Across recent work, memoryless SOC appears in continuous-time HJB formulations, linearly solvable and path-integral control, direct optimization of parametric feedback laws, martingale-based neural solvers, probabilistic-inference methods, and several application domains including chance constraints, differential games, quantum control, and generative modeling (Patil, 24 Apr 2025).

1. Conceptual scope and policy classes

In the fully observed setting, “memoryless” is synonymous with Markov state feedback: the controller maps the current state and time to an action, with no explicit dependence on the past trajectory. This includes both continuous-time diffusion control and discrete-time Markov decision processes. Memorylessness does not imply stationarity: a policy may remain time-dependent, $u_t=\pi(x_t,t)$, while still being Markov. In contrast, open-loop controls $u(t)$ are a special case that depend on time but not on state, and history-dependent policies explicitly use past states or actions. The central structural assumption is therefore not simplicity of the control law, but the sufficiency of the current state as a statistic for optimal action selection (Patil, 24 Apr 2025).

This distinction becomes sharp in partially observable settings. For POMDPs, optimal memoryless policies are generally stochastic rather than deterministic because perceptual aliasing forces a policy conditioned only on the current observation to randomize across actions. For a finite stationary POMDP $(W,S,A,\alpha,\beta,R)$, the paper on the geometry of optimal stationary control shows that any POMDP has an optimal memoryless policy of limited stochasticity, with a face-dimension bound of $|U|(|A|-1)$, where $U$ is the set of ambiguous sensor states satisfying $|\mathrm{supp}\,\beta(s|\cdot)|>1$ (Montufar et al., 2015). This establishes that memoryless control may remain optimal in partially observed models only after accepting stochasticity, whereas exact optimality in the underlying partially observed control problem still requires belief-state memory.

Risk-averse and distributionally robust formulations add another layer to the notion of memorylessness. For finite-horizon SOC and MDPs with nested conditional risk mappings, dynamic programming remains valid and yields Markov policies when stagewise independence and rectangular ambiguity hold; by contrast, non-nested Value-at-Risk is time-inconsistent and can break the Bellman principle, potentially requiring history-dependent precommitment strategies (Shapiro et al., 22 May 2025). A recurrent theme is therefore that memoryless optimality is less a generic property of “stochastic control” than a consequence of specific structural assumptions: full observability, Markov dynamics, and a dynamically consistent objective.

2. Bellman equations and feedback characterization

A canonical continuous-time memoryless SOC model uses the controlled diffusion

$$\mathrm{d}x_t = f(x_t,t)\,\mathrm{d}t + B(x_t,t)\,u_t\,\mathrm{d}t + \Sigma(x_t,t)\,\mathrm{d}W_t,$$

with cost functional

$u_t=\pi(x_{0:t},t)$0

If $u_t=\pi(x_{0:t},t)$1 denotes the value function, then the stochastic HJB equation is

$u_t=\pi(x_{0:t},t)$2

and quadratic minimization yields the optimal memoryless feedback

$u_t=\pi(x_{0:t},t)$3

The feedback law is pointwise in $u_t=\pi(x_{0:t},t)$4, even though $u_t=\pi(x_{0:t},t)$5 itself summarizes future uncertainty and cost accumulation (Patil, 24 Apr 2025).

A structurally broader formulation allows control of both drift and volatility. In that setting, with admissible memoryless feedbacks $u_t=\pi(x_{0:t},t)$6 and controlled state process

$u_t=\pi(x_{0:t},t)$7

the HJB equation takes the form

$u_t=\pi(x_{0:t},t)$8

with

$u_t=\pi(x_{0:t},t)$9

SOC-MartNet replaces explicit pointwise computation of $b_t$0 by minimizing the Hamiltonian process in expectation and enforcing the HJB through a martingale condition on a cost process. Concretely, it trains a value network $b_t$1 and a control network $b_t$2 so that

$b_t$3

is approximately a martingale, while $b_t$4 is minimized along simulated trajectories (Cai et al., 2024). This preserves the Markov interpretation—$b_t$5 remains a memoryless feedback law—while avoiding closed-form Hamiltonian minimization.

3. Linearly solvable SOC and path-integral control

A major subclass of memoryless SOC problems is linearly solvable control. In continuous time, this occurs when the control and noise satisfy the matching condition

$b_t$6

Under the desirability transform

$b_t$7

the nonlinear HJB reduces to the linear PDE

$b_t$8

and the optimal feedback becomes

$b_t$9

By the Feynman–Kac lemma,

$u_t=\pi(x_t,t)$0

where the expectation is taken over passive trajectories of the uncontrolled diffusion. The policy is therefore computed pointwise from weighted future rollouts initialized at $u_t=\pi(x_t,t)$1, which explains why the resulting controller is empirically and analytically Markov (Patil, 24 Apr 2025).

The same structure persists in discrete time through linearly solvable MDPs and KL-control. With passive kernel $u_t=\pi(x_t,t)$2, desirability satisfies

$u_t=\pi(x_t,t)$3

and the optimal controlled transition

$u_t=\pi(x_t,t)$4

is memoryless. In the linear-Gaussian specialization, path-integral KL-control recovers discrete-time LQR exactly: the Gaussian optimal policy has mean $u_t=\pi(x_t,t)$5, with $u_t=\pi(x_t,t)$6 equal to the classical LQR feedback matrix, and Monte Carlo estimation converges to this law as the number of samples increases. The dissertation further gives a finite-sample error bound showing that the required number of rollouts depends only logarithmically on the control dimension $u_t=\pi(x_t,t)$7, rather than exponentially as in exact dynamic programming (Patil, 24 Apr 2025).

Long-horizon memoryless control admits an additional spectral characterization in a special linearly solvable regime. When the uncontrolled drift is a gradient field, $u_t=\pi(x_t,t)$8, the linearized operator becomes unitarily equivalent to a Schrödinger operator with purely discrete spectrum. The semigroup solution admits an eigenfunction expansion, and the optimal long-horizon control converges to the stationary Markov policy

$u_t=\pi(x_t,t)$9

where $u(t)$0 is the strictly positive ground-state eigenfunction. In this setting, memoryless control is not merely state-feedback at each time step; it becomes asymptotically time-invariant, with runtime and memory reduced from $u(t)$1 to $u(t)$2 (Claeys et al., 24 Mar 2026).

4. Direct optimization of memoryless policies

An alternative tradition does not solve HJB or desirability equations directly, but optimizes a parametric Markov policy $u(t)$3. A variational derivation in Stratonovich form yields adjoint sensitivities for stochastic differential equations and a pathwise gradient for

$u(t)$4

with $u(t)$5. The resulting backward adjoint process $u(t)$6 leads to a gradient in which $u(t)$7 multiplies drift and diffusion sensitivity terms, thereby providing a continuous-time policy-gradient method for memoryless feedback laws without solving an HJB PDE (Massaroli et al., 2021).

A more recent on-policy framework uses Girsanov’s theorem to avoid differentiating through SDE trajectories. For dynamics

$u(t)$8

and cost $u(t)$9, the exact on-policy gradient is

$(W,S,A,\alpha,\beta,R)$0

Because this expression depends only on evaluations of $(W,S,A,\alpha,\beta,R)$1, $(W,S,A,\alpha,\beta,R)$2, and stochastic integrals along sampled paths, it removes the need for adjoint solves or backpropagation through the entire SDE computational graph (Hua et al., 2024).

The literature has also clarified that many modern SOC training losses differ less in optimization landscape than in variance. A recent taxonomy groups discrete adjoint, continuous adjoint, REINFORCE, REINFORCE with future rewards, and Cost-SOCM into one class with identical expected gradients; Adjoint Matching and Work-SOCM form a second class; SOCM, SOCM-Adjoint, and Cross-Entropy form a third. The principal practical distinction is therefore gradient variance rather than asymptotic direction, which explains the empirical advantage of lean-adjoint methods over importance-weighted or likelihood-ratio losses in high-dimensional continuous settings (Domingo-Enrich, 2024).

This perspective underlies Adjoint Matching for reward fine-tuning of flow and diffusion models. There, the generative dynamics are cast as an SOC problem with a crucial memoryless requirement on the fine-tuning noise schedule: $(W,S,A,\alpha,\beta,R)$3. Under this schedule, the initial Gaussian noise $(W,S,A,\alpha,\beta,R)$4 becomes independent of the terminal sample $(W,S,A,\alpha,\beta,R)$5, eliminating an initial-value bias and ensuring that the terminal marginal is correctly tilted by the reward. The final training objective regresses the control against a lean adjoint target,

$(W,S,A,\alpha,\beta,R)$6

thereby turning memoryless SOC into a least-squares problem over Markov controls (Domingo-Enrich et al., 2024).

5. Constraints, games, and robust variants

Memoryless SOC extends well beyond unconstrained, risk-neutral control. In chance-constrained continuous-time SOC, the dissertation reformulates the constraint through a dual problem,

$(W,S,A,\alpha,\beta,R)$7

with

$(W,S,A,\alpha,\beta,R)$8

Under the matching condition, the inner problem remains linearly solvable, and dual ascent uses Monte Carlo importance sampling to estimate failure probability. The resulting controller $(W,S,A,\alpha,\beta,R)$9 is again a memoryless state-feedback law (Patil, 24 Apr 2025).

Two-player stochastic differential games furnish a related extension. Under the generalized matching condition

$|U|(|A|-1)$0

the min–max HJI equation linearizes, and saddle-point policies admit path-integral representations. In a different but conceptually allied finite-state setting, concurrent stochastic reachability games admit a memoryless optimal strategy for Max on every state from which Max has an optimal strategy, while the same strategy can be chosen $|U|(|A|-1)$1-optimal on all remaining states; the result holds even when Max’s action sets are countably infinite, provided the state space is finite and Min’s action sets are finite (Kiefer et al., 2024).

Risk-averse and distributionally robust SOC requires additional care because dynamic consistency becomes decisive. For conditional nested risk functionals, finite-horizon SOC and MDPs satisfy Bellman recursions of the form

$|U|(|A|-1)$2

and the same Markov structure survives robustification under rectangular ambiguity sets. Non-randomized optimal policies exist under saddle-point conditions, convex ambiguity sets, and concavity of the risk map in $|U|(|A|-1)$3. By contrast, non-nested VaR is time-inconsistent and can invalidate dynamic programming, so memoryless Markov optimality may fail (Shapiro et al., 22 May 2025).

A common misconception is that memoryless control always means deterministic state feedback. In partially observable or adversarial settings this is false. The POMDP geometry results show that stochastic memoryless policies are often necessary, but that the degree of stochasticity can be sharply bounded by the observation ambiguity structure (Montufar et al., 2015). The right contrast is therefore not deterministic versus stochastic, but Markov versus history-dependent.

6. Computation, applications, and limitations

A practical attraction of memoryless SOC is that many of its estimators are embarrassingly parallel. GPU-parallelized path-integral control simulates $|U|(|A|-1)$4 trajectories concurrently, accumulates costs $|U|(|A|-1)$5, forms weights $|U|(|A|-1)$6, and computes

$|U|(|A|-1)$7

The reported implementation uses $|U|(|A|-1)$8–$|U|(|A|-1)$9 trajectories with subsecond latencies and achieves multiple orders-of-magnitude speedups over CPU for large sample counts (Patil, 24 Apr 2025). This computational profile explains why memoryless path-integral controllers are often deployed in receding-horizon form.

Open quantum systems furnish a distinctive application domain. There, stochastic Schrödinger equations generate Markov quantum trajectories, and path-integral control becomes feasible after an anti-Hermitian transformation of the unraveling. The resulting Quantum Diffusion Control algorithm updates either open-loop pulses or feedback parametrizations by adaptive importance sampling rather than gradient backpropagation. In the single-qubit state-preparation example with $U$0, $U$1, $U$2, $U$3, $U$4, and $U$5 trajectories per importance-sampling step, the reported final average fidelity is approximately $U$6; reducing $U$7 from $U$8 to $U$9 raises it to approximately $|\mathrm{supp}\,\beta(s|\cdot)|>1$0. For a $|\mathrm{supp}\,\beta(s|\cdot)|>1$1-qubit GHZ preparation problem, the reported final average fidelity is approximately $|\mathrm{supp}\,\beta(s|\cdot)|>1$2 (Villanueva et al., 2024).

In generative modeling, memoryless SOC has been used to reinterpret diffusion bridges. UniDB formulates a linear-quadratic SOC with terminal penalty coefficient $|\mathrm{supp}\,\beta(s|\cdot)|>1$3,

$|\mathrm{supp}\,\beta(s|\cdot)|>1$4

and derives a closed-form optimal controller. Existing diffusion bridges based on Doob’s $|\mathrm{supp}\,\beta(s|\cdot)|>1$5-transform emerge as the limiting case $|\mathrm{supp}\,\beta(s|\cdot)|>1$6, whereas finite $|\mathrm{supp}\,\beta(s|\cdot)|>1$7 balances control effort against terminal fidelity. This suggests a principled explanation for oversmoothing in pure $|\mathrm{supp}\,\beta(s|\cdot)|>1$8-transform bridges and a mechanism for recovering finer detail by retaining a finite control cost (Zhu et al., 9 Feb 2025).

Probabilistic-inference approaches provide yet another computational route. SOC-EM uses an EM procedure over a linear time-varying stochastic model with latent states and Gaussian memoryless policies $|\mathrm{supp}\,\beta(s|\cdot)|>1$9, coupling Kalman smoothing in the E-step with policy-parameter updates in the M-step; the analysis shows monotonic decrease of expected cumulative cost and contraction of the action covariance toward exploitation (Mallick et al., 2020). A later constrained extension generalizes this picture to nonlinear Gaussian systems, barrier-encoded inequality constraints, and structured state-feedback controllers $$\mathrm{d}x_t = f(x_t,t)\,\mathrm{d}t + B(x_t,t)\,u_t\,\mathrm{d}t + \Sigma(x_t,t)\,\mathrm{d}W_t,$$0, where the E-step smooths the joint state-control trajectory and the M-step updates only the nonzero parameter subsets imposed by a structural sparsity mask (Syed et al., 3 Sep 2025).

The principal limitations of memoryless SOC are equally structural. Full-state observability is required for exact Markov feedback; POMDPs need belief states or accept suboptimal memoryless approximations. Linear solvability in path-integral control depends on control–noise matching, and when the condition fails the same algorithms may still be used heuristically but lose optimality guarantees. Martingale neural solvers require regularity such as uniform ellipticity and access to second derivatives of the value network, which can become computationally demanding in very high dimension. Risk-averse Markov optimality depends on nested conditional risk and rectangular ambiguity, while non-rectangular robust formulations and non-nested VaR can destroy dynamic programming (Patil, 24 Apr 2025).

Memoryless SOC is therefore best understood as a structurally privileged regime of stochastic control rather than as a universal simplification. When full observability, Markov dynamics, and dynamically consistent objectives are present, memoryless policies can be characterized by HJB equations, desirability functions, adjoint or martingale identities, or inference updates, and they often admit powerful sampling-based or neural implementations. When those assumptions fail, the same term continues to denote a policy class, but no longer an exact description of the optimal controller.