---
title: Reinforced Stochastic DMD
url: https://www.emergentmind.com/topics/reinforced-stochastic-dynamic-mode-decomposition
type: topic
---

# Reinforced Stochastic DMD

Searching arXiv for the cited papers to ground the response.
Reinforced Stochastic Dynamic Mode Decomposition (RSDMD) is a data-driven framework for spectral analysis of stochastic dynamical systems that integrates Reinforcement Learning (RL) with Stochastic Dynamic Mode Decomposition (SDMD) to automatically guide the data collection process. In this formulation, the optimal sampling strategy is posed as an RL problem in which an agent learns a policy to select trajectory initial conditions, and the reward is based on *spectral consistency* balanced with an exploration bonus. The resulting workflow targets efficient estimation of Koopman spectral properties in settings where the quality and location of sampled trajectories strongly affect the accuracy of EDMD- and SDMD-type estimators [2509.04265].

## 1. Operator-theoretic setting and motivation

RSDMD is situated in the Koopman-operator program for stochastic systems. In that setting, the central object is the stochastic Koopman semigroup, written in SDMD as
\[
(\mathcal{K}^t f)(x) := \mathbb{E}[f(\mathbf{X}_t)|\mathbf{X}_0=x],
\]
with infinitesimal generator
\[
\mathcal{A} f = \lim_{t\to 0} \frac{\mathcal{K}^t f - f}{t}
= \sum_{i=1}^d b_i \frac{\partial f}{\partial x_i}
+ \frac{1}{2}\sum_{i,j=1}^d (\sigma \sigma^T)_{ij} \frac{\partial^2 f}{\partial x_i \partial x_j}.
\]
If \(\mathcal{A}\phi = \lambda \phi\), then \(\mathcal{K}^t \phi = e^{t\lambda}\phi\), so eigenvalues and eigenfunctions of the generator and semigroup encode the spectral organization of stochastic dynamics [2501.13301].

The motivation for RSDMD is not that stochastic Koopman estimators are absent, but that their accuracy is heavily dependent on the quality and location of sampled trajectory data. The RSDMD paper states this dependency explicitly for EDMD and related variants, and then reorients the problem from passive estimation to active sampling. In that sense, RSDMD augments SDMD with an adaptive data-acquisition layer rather than replacing the Koopman-semigroup estimator itself [2509.04265].

This design is tightly linked to the SDMD formulation. SDMD directly approximates the Koopman semigroup instead of the generator, explicitly incorporates the sampling time \(\Delta t\), and avoids matrix exponential computations by using
\[
\widehat{K}_{N,\Delta t,m} = I + \Delta t\cdot \widehat{G}^{-1}\widehat{H},
\]
where \(\widehat{G}\) and \(\widehat{H}\) are empirical Gram matrices. RSDMD inherits this estimator and then optimizes where the data used in these matrices come from [2501.13301].

## 2. Reinforcement-learning formulation and SDMD pipeline

In RSDMD, the data acquisition process is formulated as an RL problem in which the environment is the stochastic system under study and the action corresponds to selecting a region of state space from which a new trajectory is initialized. The action space is discrete; for 2D systems the paper uses grid cells or regions, with the example \(32\times 32\) discretization giving \(1024\) actions. The state representation depends on the RL algorithm: the bandit setting is stateless, whereas DQN and PPO use a sliding window of the last \(\ell\) sampled initial states. The sampling policy is denoted \(\pi_\theta(a|s)\) [2509.04265].

The SDMD stage then operates on trajectory data generated from the chosen initial conditions. Given a dictionary of basis functions \(\{\psi_j\}\), the paper defines
\[
[\Psi_X]_{ij} := \psi_j(x_i), \qquad
[\Psi'_X]_{ij} := \mathcal{A}\psi_j(x_i),
\]
followed by empirical matrices
\[
\widehat{G} = \frac{1}{m}\Psi_X^* \Psi_X, \qquad
\widehat{H} = \frac{1}{m}\Psi_X^* \Psi'_X.
\]
The finite-dimensional Koopman approximation is then
\[
\widehat{K}_{N,\Delta t, m} = I + \Delta t \widehat{G}^{-1} \widehat{H},
\]
and Koopman eigenpairs are extracted from \(\widehat{K}\) [2509.04265].

The RL algorithms integrated into this pipeline are a Multi-Armed Bandit, DQN, and PPO. The bandit algorithm maintains action-value estimates with the incremental update
\[
Q(a_t) \leftarrow Q(a_t) + \frac{1}{N(a_t)} (R_t - Q(a_t)).
\]
DQN learns \(Q(S_t,a_t;\theta)\) using temporal-difference updates with a target network and experience replay:
\[
\text{TD Error}
=
R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a'; \theta^-)
- Q(S_t, a_t; \theta).
\]
PPO directly optimizes policy and value networks with the clipped objective
\[
\mathcal{L}^{\text{CLIP}}(\theta)
=
\mathbb{E}_t \left[
\min\left(
r_t(\theta)\widehat{A}_t,\,
\text{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\widehat{A}_t
\right)
\right],
\]
where
\[
r_t(\theta)=\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)},
\qquad
\widehat{A}_t
=
\sum_{l=0}^{T-t-1} (\gamma\lambda)^l \delta_{t+l},
\qquad
\delta_t=R_t+\gamma V(s_{t+1})-V(s_t).
\]
These algorithmic choices make the sampling policy itself a learned object tied to Koopman estimation quality [2509.04265].

## 3. Spectral-consistency reward and exploration

The defining innovation of RSDMD is the reward design. The agent is guided by a reward based on *spectral consistency*, described as a measure of how well the estimated Koopman eigenpairs describe the system’s evolution, balanced with an exploration bonus to ensure comprehensive coverage of the state space [2509.04265].

The exploitation component is the spectral consistency loss
\[
\mathcal{L}_{\text{Spectral Consistency}}
:=
\sum_{i=1}^N
\Big\|
\phi_i(Y) - \mu_i \phi_i(X)
\Big\|_\rho^2,
\]
where \((X,Y)\) is a trajectory batch and \((\mu_i,\phi_i)\) are the estimated Koopman eigenpairs. This term measures how well the learned spectral objects propagate observables forward under the estimated semigroup [2509.04265].

To prevent the policy from collapsing onto a narrow set of states, the reward includes an exploration bonus
\[
\alpha_{\exp}\cdot \frac{1}{\eta(x_{\text{new}})+\varepsilon},
\]
where \(\eta(\cdot)\) is a kernel density estimate over starting points and \(\alpha_{\exp}\) is a tunable constant. The total reward is
\[
R_t
=
R_0
-
\mathcal{L}_{\text{Spectral Consistency}}
+
\alpha_{\exp}\cdot \frac{1}{\eta(x_{\text{new}})+\varepsilon}.
\]
Lower spectral consistency loss therefore yields higher reward, while the density-based term encourages under-explored regions to remain accessible [2509.04265].

A common misunderstanding is to interpret the reward as a generic state-exploration heuristic. In RSDMD it is more specific: the reward is shaped by the quality of the Koopman spectral approximation. This means that the learned sampling policy is targeted toward regions that improve eigenvalue and eigenfunction estimation rather than toward generic control or reachability objectives. A plausible implication is that the framework is especially well matched to metastable systems, fast-slow systems, and settings where rare but spectrally decisive transitions dominate the operator structure.

## 4. Theoretical guarantees and accuracy limits

The theoretical contribution of RSDMD is to link RL performance bounds directly to SDMD estimation error. The paper assumes a uniformly bounded error between the true operator and the SDMD estimate,
\[
\|\mathcal{K} - \widehat{\mathcal{K}}\| \leq \varepsilon_{\text{sdmd}},
\]
and then derives policy-performance guarantees whose constants depend on \(\varepsilon_{\text{sdmd}}\) [2509.04265].

For the bandit version, if \(R_a\) is the true mean reward for arm \(a\), \(\Delta_a = R^* - R_a\) is the suboptimality gap, and
\[
\varepsilon_{\text{sdmd}}
<
\frac{1}{2}\min_{a\neq a^*}\Delta_a,
\]
then under a decaying greedy-exploration policy the cumulative regret is sublinear:
\[
\lim_{T\to\infty} \frac{\mathbb{E}[\mathcal{R}_T]}{T} = 0.
\]
If the SDMD error exceeds this threshold, linear regret is incurred [2509.04265].

For DQN-SDMD, assuming the action-value function has a linear representation in basis functions, the asymptotic suboptimality satisfies
\[
\limsup_{k\to\infty} [J(\pi^*) - J(\pi_k)]
\leq
\frac{2 \gamma Q_{\max}\varepsilon_{\text{sdmd}}}{(1-\gamma)^2}.
\]
For PPO-SDMD, with linearly representable value and reward functions, the asymptotic performance gap is
\[
\limsup_{k\to\infty} [J(\pi^*) - J(\pi_k)]
\leq
\mathcal{O}\left(\frac{\varepsilon_{\text{sdmd}}}{(1-\gamma)^3}\right).
\]
These results formalize a central limitation: RL cannot outperform the quality of the underlying SDMD approximation. The paper states this point explicitly by arguing that the agent’s ability to find an optimal sampling policy is fundamentally limited only by the statistical estimation error of the SDMD method [2509.04265].

This observation also clarifies the role of “reinforcement” in RSDMD. The framework reinforces sampling, not spectral estimation in isolation. If the dictionary is inadequate or the operator approximation is poor, better sampling alone does not remove the estimator’s intrinsic bias or approximation error. Conversely, when the SDMD approximation is sufficiently accurate, policy learning can drive the collected data toward dynamically informative regions.

## 5. Canonical systems and learned sampling behavior

The empirical demonstrations in RSDMD use the double-well potential, the stochastic Duffing oscillator, and the stochastic FitzHugh–Nagumo model, with Bandit, DQN, and PPO as the RL components [2509.04265].

For the double-well potential, the dynamics are
\[
\mathrm{d}X_t = -\nabla V(X_t)\,\mathrm{d}t + \sigma\,\mathrm{d}W_t,
\]
with
\[
V(x,y) = (x^2-1)^2 + y^2,
\qquad
\sigma = (1.09,1.09),
\]
and the action space is discretized on a \(32\times 32\) grid. The reported reward map after RL training matches the potential minima, and the agent autonomously discovers the wells \((\pm 1,0)\). The leading Koopman eigenfunction is constant, while the second eigenfunction after \(4000\) steps spatially partitions the phase space according to the two wells, reflecting metastability [2509.04265].

For the stochastic Duffing oscillator,
\[
\begin{cases}
\mathrm{d}x(t) = v(t)\mathrm{d}t,\\
\mathrm{d}v(t) =
\left[-\delta v(t)-\alpha x(t)-\beta x(t)^3\right]\mathrm{d}t
+\sigma\mathrm{d}W(t),
\end{cases}
\]
with \(\delta=0.5\), \(\alpha=-1\), \(\beta=1\), and \(\sigma=0.15\), the first eigenfunction is again constant and the second eigenfunction partitions phase space into basins of attraction of each well, highlighting metastable noise-induced switching between attractors [2509.04265].

For the stochastic FitzHugh–Nagumo model,
\[
\begin{cases}
\mathrm{d}x =
\left(x-\frac{1}{3}x^3-y\right)\mathrm{d}t
+\sigma_1\,\mathrm{d}W_1(t),\\
\mathrm{d}y =
\epsilon[x+a_1-a_2y]\,\mathrm{d}t
+\sigma_2\,\mathrm{d}W_2(t),
\end{cases}
\]
with \(\epsilon=0.01\), \(a_1=0.5\), \(a_2=0.1\), \(\sigma_1=10^{-3}\), and \(\sigma_2=10^{-5}\), the phase-space exploration reveals fast-slow dynamics. The leading non-trivial Koopman eigenfunction aligns with slow manifold regions and divides space along the limit cycle, corresponding to slow drift and fast jumps. The RL agent correspondingly increases sampling of dynamically critical regions such as the slow manifold and nullcline intersections [2509.04265].

Across these examples, the stated mechanism is that the RL agent discovers by itself, without any prior knowledge, the dynamical importance of minima, slow manifolds, and basins, and avoids wasted sampling in uninformative regions. This suggests that RSDMD is most naturally interpreted as an adaptive experimental-design framework for operator learning rather than merely as a post-processing enhancement.

## 6. Relation to adjacent stochastic DMD variants

RSDMD belongs to a broader family of stochastic and noise-robust DMD methods, but its intervention point is distinct: it changes how data are acquired, not only how they are processed after acquisition.

| Method | Primary mechanism | Main issue addressed |
|---|---|---|
| Subspace DMD | Orthogonal projection of future snapshots onto the space of past snapshots | Observation noise in random dynamical systems |
| Robust DMD | Cross-correlation with a dual observable and optional SVD enhancement | Bias in noisy observables and time-delayed embeddings |
| SDMD | Direct approximation of the Koopman semigroup with explicit \(\Delta t\) | Numerical stability and efficient stochastic Koopman estimation |
| RSDMD | RL-guided selection of trajectory initial conditions | Sampling quality and location for spectral estimation |

Subspace DMD computes the orthogonal projection
\[
\bm{O} = \bm{Y}_f \mathbb{P}_{\bm{Y}_p^H}
\]
and yields an estimator whose spectrum converges to that of the stochastic Koopman operator under standard assumptions. Its purpose is to reduce the bias induced by observation noise in random dynamical systems, not to optimize the sampling locations [1705.04908].

Robust DMD addresses a related but different pathology: bias created by noisy observables and time-delayed embeddings. It constructs cross-covariance matrices using a second observable \(\tilde{\mathbf{g}}\) designed to be independent of \(\tilde{\mathbf{f}}\), and then estimates the operator through
\[
C = \tilde{G}_1 \tilde{G}_0^\dagger.
\]
This allows unbiased approximation of the stochastic Koopman operator even for Hankel-type observables built from a single trajectory [2011.00078].

SDMD provides the finite-dimensional stochastic Koopman-semigroup estimator that RSDMD uses inside its RL loop. Its defining features are the explicit incorporation of sampling time, direct semigroup approximation rather than generator exponentiation, and support for dictionary learning via neural networks [2501.13301].

Other DMD variants extend stochastic modeling in yet different directions. Stochastically forced ensemble DMD models the observed dynamics as a forced linear system in time-delay coordinates and learns a stochastic Gaussian process regression process to actuate this system; its target is forecasting for near-periodic systems rather than adaptive spectral sampling [2010.04248]. Optimized DMD with automatic differentiation and stochastic gradient descent jointly computes eigenvalues, modes, and amplitudes, and allows regularization or physical constraints to be embedded into the operator definition; here “reinforcement” refers to optimization-based robustness rather than RL-guided data collection [2312.12928]. Stochastic NODE-DMD introduces a probabilistic extension of DMD with continuous-time nonlinear latent dynamics,
\[
d\boldsymbol{\phi}_t =
\left(\Lambda \boldsymbol{\phi}_t + f_\theta(\boldsymbol{\phi}_t,t)\right)dt
+\tau\,d\mathbf{B}_t,
\]
together with neural implicit spatial bases for sparse-to-field reconstruction and predictive uncertainty; its emphasis is spatiotemporal reconstruction, uncertainty quantification, and latent-dynamics variability rather than policy-driven sampling [2511.20612].

A recurring misconception is therefore to treat all stochastic DMD extensions as interchangeable. They are not. Some methods target noise bias, some target semigroup discretization, some target uncertainty propagation, and RSDMD targets the trajectory-initialization policy that feeds the estimator. The unifying theme is Koopman spectral analysis under stochasticity, but the algorithmic locus of intervention differs substantially across methods.

Source: https://www.emergentmind.com/topics/reinforced-stochastic-dynamic-mode-decomposition