Dyna: Reinforcement Learning Architecture

• Dyna is an integrated reinforcement learning architecture that uses real experience to learn models and generate synthetic data for planning.
• It employs search-control and rollout construction to strategically simulate transitions, aiding value propagation and exploration.
• Recent variants extend Dyna with function approximation and structured models, addressing high-dimensional tasks and physics-informed control.

Searching arXiv for relevant Dyna papers and related modern variants. arxiv_search(query="Dyna reinforcement learning architecture model-based RL search-control hill climbing planning linear function approximation", max_results=10) Dyna is an integrated architecture for reinforcement learning in which an agent learns directly from real experience, learns an approximate model of environment dynamics and reward from the same data, and then uses that model to generate imagined transitions that are processed as additional experience. In Sutton’s formulation, the core loop is to act in the real environment, store the transition, update the learned model, simulate additional transitions, and update the value function or policy from both real and synthetic data; subsequent work has treated search-control, rollout construction, model class, and domain structure as the principal design dimensions of Dyna-style methods (Das et al., 9 Mar 2026).

1. Classical architecture and objective

In its standard formulation, Dyna is defined on a Markov decision process $(\mathcal S,\mathcal A,p,r,\gamma,\rho_0)$ with the usual discounted-return objective. The characteristic feature is not the objective itself but the coupling of three operations inside a single online loop: direct RL from tuples $(s_t,a_t,r_t,s_{t+1})$, model learning from those same tuples, and planning by replaying or simulating experience from the learned model (Das et al., 9 Mar 2026).

A canonical Dyna cycle is: choose an action from the current policy, observe reward and next state, store the transition, update the learned transition and reward model, generate synthetic transitions from that model, and then update value and policy networks from the union of real and simulated data. Later Dyna instantiations keep this pattern while changing the function approximator, the planner, or the distribution of model queries. In this sense, Dyna is best understood as an architectural template for interleaving acting, learning, and planning, rather than as one particular update equation or one particular model class (Pan et al., 2019).

The term search-control denotes the mechanism that chooses the states or state-action pairs from which the model is queried during planning. This is one of the defining internal degrees of freedom of Dyna, because the usefulness of synthetic experience depends not only on model accuracy but also on where imagined experience begins and how it is distributed through state space (Pan et al., 2019).

2. Search-control, rollout construction, and planning shape

A major strand of Dyna research asks how search-control should be organized. In HC-Dyna, search-control states are generated by hill climbing on the current value estimate $V_\theta(s)=\max_a Q_\theta(s,a)$, rather than by uniformly replaying previously visited states. The practical update uses a noisy projected natural-gradient step in state space, so that planning states are pushed from lower-value regions toward higher-value regions before model rollouts are sampled from them (Pan et al., 2019).

A related refinement treats Dyna planning as a sampling problem over the geometry of the value function. “Frequency-based Search-control in Dyna” argues that high-frequency regions of the value function are harder to approximate and should receive more simulated samples. The local frequency proxy proposed there is

$g(s)=\|\nabla_s V(s)\|_2^2+\|H_V(s)\|_F^2,$

so search-control can be driven by gradient and Hessian information rather than only by value magnitude or TD error. The paper reports that, in MazeGridWorld, about $25.3\%$ of search-control points for Dyna-Frequency fall within a small radius around the holes, versus $11.7\%$ for Dyna-Value, which is presented as evidence that frequency-based search-control concentrates planning near bottlenecks (Pan et al., 2020).

A separate but closely related issue is planning shape: given a fixed model-query budget, should Dyna use many short rollouts or fewer long rollouts? Experiments in ALE showed that one-step Dyna offered surprisingly little benefit over simply performing more updates with existing experience, even when the model was perfect. Reallocating the same budget into fewer, longer rollouts had a “profound impact” on performance for both perfect and learned models, suggesting that Dyna benefits when the model is used to generate unfamiliar experience rather than merely near-duplicates of replay-buffer transitions (Holland et al., 2018).

This literature corrects a common oversimplification. Dyna planning is not exhausted by “add one-step model rollouts to replay.” Search-control and rollout shape determine whether model use is genuinely exploratory, value-propagating, and distribution-expanding, or whether it merely duplicates existing experience with extra computation.

3. Function approximation and learned models

The extension of Dyna beyond tabular settings required an explicit treatment of function approximation. “Dyna-Style Planning with Linear Function Approximation and Prioritized Sweeping” develops linear Dyna-style planning with a linear transition model $\phi' \approx F\phi$ and reward model $r \approx b^\top \phi$, where $v_\theta(\phi)=\theta^\top\phi$. For policy evaluation, the paper proves convergence of linear Dyna-style planning to a unique fixed point

$\theta^*=(I-\gamma F^\top)^{-1}b,$

independent of the generating distribution under the stated conditions, and shows that this limit coincides with the LSTD solution when the model is fit by least squares. An immediate consequence is that prioritized sweeping can be extended from preceding states to preceding features (Sutton et al., 2012).

High-dimensional observation spaces create a different problem: Dyna requires a model that can generate useful imagined experience from images or other large observations. “Dyna Planning using a Feature Based Generative Model” addresses this by learning a deep belief network that maps observations to a compact feature representation, models temporal evolution in that latent space, and decodes sampled futures back to observations. In the authors’ characterization, linear expectation models predict only the expected next feature vector and are tied to linear value approximation, whereas the DBN approach is a full generative model. The reported results show that the learned generative model “significantly outperforms” linear expectation models and support what the paper describes as the first successful use of a learned dynamics model for planning in the ALE (Faulkner et al., 2018).

Taken together, these results suggest that Dyna with function approximation has two distinct technical burdens. One is analytical—stability, fixed points, and feature-space prioritization. The other is generative—whether the model can synthesize trajectories that are both plausible and useful for value learning in large observation spaces.

4. Structured and physics-informed Dyna

A prominent contemporary direction constrains the Dyna model with domain structure rather than treating it as an unconstrained black box. “Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control” describes a Dyna-style model-based RL framework in which governing equations and physical constraints inform both model learning and policy search, with model rollouts used to update the policy and value function in the standard Dyna pattern. The reported environments include control problems governed by canonical ordinary and partial differential equations, including viscous Burgers’ equations and Kuramoto–Sivashinsky equations with chaotic behaviors (Liu et al., 2021).

“Integrating Lagrangian Neural Networks into the Dyna Framework for Reinforcement Learning” makes this structural prior explicit. Its Dyna-style algorithm uses a Lagrangian neural network $(s_t,a_t,r_t,s_{t+1})$0 as the transition model, derives accelerations through the Euler–Lagrange structure, and advances the state with an RK-2 integrator. The paper compares two LNN training methods—Adam and an extended Kalman filter state-estimation method—and reports that the EKF-based method converges faster. On an inverted pendulum, the PIMBRL + LNN (EKF) variant reaches average return around $(s_t,a_t,r_t,s_{t+1})$1 at about $(s_t,a_t,r_t,s_{t+1})$2 timesteps, versus about $(s_t,a_t,r_t,s_{t+1})$3 for PIMBRL + LNN (Adam), about $(s_t,a_t,r_t,s_{t+1})$4 for PIMBRL + constrained DNN, and about $(s_t,a_t,r_t,s_{t+1})$5 for the model-free baseline (Das et al., 9 Mar 2026).

A structurally related variant for nonlinear control combines SINDy and TD3 in a Dyna-style loop. There, SINDy identifies a sparse dynamics model from real trajectories, synthetic rollouts from that model are injected into the TD3 replay buffer, and the resulting controller is evaluated on a bi-rotor system. The reported comparison is about $(s_t,a_t,r_t,s_{t+1})$6 episodes for the SINDy–TD3 approach versus about $(s_t,a_t,r_t,s_{t+1})$7 episodes for TD3-only to obtain comparable sine-wave tracking performance (Abdelsalam et al., 24 Dec 2025).

Counter-Dyna applies the same structural impulse to HVAC control by factoring the state as $(s_t,a_t,r_t,s_{t+1})$8, where $(s_t,a_t,r_t,s_{t+1})$9 collects exogenous disturbances such as weather forecasts, electricity price forecasts, and time features. The key assumption is that $V_\theta(s)=\max_a Q_\theta(s,a)$0 is unaffected by control actions; only $V_\theta(s)=\max_a Q_\theta(s,a)$1 is action-dependent. The resulting counterfactual surrogate model predicts the next thermal state while reusing stored disturbance trajectories directly during rollouts. In the reported BOPTEST study, this design reduces the required interaction horizon to $V_\theta(s)=\max_a Q_\theta(s,a)$2 weeks, compared with prior state-of-the-art results using $V_\theta(s)=\max_a Q_\theta(s,a)$3-$V_\theta(s)=\max_a Q_\theta(s,a)$4 months of environment interactions (Vargas et al., 6 May 2026).

These papers collectively indicate a shift from “Dyna with a learned model” to “Dyna with a structured model.” A plausible implication is that in physical control domains, the main question is no longer whether to use model-based rollouts, but what inductive bias the rollout model should encode.

5. Domain-specific variants and applied Dyna systems

The Dyna template has also been specialized in application-driven ways that alter the planning distribution, the rollout policy, or the source of model knowledge.

“Dyna-H” modifies classic Dyna-Q by replacing random planning updates with heuristic-guided “nightmare” trajectories. In the reported $V_\theta(s)=\max_a Q_\theta(s,a)$5 RPG grid world, the heuristic planning loop selects actions that maximize a heuristic distance from the goal, thereby simulating bad branches rather than promising ones. Against one-step Q-learning and Dyna-Q, Dyna-H is reported to converge much faster; with $V_\theta(s)=\max_a Q_\theta(s,a)$6 planning steps, it reaches an approximately optimal policy around episode $V_\theta(s)=\max_a Q_\theta(s,a)$7, whereas Dyna-Q remains far above that level after $V_\theta(s)=\max_a Q_\theta(s,a)$8 episodes (Santos et al., 2011).

For new-product inventory control, “Data-driven inventory management for new products: An adjusted Dyna-$V_\theta(s)=\max_a Q_\theta(s,a)$9 approach with transfer learning” keeps the Dyna-Q structure but makes two systematic changes. First, both the exploration rate and the number of planning steps follow a search-then-convergence schedule, so planning effort decays over training. Second, a warm-start based on demand data from similar products initializes both the Q-table and the learned model. In the bakery case study, the adjusted Dyna-$g(s)=\|\nabla_s V(s)\|_2^2+\|H_V(s)\|_F^2,$0 shows up to a $g(s)=\|\nabla_s V(s)\|_2^2+\|H_V(s)\|_F^2,$1 reduction in average daily cost relative to Q-learning and up to a $g(s)=\|\nabla_s V(s)\|_2^2+\|H_V(s)\|_F^2,$2 reduction in training time within the same horizon relative to classic Dyna-$g(s)=\|\nabla_s V(s)\|_2^2+\|H_V(s)\|_F^2,$3 (Qu et al., 14 Jan 2025).

“Dyna-Think” moves the architecture into LLM-based computer-use agents. There the “world model” is not a separate transition network but an internal simulation behavior of the same LLM that also acts. Dyna-Think Imitation Learning reconstructs R1’s thinking so that it emphasizes world-model simulation relevant to the chosen action, and Dyna-Think Dyna Training first improves the world-model ability—via next-state prediction, state-difference prediction, or critique generation—and then improves the policy. On OSWorld, the reported result is that Dyna-Think achieves similar best-of-$g(s)=\|\nabla_s V(s)\|_2^2+\|H_V(s)\|_F^2,$4 performance to R1 while generating $g(s)=\|\nabla_s V(s)\|_2^2+\|H_V(s)\|_F^2,$5 fewer tokens on average, and critique generation is identified as the most effective world-model training objective (Yu et al., 31 May 2025).

Variant	Core modification	Reported setting
Dyna-H	Heuristic planning with “worst” trajectories	RPG path-finding (Santos et al., 2011)
Adjusted Dyna-$g(s)=\|\nabla_s V(s)\|_2^2+\|H_V(s)\|_F^2,$6	Decaying exploration and planning; transfer warm-start	Bakery inventory (Qu et al., 14 Jan 2025)
Dyna-Think	Single-LLM policy plus internal world-model simulation	OSWorld agents (Yu et al., 31 May 2025)

A plausible implication is that Dyna persists not as a single algorithmic recipe but as a reusable decomposition: collect real interaction, improve a model or internal simulator, and reuse that improvement to alter future decision updates.

6. Misconceptions, limitations, and terminological extensions

One recurring misconception is that any use of a learned model in RL is “Dyna.” The literature summarized here points to a narrower interpretation: Dyna uses the model to generate synthetic experience that is then consumed by the same update machinery used for real experience. This suggests that Dyna is a particular operational style within model-based RL, not a synonym for model-based RL as a whole (Das et al., 9 Mar 2026).

Another misconception is that adding more model rollouts automatically helps. In ALE, one-step Dyna yielded surprisingly little benefit over extra replay updates even with a perfect model, and with learned models the benefit depended strongly on rollout length because long rollouts introduce compounding model error (Holland et al., 2018). Structured-model papers make analogous limitations explicit: LNNs are less suitable for strongly non-Lagrangian, discontinuous, or contact-rich systems, EKF-based training adds covariance and Jacobian overhead, and Counter-Dyna currently assumes perfect disturbance forecasts (Das et al., 9 Mar 2026).

The name “Dyna” has also drifted beyond Sutton’s architecture. In “Dyna-LfLH,” the prefix refers to dynamic environments and dynamic obstacles, not to classic Dyna; the paper explicitly states that Dyna-LfLH is not a model-based RL algorithm, even though it shares the broad idea of synthesizing additional training experience (Ghani et al., 2024). In “Hi-Dyna Graph,” “Dyna” refers to a dynamic scene graph that combines a global topological graph with local dynamic relation subgraphs for robotic autonomy in human-centric environments (Hou et al., 30 May 2025). “DYNA” in Dynamic Episodic Memory Networks denotes a temporal-knowledge-graph memory system for frozen LLMs, where events are nodes and temporal relations are weighted, timestamped edges, and “DynaGraph” denotes dynamic topological reconfiguration in a multi-model LLM framework rather than a reinforcement-learning planner (Sarabadani et al., 14 Jun 2026, Guo et al., 28 May 2026).

This terminological spread matters because it can obscure the original technical meaning. In the reinforcement-learning literature, Dyna still denotes the integrated loop of acting, model learning, and planning through imagined experience. Outside RL, the label increasingly functions as shorthand for “dynamic.” Any encyclopedia treatment therefore needs to distinguish the classical architecture from later homonymous but conceptually separate systems.