---
title: LSTM-based Recurrent Policies
url: https://www.emergentmind.com/topics/long-short-term-memory-lstm-based-recurrent-policies
type: topic
---

# LSTM-based Recurrent Policies

Long Short-Term Memory (LSTM)-based recurrent policies are a class of temporal function approximators for sequential decision making, where the policy or value function is parameterized by an LSTM recurrent neural network. LSTM-based policies are explicitly designed to handle long-range temporal dependencies and partial observability in stochastic control and reinforcement learning (RL), leveraging the gating and memory cell structure of LSTMs to encode information from arbitrarily long histories of observations and actions.

## 1. Fundamental Principles of LSTM-based Recurrent Policies

LSTM-based policies replace Markovian policies $\pi(a_t|s_t)$ with functions of the history, encoding the partial-observation sequence via the LSTM’s internal state. The essential architectural element is the LSTM cell, defined by input, forget, and output gates; at each time $t$, the LSTM ingests the current input, previous hidden state, and previous cell state, producing a new memory summary that parameterizes either the action distribution (policy) or value estimate.

Let $x_t$ denote the per-step input to the LSTM (e.g., current observation $o_t$, prior action $a_{t-1}$). The recurrence is:
\[
\begin{aligned}
f_t &= \sigma(W_f x_t + U_f h_{t-1} + b_f) \\
i_t &= \sigma(W_i x_t + U_i h_{t-1} + b_i) \\
o_t &= \sigma(W_o x_t + U_o h_{t-1} + b_o) \\
\tilde{c}_t &= \tanh(W_c x_t + U_c h_{t-1} + b_c) \\
c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\
h_t &= o_t \odot \tanh(c_t)
\end{aligned}
\]
The final hidden state $h_t$ is mapped to the policy output, e.g., $\pi_\theta(a_t | h_t)$ or $Q_\omega(h_t, a_t)$.

## 2. Architectures and Representative Algorithms

LSTM-based recurrent policies have been developed for both model-free RL (off-policy and on-policy), adaptive control, stochastic control with delay, and trading. Core architectures include:

- **Model-free RL**:
    - *Recurrent DPG (RDPG)* and *Recurrent SAC (RSAC)* prepend one or more LSTM layers to the standard multilayer perceptron (MLP) actor-critic backbone. In each case, the LSTM outputs are used as hidden summaries to condition policy/actions and Q-values [1512.04455, 2110.12628].
    - *Deep Recurrent Q-Network (DRQN)* replaces the first post-convolutional layer of DQN with an LSTM, consuming per-frame features instead of frame stacks [1507.06527].
    - *LSTM-TD3*: Both actor and critic ingest recent L observation–action pairs with an LSTM that fuses memory and current-feature MLPs, supporting robust partially observable RL (POMDPs) [2102.12344].
    - *Perception–Prediction–Reaction (PPR) Agent*: Multicore hierarchical LSTM structure, incorporating a slow-ticking core for long-term memory and three fast-ticking LSTMs, combined by KL-regularization of the induced policies [2006.15223].

- **Hybrid methods**:
    - Supervised-LSTM for state representation is jointly trained with a DQN head (hybrid SL+RL objective), explicitly learning hidden state representations from sequence data [1509.03044].

- **Adaptive control**:
    - LSTM augments standard feedforward adaptive neural network (ANN) controllers, providing rapid correction for transients, improving response to abrupt changes, and supporting formal UUB stability results via Lyapunov analysis [2301.02316].

- **Continuous-time control with memory**:
    - LSTM parameterizes policies for stochastic control with delay, outperforming finite window-based feedforward nets, especially in infinite-delay tasks [2101.01385].

- **Financial RL**:
    - Policy parameterized as an L-layer LSTM stack, mapping raw return histories to continuous trade signals; optimized by directly maximizing risk-adjusted objectives (e.g., Sharpe ratio) [1707.07338].

## 3. Training Methodologies and Optimization

Training of LSTM-based policies employs backpropagation through time (BPTT) over rollouts, with objectives and update rules depending on the base algorithm:

| Objective Class            | Example Algorithms              | Loss / Update Mechanism                                                                                               |
|----------------------------|---------------------------------|----------------------------------------------------------------------------------------------------------------------|
| Policy gradient / Actor-Critic | RDPG, RSAC, RTD3, LSTM-TD3       | Policy or actor loss is computed along LSTM-unrolled sequences, optionally with entropy terms; Q-functions/Bellman error for critics; soft target network updates [2110.12628, 2102.12344]. |
| Value-based (Q-Learning)   | DRQN, Hybrid RL–LSTM DQN        | Temporal-difference targets, sequence-based batches, clipped gradients, target network for stability [1507.06527, 1509.03044].             |
| Direct risk-sensitive      | LSTM-trader                     | Loss corresponds to Sharpe or Downside Deviation Ratio; gradients via BPTT, with regularization [1707.07338].          |
| Adaptive control           | Hybrid ANN + LSTM               | Lyapunov-driven policy update, LSTM trained via MSE to predict residual errors, alternating with ANN adaptation [2301.02316].    |
| Stochastic control with delay| LSTM delayed-control            | Loss corresponds to simulated cost-to-go or utility; optimized by BPTT and Adam [2101.01385].                             |

Key practical elements include experience replay buffers storing entire episodes or subsequences, zero-initialization of hidden states when sampling, gradient clipping to stabilize recurrent optimization, and layer normalization for LSTM parameters [1512.04455, 2110.12628]. Layer stacking and non-recurrent input dropout are additional common engineering choices [1707.07338].

## 4. Empirical Performance and Observed Benefits

LSTM-based recurrent policies consistently match or outperform comparable feedforward and non-recurrent policies in settings where partial observability, path dependence, or long-term credit assignment is present:

- In continuous control POMDPs with missing/noisy observations, LSTM-equipped agents resolve unobserved states, infer dynamics, and integrate temporal context [1512.04455, 2102.12344].
- Deterministic recurrent off-policy methods (RDPG, RTD3) may struggle with long-term credit assignment and exploration, whereas stochastic policies (RSAC) with LSTM memory achieve more robust performance on memory-demanding benchmarks [2110.12628].
- In adaptive control, LSTM augmentation of adaptive NNs yields dramatically reduced overshoot, faster settling (<1s, 70% RMS tracking error reduction), and superior transient compensation of sharp dynamic changes [2301.02316].
- For stochastic control with delay, LSTM policies converge more quickly, exhibit lower variance, and match analytical optima even with infinite memory, unlike fixed-window feedforward nets [2101.01385].
- In high-frequency trading, LSTM-based trading agents discover profitable and low-variance strategies, outperforming shallow recurrent baselines and enabling position inertia [1707.07338].
- Designed architectures like the PPR agent leverage hierarchical LSTMs plus auxiliary KL losses to sharply reduce sample complexity and raise asymptotic scores in multi-task, high-memory environments [2006.15223].
- Value-based DRQN shows greater stability than DQN in flickering/missing observation regimes, facilitating learning without explicit frame stacking, and delivering robust generalization to new observation regimes [1507.06527].

## 5. Theoretical Guarantees and Analysis

Several works provide formal and empirical support for the stability and effectiveness of LSTM-based recurrent policies:

- **Lyapunov-based guarantees**: In adaptive control, integration of LSTM residual predictors with ANN controllers yields uniform ultimate boundedness for all plant and network signals, enforced by additional robustifying control laws and Lyapunov analysis [2301.02316].
- **Gradient stability**: The LSTM gating structure mitigates vanishing/exploding gradients, permitting BPTT over long sequences and supporting learning in tasks requiring memory over hundreds of steps [1512.04455, 2101.01385, 2110.12628].
- **Auxiliary objectives for representation**: Hybrid supervised+RL architectures and KL-regularized auxiliary losses (as in the PPR agent) guide the internal memory towards compressive and predictive state summaries, enhancing generalization and representation learning [2006.15223, 1509.03044].
- Empirical ablations demonstrate that removing memory, switching to windowed input, or omitting past actions from the LSTM input sharply degrades performance, confirming the necessity of recurrent structure for long-range temporal credit assignment and POMDP reasoning [2102.12344, 2110.12628].

## 6. Architectural Innovations and Design Considerations

Architectural design choices in LSTM-based recurrent policies include:

- **Depth**: Single versus multi-layer LSTM stacks; two-layer configurations (hidden size 256) are common in high-dimensional RL [2110.12628].
- **Fusion strategies**: Fusing LSTM memory summaries with current features via additional MLPs in actor–critic architectures substantially improves both MDP and POMDP performance over naïve memory integration [2102.12344].
- **Hierarchical memory**: Temporal hierarchies (e.g., slow-ticking cores in PPR), parameter sharing, and auxiliary consistency losses further enhance long-term retention and sample efficiency [2006.15223].
- **Hybrid modules**: Augmenting feedforward policy heads or base controllers (ANNs) with LSTM-driven correction policies enables better handling of high-frequency or abrupt dynamics [2301.02316].
- **Regularization**: Non-recurrent input dropout, BPTT gradient clipping, and normalization are essential for stable LSTM training in RL.

A recurring theme is that LSTM-based recurrence allows differentiable policies to learn the minimal sufficient statistic—compressing the entire observation-action history into a fixed-dimensional latent state on which to condition decisions—without hand-crafted memory windows or explicit state modeling [1512.04455, 1509.03044].

## 7. Applications, Limitations, and Future Directions

LSTM-based recurrent policies are now standard in deep RL for robotics, adaptive control, finance, and sequential decision problems with incomplete state information or latent system parameters. Their application spans memory-based locomotion, high-frequency autonomous trading, resource allocation in customer relationship management, and continuous-time stochastic control with delay.

Limitations include the interpretability of learned memory states, the potential sensitivity to hyperparameters such as history truncation length, and computational overhead for long sequence unrolls during training [2102.12344]. Extensions include regularized or disentangled representations of memory, dynamic memory-length adaptation, hierarchical module design, combination with explicit Bayesian filtering, and more robust exploration via stochastic policy classes [2110.12628, 2006.15223].

In sum, LSTM-based recurrent policies provide a scalable, model-free, end-to-end method for deep temporal abstraction and control, representing a key advance for RL and adaptive sequential decision-making in partially observed and memory-intensive environments.

Source: https://www.emergentmind.com/topics/long-short-term-memory-lstm-based-recurrent-policies