---
title: Dynamic Reinforcement Recurrent Network (DRRN)
url: https://www.emergentmind.com/topics/dynamic-reinforcement-recurrent-network-drrn
type: topic
---

# Dynamic Reinforcement Recurrent Network (DRRN)

The Dynamic Reinforcement Recurrent Network (DRRN) refers to a class of reinforcement learning (RL) architectures designed to address environments where both states and actions are structured or variable—commonly as sequences or sets of natural language strings—and/or where the environment is only partially observable, necessitating memory and sequence modeling. There are effectively two complementary lines of DRRN methodology: the original Deep Reinforcement Relevance Network, which disentangles state and action embedding for large, natural language action-spaces [1511.04636], and the recurrent, memory-based DRRN variants that leverage LSTM-based sequence modeling for control in Partially Observable Markov Decision Processes (POMDPs) subject to unknown and time-varying disturbances [2307.15931]. Both frameworks enable robust Q-value approximation in nontrivial, non-tabular settings, but employ distinct architectural strategies.

## 1. Foundational Architectures of DRRN

The DRRN introduced in “Deep Reinforcement Learning with a Natural Language Action Space” [1511.04636] targets problems where both the state and available actions are described by unconstrained natural language. The architecture uses two deep, parameter-disjoint feed-forward networks: one encodes the state text, and the other encodes each action text, yielding continuous embeddings for each. An interaction function $g(\cdot,\cdot)$—typically the inner product—scores each (state, action) embedding pair to approximate the Q-function $Q(s,a)$.

In parallel, recurrent DRRN variants are developed to address partial observability and dynamic disturbances, as described in “Dynamic deep-reinforcement-learning algorithm in Partially Observed Markov Decision Processes” [2307.15931]. Here, the agent employs LSTM-based actor and critic networks to aggregate historical sequences of observations and actions for internal belief-state estimation, supporting policy optimization under incomplete information.

## 2. Mathematical Formulation and Network Structure

### 2.1 DRRN for Natural Language Action Spaces

Let $s\in\mathcal S$ (state text) and $a\in\mathcal A$ (action text). The embedding functions are
\[
\phi_s(s; \theta_s) = h_{L,s},
\qquad
\phi_a(a; \theta_a) = h_{L,a}
\]
where $h_{l,s} = f(W_{l,s} h_{l-1,s} + b_{l,s})$ and $h_{l,a} = f(W_{l,a} h_{l-1,a} + b_{l,a})$, starting from bag-of-words or bag-of-n-grams representations. Q-values are estimated via
\[
Q(s,a;\Theta) = g(\phi_s(s),\phi_a(a))
\]
with $g(u,v)=u^\top v$ (inner product) or $g(u,v)=u^\top M v$ (bilinear).

### 2.2 Recurrent DRRN for POMDPs

Given a POMDP $(\mathcal S, \mathcal A, \mathcal O, \mathcal T, \mathcal Z, \mathcal R)$, the history $I_t^{\mathfrak c} = (o_0...o_t, a_0...a_{t-1})$ is summarized by the LSTM hidden state $h_t$. The critic evaluates
\[
Q^\pi(h_t, a_t) = E \bigl[\sum_{k=0}^\infty \gamma^k r_{t+k+1} \mid h_t, a_t\bigr]
\]
with the Bellman target (for TD3 backbone)
\[
y_t = r_t + \gamma\, \min_{i=1,2} \bar Q_{\theta_i'}(h_{t+1}, \bar\pi_{\phi'}(h_{t+1}))
\]
and loss
\[
L(\theta_i) = E_{(h_t,a_t,r_t,h_{t+1})\sim D}[(Q_{\theta_i}(h_t,a_t)-y_t )^2]
\]
All networks are trained via BPTT.

## 3. Training Procedures and Exploration Policies

### 3.1 DRRN for Natural Language

The training uses standard Q-learning with experience replay. At each step:
1. Encode state and each available action;
2. Compute $Q$-values for all legal (state, action) pairs;
3. Sample actions via a softmax policy:
   \[
   \pi(a_t = a^i \mid s) = \frac{\exp(\alpha Q(s,a^i))}{\sum_j \exp(\alpha Q(s,a^j))}
   \]
4. Update via stochastic gradient descent on the squared temporal-difference loss using Q-learning target with delayed target network parameters.

### 3.2 Recurrent DRRN for POMDPs

Training employs off-policy RL (TD3 backbone), with LSTM-based actor and twin-critic architectures. Key steps include:
- Sequence sampling of fixed length $l$ for LSTM input.
- Optimization of actor and critic losses over minibatches.
- Replay buffer storing full trajectories, including hidden/cell states for efficient critic updates (notably in the H-TD3 variant).

## 4. Empirical Performance and Robustness

### 4.1 Natural Language Domains

On text-based games “Saving John” (deterministic transitions) and “Machine of Death” (stochastic, paraphrased actions), DRRN achieved:
- Faster convergence and higher final reward than linear or monolithic DQN baselines.
- “Saving John”: DRRN ≈ 18.7 average cumulative reward, best baseline ≈ 9.0.
- “Machine of Death”: DRRN ≈ 11.2, best baseline ≈ 5.2.
- For paraphrased actions, reward decremented only slightly (11.2 to 10.5), and $R^2\approx0.95$ on Q-values for original vs. paraphrase strings.
- DRRN performance is on par with strong human players in terms of reward, exceeding novice-level human baselines [1511.04636].

### 4.2 POMDPs and Dynamic Disturbance Robustness

In Pendulum-v0 with structured and unstructured disturbance, performance normalized returns were:
- Vanilla TD3: ≈ 0 under disturbance.
- LSTM-TD3 (no past action): ≈ 0.45 under random sinusoidal disturbance.
- LSTM-TD3 (+actions): ≈ 0.68.
- Single-head LSTM-TD3 (1ha1hc): ≈ 0.82, best among variants.
- H-TD3: 0.70, with reduced computational cost (30–50% less wall-clock per iteration) [2307.15931].

Key findings include the criticality of including past action sequences for dynamic disturbance environments, and benefits in both structured and unstructured POMDP regimes of longer LSTM input sequences.

## 5. Generalization, Ablations, and Architectural Analysis

DRRN’s separation of state and action embeddings enables:
- Robust semantic generalization; architectures that learn a joint embedding or rely on text token enumeration overfit the specific wording and perform poorly on paraphrased inputs [1511.04636].
- Empirical ablations in recurrent DRRN variants show the necessity of including both past observations and actions for robust performance in environments with temporal or dynamic disturbances [2307.15931].
- Sequence length ablation indicates longer LSTM windows enhance identification of environment periodicity and noise suppression.

The structure of single-headed LSTM networks outperformed multi-head variants, supporting the design of compact, integrated sequential modules for both policy and value networks.

## 6. Practical Implications and Scalability

The design of DRRN frameworks makes them naturally extensible to:
- Large or unbounded, even open-vocabulary, action spaces (e.g., natural language command spaces), without requiring architecture reconfiguration [1511.04636].
- Complex, memory-dependent control in physical or simulated systems with partial observability, by leveraging recurrent belief-state estimation and action-augmented sequence input [2307.15931].

Efficient variants, such as H-TD3, reduce inference and training overhead without major performance degradation, supporting real-time deployment in operational POMDP control.

## 7. Comparative Analysis and Key Insights

- The explicit decoupling of state and action encoders permits each network to specialize to its respective domain (long-form narrative for state, short imperative for action), a property not shared by single-encoder DQN or linear concatenative models.
- Embedding both state and action in continuous vector spaces is essential for semantic generalization and robust value estimation in combinatorially large domains.
- In LSTM-based DRRN, explicit inclusion of the action sequence is causally and empirically critical to distinguish between latent environmental dynamics and spurious disturbance inputs; omitting this information results in significant degradation of control performance in disturbed environments.
- Recurrent DRRN variants provide a framework for practically robust POMDP control, with quantifiable advantages over non-recurrent or shallow RL approaches in both performance and training efficiency [2307.15931].

Further advances may involve extending DRRN-style models to hierarchical RL and multi-agent POMDPs, as well as integrating transformers and attention mechanisms for richer sequential processing within the recurrent DRRN paradigm.

Source: https://www.emergentmind.com/topics/dynamic-reinforcement-recurrent-network-drrn