---
title: Dynamic Memory Network (DMN)
url: https://www.emergentmind.com/topics/dynamic-memory-network-dmn
type: topic
---

# Dynamic Memory Network (DMN)

A Dynamic Memory Network (DMN) is a neural architecture for machine reasoning over sequential inputs, developed to address natural language question answering (QA), sentiment analysis, part-of-speech tagging, and multimodal tasks. DMN frameworks integrate compositional encoding, iterative attention-based memory, and generative decoding into an end-to-end trainable system, differing from prior memory architectures by leveraging “episodic memory”—multi-pass, question-dependent reasoning on contextualized facts [1506.07285].

## 1. Core Components and Workflow

A standard DMN consists of four differentiable modules:

1. **Input Module**: Encodes raw input sequences (words, sentences, images) into distributed vector representations via a Gated Recurrent Unit (GRU) or Bi-GRU.
2. **Question Module**: Encodes the textual question as a fixed-length vector, also via a GRU, typically sharing word embeddings with the input encoder.
3. **Episodic Memory Module**: Performs iterative attention over encoded input facts, dynamically updating a shared memory vector through multiple passes (“episodes”). This module conditions attention on both the question and memory state.
4. **Answer Module**: Generates the output (answer, class label, prediction sequence) by decoding from the final memory state using a GRU decoder or linear classifier, optionally concatenating the question encoding.

The interaction between these modules allows a DMN to perform both retrieval (via attention) and composition (via recurrent memory). The high-level workflow is: encode input and question → iterate attention/memory (“episodes”) → decode answer [1506.07285].

## 2. Input and Question Representation

Both input and question are embedded using pre-trained word vectors (e.g., GloVe) which are fine-tuned during training. For an input sequence \( w_1, ..., w_T \), each token \( w_t \) is mapped to an embedding \( x_t = L[w_t] \) through a shared matrix \( L \). A GRU processes these as

\[
\begin{aligned}
z_t &= \sigma(W^{(z)} x_t + U^{(z)} h_{t-1} + b^{(z)}) \\
r_t &= \sigma(W^{(r)} x_t + U^{(r)} h_{t-1} + b^{(r)}) \\
\tilde h_t &= \tanh(W x_t + r_t \circ U h_{t-1} + b^{(h)}) \\
h_t &= z_t \circ h_{t-1} + (1 - z_t) \circ \tilde h_t
\end{aligned}
\]

For multi-sentence inputs, end-of-sequence markers allow partitioning hidden state sequences at sentence boundaries into fact vectors \( \{c_1,...,c_{T_C}\} \). The question is encoded via the same GRU to a single state \( q \) [1506.07285].

DMN+ variants introduce hierarchical encoding (sentence-level followed by bidirectional GRU “fusion”) and image feature spatial encoding with bidirectional processing over image-region vectors [1603.01417].

## 3. Episodic Memory and Iterative Attention

The episodic memory module enables multi-step reasoning by iteratively focusing on relevant facts, adjusting its memory with each pass. At iteration \( i \), for each fact \( c_t \), a scalar gate \( g^i_t \) is computed as:

\[
z_t = [c_t, m^{i-1}, q,\; c_t \circ q,\; c_t \circ m^{i-1},\; |c_t - q|,\; |c_t - m^{i-1}|,\; c_t^\top W^{(b)} q,\; c_t^\top W^{(b)} m^{i-1}]
\]
\[
g^i_t = \sigma(W^{(2)} \tanh(W^{(1)} z_t + b^{(1)}) + b^{(2)})
\]

An attention-gated GRU updates its state as:

\[
h^i_t = g^i_t \;\mathrm{GRU}(c_t, h^i_{t-1}) + (1-g^i_t) h^i_{t-1}
\]

The final episode vector is \( e^i = h^i_{T_C} \), which is used to update memory via

\[
m^i = \mathrm{GRU}(e^i, m^{i-1})
\]

The process is repeated for up to \( T_M \) episodes, or until a learned condition is satisfied [1506.07285]. DMN+ adapts the aggregation by using attention-based GRUs and untied per-episode memory updates via ReLU transforms [1603.01417].

Tensor-based DMN extensions (DMTN) replace the hand-crafted feature vector \( z_t \) in the gating function with a neural tensor network (NTN), modeling richer pairwise relations among fact, memory, and question representations [1703.03939].

## 4. Variants and Extensions

The DMN architecture has been adapted for multiple modalities and learning regimes:

- **DMN+ (Visual/Textual QA)**: Incorporates contextual input fusion layers, attention-based recurrent aggregation, modality-specific input modules (e.g., region-level image GRU), and untied memory updates, supporting both text and visual question answering [1603.01417].
- **DMTN (Tensor Attention)**: Implements a tensor-based attention mechanism in the episodic memory module, enabling finer-grained modeling of fact–question–memory interactions. DMTN achieves substantial gains on weakly supervised QA benchmarks [1703.03939].
- **DMN in Few-Shot Meta-Learning**: By integrating a multi-hop DMN into prototypical network frameworks (e.g., DMB-PN), the episodic memory process refines both instance and class prototype embeddings in low-data regimes, improving event detection performance in few-shot settings [1910.11621].

## 5. Training, Optimization, and Hyperparameters

DMNs are trained end-to-end with standard cross-entropy objectives over the target answer sequence. In datasets with supervision available for the attention gates (e.g., bAbI), an additional gate-loss term is weighted in the total loss. Adam optimizer with default hyperparameters is typically employed. Regularization is provided by $L_2$ weight decay and dropout.

Typical hidden state sizes are in the range $100$–$200$, memory iterations $T_M$ up to $5$, and pre-trained GloVe embeddings are fine-tuned in learning [1506.07285]. Variant models utilize SGD or other optimizers as explained in experimental sections [1910.11621]. Hyperparameters in few-shot settings include distinct embedding dimensions for different positional and lexical features [1910.11621].

## 6. Empirical Findings

DMN and its variants achieve strong empirical results across various domains:

- **bAbI QA (textual QA; 20 tasks)**: DMN passes 18/20 tasks (≥95% accuracy, mean 93.6%), surpassing the original Memory Network (MemNN) [1506.07285]. DMTN (with tensor attention) passes 16/20 in the weakly-supervised 1K bAbI setup, an 80% improvement over baseline DMN [1703.03939]. DMN+ reaches mean error 2.8% on bAbI-10k, outperforming prior end-to-end networks [1603.01417].
- **Stanford Sentiment Treebank**: DMN attains 88.6% in binary and 52.1% in fine-grained classification (prev. bests: 88.1% CNN, 51.0% CT-LSTM) [1506.07285].
- **WSJ Penn Treebank (POS tagging)**: Single DMN reaches 97.56% accuracy, matching or exceeding previous best [1506.07285].
- **Visual Question Answering (VQA)**: DMN+ records test-dev overall accuracy 60.3%, compared to 58.7% for prior stacked attention networks; it captures spatially precise visual focuses [1603.01417].
- **Few-Shot Event Detection**: DMB-PN (with DMN) produces robust class prototypes and sentence encodings, outperforming baseline prototypical and matching network approaches in low-data regimes [1910.11621].

In all reported settings, a single DMN model architecture, with minimal task-specific adaptation (preprocessing or answer module), attains or surpasses state-of-the-art performance on diverse NLP and VQA tasks [1506.07285][1603.01417].

## 7. Limitations and Future Directions

Proposed limitations and directions include:

- **Computational Complexity**: Episodic memory and tensor-based attention increase compute and parameter cost, particularly for multiway tensor contractions. For tractability, DMTN restricts to two-way tensor interactions and limits slice counts $K$ [1703.03939].
- **Supervision Requirements**: While DMN can function without supporting-fact supervision, richer attention mechanisms benefit from labeled attention (where available), and weak supervision remains challenging for certain tasks.
- **Expressivity versus Scalability**: Bidirectional fusions and attention-gated recurrence offer improved reasoning, yet further developments in scalable sparse or hard attention are being explored [1603.01417].
- **Extending Modalities**: Future adaptations target video QA and multimodal dialog, aiming to leverage iterative memory capabilities for joint textual/visual or broader input types [1603.01417].
- **Meta-Learning Regimes**: Leveraging DMN multi-hop mechanisms within meta-learning frameworks improves robustness under sample scarcity, a trend expected to continue for structured event extraction and few-shot learning [1910.11621].

A plausible implication is that optimizing the trade-off between flexible memory composition and computational efficiency will continue to motivate architectural innovation in DMN-inspired systems.

Source: https://www.emergentmind.com/topics/dynamic-memory-network-dmn