---
title: Recursive Recurrent Neural Networks with Attention
url: https://www.emergentmind.com/topics/recursive-recurrent-neural-networks-with-attention-modeling-r-am
type: topic
---

# Recursive Recurrent Neural Networks with Attention

Recursive Recurrent Neural Networks with Attention Modeling (R²AM) describe a class of neural architectures that integrate deep recursive or recurrent structures with attention mechanisms to model complex dependencies in sequence processing and pattern recognition tasks. These models extend classical recurrent and convolutional approaches by incorporating explicit memory or recursion at distinct levels (spatial or temporal), and by leveraging differentiable attention to guide selective information extraction. Prominent instantiations include per-source-position recurrent attention for neural machine translation and recursive recurrent architectures for unconstrained scene text recognition, achieving state-of-the-art performance without domain-specific constraints [1607.05108][1603.03101].

## 1. Key Architectural Principles

R²AM architectures are defined by the combination of the following three modules:

1. **Recursive Feature Encoding:** Uses recursive convolutional or recurrent operations to extract hierarchical or context-sensitive representations from the input (e.g., images or sequences), improving the capacity to model long-range dependencies without excessive parameterization.
2. **Recurrent Sequence Modeling:** Employs RNNs (simple, factored, or LSTM) for capturing temporal or sequential structure, enabling implicit or explicit language modeling.
3. **Attention Modeling:** Implements soft-attention mechanisms to dynamically weight input features (spatial or sequential), allowing the network to selectively focus on pertinent regions or tokens during decoding.

These components are trained end-to-end within a unified differentiable framework, typically using cross-entropy loss and backpropagation (including Backpropagation Through Time when required) [1603.03101].

## 2. R²AM in Neural Machine Translation

In neural machine translation, R²AM augments the standard encoder–decoder attention model (Bahdanau et al., 2014) by introducing one small recurrent network (typically an LSTM) per source word, advancing along the target decoding time axis [1607.05108]. For a source sequence $x_1, \ldots, x_S$ and target sequence $y_1, \ldots, y_T$:

- **Encoder:** Bidirectional LSTM produces context-rich source annotations $h_i = [\overrightarrow{h}_i; \overleftarrow{h}_i]$ for $i = 1 \ldots S$.
- **Recurrent Attention Bank:** For each source position $i$, an attention-history LSTM accumulates information about recent attention allocations via
  $$
  d_{i,j} = \mathrm{LSTM}(d_{i,j-1},\, \tilde\alpha_{i,j})
  $$
  where $\tilde\alpha_{i,j}$ is a window of attention weights around $i$ at decoding step $j$.
- **Attention Scoring:** The score for $h_i$ at step $j$ is modulated by $d_{i,j}$:
  $$
  e_{ij} = v_a^\top \tanh(W_a [h_i; d_{i,j}] + U_a s_j)
  $$
  yielding normalized weights $\alpha_{ij}$ and context vector $c_j$.

The dynamic memory states $d_{i,j}$ enable the model to track coverage (fertility) and relative local ordering (distortion), facilitating improved translation accuracy. Empirical evaluations on English→German and Chinese→English demonstrate consistent BLEU improvements over baseline RNNSearch (up to +1.5 BLEU) [1607.05108].

## 3. Recursive Recurrent Architectures for Scene Text Recognition

In lexicon-free OCR for natural scene images, R²AM realizes its recursive feature extraction component via untied recursive convolutional layers:

- **Base CNN:** A deep convolutional backbone (8 layers, up to 512 channels) with interleaved max pooling on 32×100 gray-scale image inputs, culminating in 4096-unit fully-connected layers.
- **Untied Recursion:** Each recursive block applies a 3×3 convolution (shared shape, separately learned weights) for $T$ iterations ($T = 3$), extending the effective receptive field without increasing parameter count.
- **Factored Two-Layer RNN:** Implements character-level sequence modeling, with the first RNN layer modeling sequence context and the second integrating attended image contexts.
- **Soft Attention:** At each timestep $t$, the model computes spatial attention over feature map locations, generating
  $$
  c_t = \sum_{i=1}^D \alpha_{t,i} I_i
  $$
  where $\alpha_{t,i}$ is derived from energy scores combining image features and RNN state via linear or small MLP transformations.

The model achieves state-of-the-art unconstrained recognition accuracy on benchmarks such as Synth90k (95.3%), IIIT5k (96.8%), SVT (80.7%), ICDAR03-Full (97.0%), and ICDAR13 (90.0%), outperforming prior lexicon-free approaches by substantial margins [1603.03101].

## 4. Training Methods and Optimization

R²AM models are optimized end-to-end via stochastic gradient descent with modern architectural practices:

- **Loss Function:** Cross-entropy over target sequences, summed across minibatches. Teacher forcing is employed for RNN-based decoding.
- **Gradient Clipping:** Applied throughout to restrict the $L_2$ norm and avoid instability (e.g., norm ≤ 10 in OCR, ≤ 3 in translation).
- **Hyperparameters:** Embedding size, hidden state size (e.g., 1000 or 1024 per layer), batch size (128–256), number of recursive iterations ($T=3$ typical).
- **Optimizer and Scheduling:** SGD with dropout, stepwise learning rate annealing based on validation performance or epochs elapsed. No momentum/weight decay is required in the OCR instantiation. Early stopping is used for regularization.

Implementation on contemporary GPU hardware (Titan-X) yields inference times around 2.2 milliseconds per image for OCR tasks [1603.03101].

## 5. Modeling Fertility, Relative Distortion, and Coverage

The introduction of per-source recurrent attention enables R²AM to capture translation-specific biases:

- **Fertility (Coverage):** Attention-history states $d_{i,j}$ summarize how much a source word has been attended, preventing over-translation or neglect. The model learns to modulate future attention scores based on cumulative allocation, which is crucial in many-to-one or one-to-many alignments.
- **Relative Distortion:** By conditioning each $d_{i,j}$ on a local window of past attention (not just $\alpha_{i,j}$, but $\{\alpha_{i-k,j},\ldots,\alpha_{i+k,j}\}$), R²AM models the tendency of sequential source positions to be attended in order. This allows learning appropriate reordering patterns directly from data.
- **Generalization:** Such mechanisms inject a temporal structure into content-based attention, supporting extensions to other tasks—with appropriate per-encoder-position recurrent cells and alternately-sized context windows [1607.05108].

## 6. Extensions, Syntheses, and Broader Implications

R²AM provides a general template for integrating temporal structure, recursive depth, and adaptive focus in neural architectures:

- **Applicability to Other Sequence-to-Sequence Tasks:** The per-position recurrent attention or recursive convolutional encoding can be adapted to domains where coverage and reordering are critical, such as summarization, dialogue, or captioning.
- **Alternative Memory Architectures:** Extensions include fully dynamic memory (where the content vectors themselves are updated based on attention history), alternate recurrent cell types (GRU, convolutional, highway networks), or expanded local attention windows.
- **Parameter Efficiency:** Recursive untied convolutional layers provide increased model depth and receptive field without proportional parameter growth, which is especially important for scalability and efficiency in large-scale settings.
- **Empirical Gains:** BLEU and accuracy improvements affirm the practical value of integrating recursive/recurrent dynamics into attention modeling frameworks, and the approach is validated on several publicly recognized benchmarks [1607.05108][1603.03101].

## 7. Implementation Details and Empirical Results

The following summarizes critical design properties and benchmark results from the canonical R²AM implementations:

| Task                           | Architecture                | Notable Metrics                           | Benchmark Datasets                  |
|---------------------------------|-----------------------------|-------------------------------------------|-------------------------------------|
| Neural Machine Translation      | Bi-LSTM encoder, per-word LSTM attention | BLEU +0.5 to +1.5 over RNNSearch        | EN–DE (newstest2014/15), ZH–EN (MT05)|
| Lexicon-Free Scene Text OCR     | 8-layer recursive CNN, 2-layer RNN, attention | Synth90k 95.3%, IIIT5k 96.8%, SVT 80.7% | Synth90k, IIIT5k, SVT, ICDAR03/13   |

Configurations are chosen based on dataset and task, with differences in the type of recurrent cell, network width, optimizer setup, and attention energy parameterization. Training uses standard cross-entropy objectives, and at inference, the attention mechanism allows for output without lexicon or external N-gram language modeling [1603.03101][1607.05108].

Source: https://www.emergentmind.com/topics/recursive-recurrent-neural-networks-with-attention-modeling-r-am