---
title: Deformation-aware LSTM (DT-LSTM)
url: https://www.emergentmind.com/topics/deformation-aware-lstm-dt-lstm
type: topic
---

# Deformation-aware LSTM (DT-LSTM)

Deformation-aware Long Short-Term Memory networks (DT-LSTM) are a class of recurrent neural modules designed to enhance robustness to geometric deformations (such as scaling and rotation) in sequential visual data. By integrating deformable convolutions into the input-to-state transitions of ConvLSTM cells, DT-LSTM enables adaptive spatio-temporal modeling for tasks including human body emotion recognition in videos of arbitrary length [2010.14607].

## 1. Architectural Principles of DT-LSTM

DT-LSTM modifies the standard ConvLSTM cell by incorporating deformable convolutions (⊗) in place of fixed convolutions (∗) for input-to-state transformations. Given input tensor $X_t \in \mathbb{R}^{C \times H \times W}$, previous hidden state $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$, and previous cell state $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$, the cell computes:

$$
\begin{aligned}
  i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \\
  f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \\
  g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \\
  o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \\
  C_t &= f_t \circ C_{t-1} + i_t \circ g_t \\
  H_t &= o_t \circ \tanh(C_t)
\end{aligned}
$$

Here, $\sigma(\cdot)$ denotes the sigmoid function, $\tanh(\cdot)$ the hyperbolic tangent, and $\circ$ the Hadamard (elementwise) product. All state-to-state transitions utilize standard convolutions, whereas the input-to-state transitions for the input and candidate gates ($i_t$ and $g_t$) utilize deformable convolutions with offsets $\Delta p$ learned at each time step by a lightweight offset network [2010.14607].

## 2. Deformable Convolution Formulation

The deformable convolution operation replaces fixed-grid sampling with spatially adaptive sampling. For a convolution weight kernel $w(\cdot)$ over a standard receptive field $R$ (e.g., $3 \times 3$), standard convolution computes:

$$
y(p_0) = \sum_{p_n \in R} w(p_n) \cdot x(p_0 + p_n)
$$

In deformable convolution, the sampling locations are perturbed by learned offsets $\Delta p_n(p_0)$:

$$
y(p_0) = \sum_{p_n \in R} w(p_n) \cdot x(p_0 + p_n + \Delta p_n(p_0))
$$

Off-grid sampling is handled by bilinear interpolation. The set of offsets $\{\Delta p_n(p_0)\}_{p_n \in R}$ is produced via a learnable offset subnetwork $\phi_\text{offset}$:

$$
\Delta P = \phi_\text{offset}(X; W_\text{off}) \in \mathbb{R}^{2N \times H \times W}
$$

where $N = |R|$ and $W_\text{off}$ are the weights of a $3 \times 3$ convolution outputting $2N$ offset maps per input location [2010.14607].

## 3. Offset Subnetwork and Gate-specific Adaptation

For each gate $k \in \{i, g\}$, the corresponding offset network produces gate-specific offsets per time step:

$$
\Delta P_t^k = \phi_\text{off}^k(X_t) \in \mathbb{R}^{2N \times H \times W}
$$

Each spatial location $p_0$ thus possesses $N$ learned 2D offsets. These offsets deform the regular grid $R$ in the deformable convolution, yielding gate-adaptive spatial behavior. This architectural design enables the cell to locally adapt the spatial receptive fields of feature integration, increasing tolerance to geometric image transformation [2010.14607].

## 4. Training Protocol and Initialization

DT-LSTM models are trained end-to-end with a standard cross-entropy loss over target classes. All convolutional kernels—including deformable and offset networks—utilize He-normal initialization. Offset biases are initialized to zero, ensuring initial behavior matches a standard convolution. $L_2$ regularization (weight decay coefficient $10^{-4}$) is applied for all learnable parameters. The Adam optimizer is employed with an initial learning rate of $10^{-3}$, decayed by a factor of $0.1$ every 20 epochs, with early stopping based on validation accuracy [2010.14607].

## 5. Experimental Configuration and Workflow

The GEMEP corpus (145 clips, 17 emotion categories) serves as the primary testbed. Video preprocessing includes:

- Uniform temporal sampling with jitter (T=32 frames per clip)
- Frame resizing (from $720 \times 576$ to $112 \times 112$)
- Data augmentation: random translations ($\pm25$ pixels), rotations ($\pm30^\circ$), brightness shifts, and Gaussian blur

This yields approximately 9,052 augmented video clips, partitioned 80%/20% for training/validation. The network architecture comprises:

1. 3D-CNN block ($2 \times 3$D convolution + pooling) for short-term spatio-temporal features
2. Three DT-LSTM cells applied at frame indices $\{8, 16, 24\}$, propagating hidden states temporally
3. Shallow 2D CNN ($3 \times$ conv + average pooling) over $H_t$ for spatial feature extraction
4. Global average pooling (spatial and temporal) yielding a $1 \times 1 \times C$ feature representation
5. Fully-connected layer with softmax activation for 17-way emotion classification

Pseudocode for the DT-LSTM cell and the end-to-end training loop is provided in the original work, specifying distinct offset computations per gate and detailed sequential data flow [2010.14607].

## 6. Empirical Performance and Application

DT-LSTM achieves 98.8% accuracy for whole human body emotion recognition on the GEMEP dataset validation set, establishing state-of-the-art performance within the experimental context. The framework is tailored for robust emotion analysis from body pose and motion, directly addressing the challenge of intra-class variability due to pose and geometric deformation. The deformation-aware mechanism enables the model to generalize across diverse visual appearances and movement styles, a key advantage over prior approaches focusing only on static or facial information [2010.14607].

## 7. Implementation Highlights and Pseudocode

The following table summarizes principal computational blocks as instantiated in the DT-LSTM approach:

| Block             | Operation Type            | Distinguishing Feature             |
|-------------------|--------------------------|------------------------------------|
| Offset Subnet     | 2D Convolution           | Outputs $2N$ offsets per spatial position |
| Input-to-state    | Deformable Convolution   | Per-gate, per-timestep learned offsets    |
| State-to-state    | Standard 2D Convolution  | Fixed convolutional neighborhoods          |

Training pseudocode:

```python
for epoch in 1..E:
  for each minibatch {video_clips, labels}:
    X_3D_feats = ThreeD_CNN(video_clips)   # [B,T,C,H,W]
    H, C = zeros, zeros
    for t in 1..T:
      H, C = DefConvLSTMCell(X_3D_feats[:,t], H, C)
    spatial_feats = TwoD_CNN(H)
    pooled = GlobalAvgPool(spatial_feats)
    logits = FC(pooled)
    loss = CrossEntropy(logits, labels)
    loss.backward(); optimizer.step()
```

The DefConvLSTMCell operation involves offset generation (per gate, via $\phi_\text{off}$) and deformable convolution for the input-to-state connections, with state updates as defined by the equations above.

---

Source: https://www.emergentmind.com/topics/deformation-aware-lstm-dt-lstm