Deformation-aware LSTM (DT-LSTM)

• Deformation-aware LSTM (DT-LSTM) is a recurrent neural module that integrates deformable convolutions to adaptively adjust spatial receptive fields, enhancing robustness to geometric deformations in sequential visual data.
• The architecture replaces fixed input-to-state convolutions with learned offset-based deformable convolutions, achieving state-of-the-art performance (98.8% accuracy on GEMEP) on human body emotion recognition tasks.
• Its training protocol uses cross-entropy loss with He-normal initialization and Adam optimization, ensuring effective end-to-end learning for adaptive spatio-temporal feature integration.

Deformation-aware Long Short-Term Memory networks (DT-LSTM) are a class of recurrent neural modules designed to enhance robustness to geometric deformations (such as scaling and rotation) in sequential visual data. By integrating deformable convolutions into the input-to-state transitions of ConvLSTM cells, DT-LSTM enables adaptive spatio-temporal modeling for tasks including human body emotion recognition in videos of arbitrary length (Tahghighi et al., 2020).

1. Architectural Principles of DT-LSTM

DT-LSTM modifies the standard ConvLSTM cell by incorporating deformable convolutions (⊗) in place of fixed convolutions (∗) for input-to-state transformations. Given input tensor $X_t \in \mathbb{R}^{C \times H \times W}$, previous hidden state $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$, and previous cell state $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$, the cell computes:

$$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$

Here, $\sigma(\cdot)$ denotes the sigmoid function, $\tanh(\cdot)$ the hyperbolic tangent, and $\circ$ the Hadamard (elementwise) product. All state-to-state transitions utilize standard convolutions, whereas the input-to-state transitions for the input and candidate gates ($i_t$ and $g_t$) utilize deformable convolutions with offsets $\Delta p$ learned at each time step by a lightweight offset network (Tahghighi et al., 2020).

2. Deformable Convolution Formulation

The deformable convolution operation replaces fixed-grid sampling with spatially adaptive sampling. For a convolution weight kernel $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$0 over a standard receptive field $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$1 (e.g., $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$2), standard convolution computes:

$H_{t-1} \in \mathbb{R}^{C' \times H \times W}$3

In deformable convolution, the sampling locations are perturbed by learned offsets $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$4:

$H_{t-1} \in \mathbb{R}^{C' \times H \times W}$5

Off-grid sampling is handled by bilinear interpolation. The set of offsets $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$6 is produced via a learnable offset subnetwork $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$7:

$H_{t-1} \in \mathbb{R}^{C' \times H \times W}$8

where $H_{t-1} \in \mathbb{R}^{C' \times H \times W}$9 and $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$0 are the weights of a $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$1 convolution outputting $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$2 offset maps per input location (Tahghighi et al., 2020).

3. Offset Subnetwork and Gate-specific Adaptation

For each gate $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$3, the corresponding offset network produces gate-specific offsets per time step:

$C_{t-1} \in \mathbb{R}^{C' \times H \times W}$4

Each spatial location $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$5 thus possesses $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$6 learned 2D offsets. These offsets deform the regular grid $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$7 in the deformable convolution, yielding gate-adaptive spatial behavior. This architectural design enables the cell to locally adapt the spatial receptive fields of feature integration, increasing tolerance to geometric image transformation (Tahghighi et al., 2020).

4. Training Protocol and Initialization

DT-LSTM models are trained end-to-end with a standard cross-entropy loss over target classes. All convolutional kernels—including deformable and offset networks—utilize He-normal initialization. Offset biases are initialized to zero, ensuring initial behavior matches a standard convolution. $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$8 regularization (weight decay coefficient $C_{t-1} \in \mathbb{R}^{C' \times H \times W}$9) is applied for all learnable parameters. The Adam optimizer is employed with an initial learning rate of $$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$0, decayed by a factor of $$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$1 every 20 epochs, with early stopping based on validation accuracy (Tahghighi et al., 2020).

5. Experimental Configuration and Workflow

The GEMEP corpus (145 clips, 17 emotion categories) serves as the primary testbed. Video preprocessing includes:

• Uniform temporal sampling with jitter (T=32 frames per clip)
• Frame resizing (from $$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$2 to $$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$3)
• Data augmentation: random translations ($$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$4 pixels), rotations ($$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$5), brightness shifts, and Gaussian blur

This yields approximately 9,052 augmented video clips, partitioned 80%/20% for training/validation. The network architecture comprises:

1. 3D-CNN block ($$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$6D convolution + pooling) for short-term spatio-temporal features
2. Three DT-LSTM cells applied at frame indices $$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$7, propagating hidden states temporally
3. Shallow 2D CNN ($$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$8 conv + average pooling) over $$\begin{aligned} i_t &= \sigma\left(W_x^i \mathbin{\otimes}_{\Delta p_t^i} X_t + W_h^i * H_{t-1} + b^i\right) \ f_t &= \sigma\left(W_x^f * X_t + W_h^f * H_{t-1} + b^f\right) \ g_t &= \tanh\left(W_x^g \mathbin{\otimes}_{\Delta p_t^g} X_t + W_h^g * H_{t-1} + b^g\right) \ o_t &= \sigma\left(W_x^o * X_t + W_h^o * H_{t-1} + b^o\right) \ C_t &= f_t \circ C_{t-1} + i_t \circ g_t \ H_t &= o_t \circ \tanh(C_t) \end{aligned}$$9 for spatial feature extraction
4. Global average pooling (spatial and temporal) yielding a $\sigma(\cdot)$0 feature representation
5. Fully-connected layer with softmax activation for 17-way emotion classification

Pseudocode for the DT-LSTM cell and the end-to-end training loop is provided in the original work, specifying distinct offset computations per gate and detailed sequential data flow (Tahghighi et al., 2020).

6. Empirical Performance and Application

DT-LSTM achieves 98.8% accuracy for whole human body emotion recognition on the GEMEP dataset validation set, establishing state-of-the-art performance within the experimental context. The framework is tailored for robust emotion analysis from body pose and motion, directly addressing the challenge of intra-class variability due to pose and geometric deformation. The deformation-aware mechanism enables the model to generalize across diverse visual appearances and movement styles, a key advantage over prior approaches focusing only on static or facial information (Tahghighi et al., 2020).

7. Implementation Highlights and Pseudocode

The following table summarizes principal computational blocks as instantiated in the DT-LSTM approach:

Block	Operation Type	Distinguishing Feature
Offset Subnet	2D Convolution	Outputs $\sigma(\cdot)$1 offsets per spatial position
Input-to-state	Deformable Convolution	Per-gate, per-timestep learned offsets
State-to-state	Standard 2D Convolution	Fixed convolutional neighborhoods

Training pseudocode:

$\sigma(\cdot)$3

The DefConvLSTMCell operation involves offset generation (per gate, via $\sigma(\cdot)$2) and deformable convolution for the input-to-state connections, with state updates as defined by the equations above.

---