---
title: ConvLSTM with Attention
url: https://www.emergentmind.com/topics/convlstm-with-attention
type: topic
---

# ConvLSTM with Attention

ConvLSTM with attention denotes a family of architectures that combine convolutional recurrent state updates with attention-based selection or reweighting. In the narrow sense, it refers to models that insert attention directly into the ConvLSTM transition so that the input-to-state or state-to-state mappings are no longer purely local convolutions. In a broader usage, it also includes systems in which ConvLSTM remains standard but attention reweights its inputs, hidden states, scales, spectral bands, or decoder features. The literature is therefore heterogeneous: some models are true attention-augmented ConvLSTM cells, some are ConvLSTM-centered pipelines with external attention modules, some generalize ConvLSTM to non-chain topologies, and some papers labeled “deep ConvLSTM with attention” are in fact CNN–LSTM stacks followed by attention rather than canonical ConvLSTM cells [2010.09662][1904.09405][2005.00698].

## 1. Core formulation and terminological scope

A canonical ConvLSTM replaces the affine maps of an LSTM with convolutions so that hidden state and memory retain spatial structure. One standard formulation used in the literature is
$$
\begin{aligned}
i_t &= \sigma(W_{xi} * \mathcal{X}_t + W_{hi} * \mathcal{H}_{t-1} + W_{ci} \circ \mathcal{C}_{t-1} + b_i),\\
f_t &= \sigma(W_{xf} * \mathcal{X}_t + W_{hf} * \mathcal{H}_{t-1} + W_{cf} \circ \mathcal{C}_{t-1} + b_f),\\
\mathcal{C}_t &= f_t \circ \mathcal{C}_{t-1} + i_t \circ \tanh(W_{xc} * \mathcal{X}_t + W_{hc} * \mathcal{H}_{t-1} + b_c),\\
o_t &= \sigma(W_{xo} * \mathcal{X}_t + W_{ho} * \mathcal{H}_{t-1} + W_{co} \circ \mathcal{C}_t + b_o),\\
\mathcal{H}_t &= o_t \circ \tanh(\mathcal{C}_t).
\end{aligned}
$$
This baseline is the reference point for many attention-augmented variants [2010.09662].

Attention can intervene at several points in this recurrence. It may alter the current input before it reaches the gates, alter the recurrent contribution from previous hidden states, act after the recurrent update as a spatial or channel refinement, or operate on a separate sequence of compact embeddings extracted from the recurrent states. The resulting models preserve the defining recurrent memory semantics of ConvLSTM to different degrees.

The term is not used uniformly. One important case study is the wearable-sensing model described as a “deep ConvLSTM with self-attention,” whose actual structure is convolutional embedding over sensor channels, a standard LSTM encoder over time, and an additive self-attention layer over encoder outputs. That model does not present ConvLSTM gate equations and does not insert convolution inside LSTM gates; accordingly, it is best understood as a CNN–LSTM–attention pipeline rather than a canonical ConvLSTM cell [2005.00698]. This terminological slippage is recurrent across application areas and materially affects reproduction.

## 2. Architectural patterns of combination

The reported literature falls into several recurring patterns.

| Pattern | Representative systems | Mechanism |
|---|---|---|
| In-cell attention augmentation | TAAConvLSTM, SAAConvLSTM, FACLSTM, flow-guided multi-kernel ConvLSTM | Attention modifies recurrent or input transitions |
| External attention around ConvLSTM | AttTreeCLSTM, A3CLNN, ACLAE-DT, FAConvLSTM | Attention reweights ConvLSTM features, scales, or embeddings |
| ConvLSTM generalization or replacement | Tree-structured ConvLSTM, ConvTransformer, CNN–LSTM+self-attention HAR | Recurrence is re-topologized, replaced, or only loosely related |

In the most integrated designs, attention changes the transition operator itself. TAAConvLSTM replaces the standard state-to-state convolution with a temporal attention augmented operator over past hidden states, whereas SAAConvLSTM replaces the input-to-state convolution with a self-attention augmented operator on the current frame. FACLSTM computes a state-conditioned spatial attention map over 2D encoder features and feeds the attention-weighted tensor $\hat{x}_t = attn_t \circ x$ into a ConvLSTM decoder. The multi-kernel video-recognition model instead applies kernel-specific sigmoid masks, derived from optical flow, to the current input before each branch’s input-to-hidden convolution, leaving the hidden-to-hidden path unchanged [2010.09662][1904.09405][1908.08990].

A second class leaves the recurrent cell standard and places attention around it. In the coronary-artery framework, an attention FCN extracts or refines node-level features and a tree-structured ConvLSTM propagates contextual information across the anatomical tree; attention is not inserted into the ConvLSTM gates. In the multisource remote-sensing model, ConvLSTM2D and ConvLSTM3D extract LiDAR and HSI features, after which spatial, spectral, and multiscale attention modules reweight those features and guide three-level fusion. In the anomaly-detection autoencoder, Bahdanau additive attention aggregates informative hidden states across time. In FAConvLSTM, the recurrent core remains ConvLSTM-like, but hidden states receive sparse axial spatial attention, mixed features receive squeeze-and-excitation channel attention, and compact timestep embeddings receive temporal multi-head self-attention [1902.10053][2204.04462][2201.09172][2601.10914].

A third pattern treats attention as an alternative to ConvLSTM rather than an augmentation of it. ConvTransformer replaces recurrent state propagation with multi-head convolutional self-attention over frame features, arguing that attention learns long-range sequential dependence more directly and in parallel. The broader implication is that “ConvLSTM with attention” and “attention instead of ConvLSTM” form adjacent but distinct research programs [2011.10185].

## 3. Attention axes and structural priors

The attention mechanism paired with ConvLSTM is usually matched to the latent structure of the task.

Spatial attention is the most direct form. FACLSTM computes 2D attention maps from the previous cell and hidden states together with encoder feature maps and character center masks, then reweights the feature tensor before ConvLSTM decoding. SaAB in the LiDAR branch of A3CLNN learns a spatial attention map $\alpha^{L}_{Sa}$ and applies $\hat{X}^{L}_{l} = X^{L}_{l} \odot \alpha^{L}_{Sa}$. FAConvLSTM adds axial spatial self-attention to the hidden state only every $K$ timesteps, sequentially along row and column dimensions, as a cheap mechanism for long-range spatial structure [1904.09405][2204.04462][2601.10914].

Temporal attention appears in two distinct forms. In TAAConvLSTM, attention is part of the recurrent state transition and attends from the current hidden representation to a bank of older hidden states over an attention horizon $H_a$. In ACLAE-DT, temporal attention is Bahdanau-style additive attention over encoder hidden states,
$$
c^{t} = \sum_{t'=1}^{T_x} \alpha^{t,t'} h^{t'},
\qquad
\alpha^{t,t'} = \frac{\exp(u^{t,t'})}{\sum_{k=1}^{T_x} \exp(u^{t,k})},
$$
with alignment score $u^{t,t'} = a(s^{t-1}, h^{t'})$. In the wearable HAR model, the attention is additive self-attention over time-indexed LSTM states rather than over spatial maps, which again illustrates the breadth of the label “ConvLSTM with attention” [2010.09662][2201.09172][2005.00698].

Spectral and multiscale attention are prominent where the input has non-image axes worth treating as structured sequences. In A3CLNN, SeAB decomposes an HSI tensor along the spectral dimension, processes the resulting sequence with ConvLSTM2D layers, and produces a spectral attention vector $\alpha^{H}_{Se}$ used in $\hat{X}^{H}_{l} = X^{H}_{l} \odot \alpha^{H}_{Se}$. MSRAB constructs multiscale ConvLSTM features with $1\times1$, $3\times3$, and $5\times5$ receptive fields, concatenates them in the time-step dimension, learns a softmax weighting over scales, and then fuses the attended multiscale sequence with another ConvLSTM and a residual connection. This suggests that ConvLSTM can model dependencies over spectral or scale axes even when the native task is not temporal in the ordinary sense [2204.04462].

Topology-aware attention introduces a different prior: structural correctness. In EM segmentation, the TopologyAttention ConvLSTM Network proposes a Spatial Topology-Attention module that processes a 3D image as a stack of 2D slices and uses ConvLSTM to leverage contextual structure information from adjacent slices; it also proposes an Iterative-Topology Attention module to provide a more stable topology-critical map for segmentation [2202.03430]. A related but distinct structural prior appears in tree-structured ConvLSTM, where the recurrence itself is defined over a tree rather than a chain:
$$
\mathcal{H}_j' = \sum_{l \in \mathcal{N}(j)} \mathcal{H}_l,
\qquad
\mathcal{C}_j = \sum_{l \in \mathcal{N}(j)} f_{jl} \odot \mathcal{C}_l + i_j \odot \mathcal{M}_j.
$$
There the attention module is feature-level spatial-and-channel weighting in the FCN, while the ConvLSTM handles inter-node propagation over the topology [1902.10053].

## 4. Application domains and domain-specific adaptations

In autonomous-driving environment prediction, the input is not RGB video but bird’s-eye-view Dempster–Shafer Theory occupancy grids of size $128 \times 128$ with free-space and occupied-space belief masses. The target difficulty is long-horizon prediction without blurring, loss of static structure, or vanishing of moving objects. TAAConvLSTM and SAAConvLSTM are embedded in a 4-layer PredNet and modify only selected upper layers because attention cost scales strongly with spatial extent [2010.09662].

In biomedical image analysis, ConvLSTM with attention appears in at least two structurally different roles. In coronary-artery segmentation, attention-driven tree-structured ConvLSTM operates on trees of $41 \times 41$ cross-sectional patches derived from centerlines, so the central issue is not temporal order but branch topology. In EM segmentation, ConvLSTM is used to transfer structure information across adjacent slices in anisotropic volumes, with topology-critical maps intended to preserve fine structural correctness [1902.10053][2202.03430].

In scene text recognition, FACLSTM treats recognition as a spatiotemporal prediction problem over 2D feature maps rather than a sequence problem over flattened vectors. An encoder–decoder based on VGG-16 produces feature maps and character center masks, the attention-equipped ConvLSTM decodes up to $T=20$ steps, and the auxiliary masks are learned from character-level bounding boxes shrunk by ratio $r=0.25$. The attention mechanism is therefore explicitly supervised by spatial priors, a design not shared by most other ConvLSTM-with-attention systems [1904.09405].

In remote sensing, A3CLNN uses dual branches because HSI and LiDAR expose different discriminative axes. ConvLSTM3D is assigned to HSI to preserve spatial-spectral structure, ConvLSTM2D to LiDAR to preserve spatial structure, and attention is decomposed into spectral, spatial, multiscale, and cross-modal composite components. The model also adopts stepwise training with separate branch losses $Loss_H$ and $Loss_L$ and a fused loss $Loss_F$ [2204.04462].

In multivariate time-series anomaly detection, ConvLSTM with attention is used only after the data are converted into feature images $M^t \in \mathbb{R}^{(n+p)\times(n+p)}$ that encode pairwise inner products across local windows. ConvLSTM is thus applied to sequences of relationship matrices rather than to raw sensor vectors. The same paper couples Bahdanau attention to dynamic thresholding, where each feature-pair threshold is $\epsilon_{ij} = (\mu(e_{ij}) + z \sigma(e_{ij}))^{T}$ and $z$ is chosen in the range 2 to 5 [2201.09172].

Other domains show still broader usage. In wearable HAR, self-attention is attached to a CNN–LSTM stack and evaluated under split protocols such as SNOW, FNOW, and LOTO. In climate modeling, FAConvLSTM positions axial attention, SE, and temporal MHA as efficient refinements around a factorized ConvLSTM2D replacement. In video frame synthesis, ConvTransformer explicitly abandons recurrence in favor of attention, yet remains directly relevant because it is benchmarked against ConvLSTM-family methods and targets the same long-range video dependence problem [2005.00698][2601.10914][2011.10185].

## 5. Empirical evidence across tasks

Reported gains are task-dependent, but attention is frequently beneficial when the modeled dependence is long-range, structurally nonlocal, or multimodal.

| Setting | Baseline vs attention-enhanced model | Reported outcome |
|---|---|---|
| Occupancy prediction on KITTI/Waymo [2010.09662] | PredNet vs TAAConvLSTM/SAAConvLSTM | Best KITTI TAAConvLSTM: IS \(6.91 \pm 0.06\), MSE \(3.62 \times 10^{-2} \pm 0.0012\); best Waymo TAAConvLSTM: IS \(3.51 \pm 0.04\) |
| Coronary artery segmentation [1902.10053] | CLSTM / TreeCLSTM / AttTreeCLSTM | Total Dice \(0.8591 / 0.8678 / 0.8691\); bifurcation Dice \(0.8120 / 0.8438 / 0.8491\) |
| Scene text recognition [1904.09405] | FAFLSTM\_base2 vs FACLSTM | CUTE \(75.7 \rightarrow 83.3\); IIIT5K \(87.8 \rightarrow 90.5\); SVT \(78.2 \rightarrow 82.2\) |
| Wearable HAR under LOTO [2005.00698] | CNN–LSTM vs CNN–LSTM+self-attention | UTD-MHAD1 accuracy \(43.92 \pm 3.71 \rightarrow 58.02 \pm 2.29\) |
| CNC anomaly detection, longest window [2201.09172] | ACLAE-DT No-Attention vs Full | F1 \(0.98 \rightarrow 1.00\) |
| HSI+LiDAR classification [2204.04462] | SSCL3DNN(H+L) vs \(A^3\)CLNN(H+L) | Houston OA \(86.01 \rightarrow 90.55\); Trento OA \(96.46 \rightarrow 98.73\) |

The strongest claims often concern specific failure modes rather than uniform average gains. In occupancy prediction, the main assertion is that temporal attention better preserves moving objects and static environment structure over long rollouts; the paper explicitly emphasizes that the IS gap between TAAConvLSTM and baselines increases with prediction horizon, while PredRNN++ can obtain better MSE because of blurred predictions [2010.09662]. In tree-structured segmentation, the largest benefit of attention-enhanced tree recurrence occurs around bifurcation nodes rather than in the overall mean, which is consistent with the architectural focus on multi-branch topology [1902.10053].

Attention benefits are not always large or uniformly significant. In wearable HAR, statistically significant improvements are reported for UTD-MHAD1, USC-HAD, UTD-MHAD2, and WHARF, but not for MHEALTH or WISDM, where the authors attribute the lack of significance to those datasets already being relatively easy with performance above 90% [2005.00698]. In the multi-kernel video-recognition model, the principal gain comes from the multi-kernel recurrent architecture itself, while flow-guided attention adds only \(+0.18\%\) over simple multi-kernel on the reported UCF-101 attention ablation [1908.08990]. This suggests that “attention helps” is too coarse a summary; the dominant contribution may be recurrent redesign, feature representation, or evaluation protocol.

A parallel result is that attention can outperform ConvLSTM by replacing it rather than augmenting it. For video extrapolation, ConvTransformer improves over the ConvLSTM-family baseline MCNet by \(+1.9285\) dB PSNR and \(+0.0681\) SSIM for next-frame extrapolation, and by \(+2.1913\) dB PSNR and \(+0.1229\) SSIM for next-3-frame extrapolation on the reported averages [2011.10185]. This does not diminish ConvLSTM-with-attention work, but it shows that the comparison class increasingly includes fully attention-based temporal models.

## 6. Design trade-offs, recurring misconceptions, and open directions

A first recurring misconception is terminological. Some papers use “ConvLSTM with attention” to denote a true recurrent cell with attention-augmented state updates, while others use it for a ConvLSTM-centered pipeline with external attention, and others still use it for CNN–LSTM models that never instantiate a canonical ConvLSTM cell. The wearable HAR model is the clearest warning: it is a convolution + LSTM stack with additive self-attention over time, not a ConvLSTM recurrent cell [2005.00698]. Precise architectural reading is therefore necessary before comparing results or transferring code.

A second issue is placement. Attention inside the recurrent transition, as in TAAConvLSTM or SAAConvLSTM, directly changes memory dynamics. Attention outside the cell, as in AttTreeCLSTM, A3CLNN, or ACLAE-DT, instead refines features before or after recurrence. The two choices are not functionally equivalent. The literature also shows that placement may need to respect feature semantics: tree-structured ConvLSTM works best before the decoder rather than after Conv3\_2 or Conv4\_2, and A3CLNN deliberately assigns spectral attention to HSI and spatial attention to LiDAR [1902.10053][2204.04462].

A third issue is computational cost. Attention can substantially increase memory and runtime. TAAConv has memory complexity $O((HW)^2 N_h H_a)$ and SAAConv has $O((HW)^2 N_h)$, which is why the occupancy-prediction paper applies the augmented cells only in top PredNet layers. FAConvLSTM explicitly responds to this by factorizing gate computation, using axial attention rather than full 2D global attention, applying that attention only every $K$ steps, and restricting temporal MHA to compact subspace embeddings rather than dense hidden maps [2010.09662][2601.10914].

A fourth issue is supervision and side information. Some attention mechanisms depend on extra annotations or preprocessing. FACLSTM requires character-level bounding boxes to create character center masks when the focused-attention module is used. Tree-structured ConvLSTM relies on centerline extraction and a tree of frames. A3CLNN depends on paired HSI and LiDAR modalities and a stepwise training procedure. These requirements are not incidental engineering choices; they are part of the inductive bias that makes the attention useful in each domain [1904.09405][1902.10053][2204.04462].

Finally, topology-aware ConvLSTM remains a promising but incompletely exposed direction in the available record. For the EM-segmentation model, only high-level facts are explicit: the method was originally called TACNet and later renamed TACLNet; “topology-attention” was revised to “topological-attention”; the revision added an ablation study for the number of input slices in Table 3, added Dice score to Table 1, added experiments on the ISBI13 dataset, and changed the conclusion to emphasize anisotropic datasets [2202.03430]. This suggests that topology-critical information transfer across slices is being treated as a first-class objective alongside conventional segmentation accuracy, but the exact architectural and loss-level mechanisms remain underspecified in the presently available description.

Taken together, the literature shows that ConvLSTM with attention is not a single architecture but a design space. Its unifying principle is the retention of convolutional recurrent memory together with an additional mechanism for selective routing, reweighting, or long-range aggregation. The key technical question is less whether attention is present than where it operates: on the current input, on past hidden states, on external feature tensors, on scale or spectral axes, on compact summaries, or on topological neighborhoods. That choice largely determines what kind of dependence the model can express and which empirical failure modes it is designed to correct.

Source: https://www.emergentmind.com/topics/convlstm-with-attention