CNN-LSTM Autoencoder: Architectures & Applications

• CNN-LSTM autoencoders are encoder–decoder networks that integrate CNNs for spatial feature extraction with LSTMs for modeling temporal or sequential dependencies.
• They offer versatile architectures—from reconstructing feature sequences and images to learning joint multimodal representations—tailored for tasks like video anomaly detection and image restoration.
• The models achieve robust performance by optimizing reconstruction loss and latent semantics, with empirical results demonstrating high accuracy in driver drowsiness, image denoising, and news clustering.

Searching arXiv for recent and foundational papers relevant to CNN-LSTM autoencoders. A CNN-LSTM autoencoder is an encoder–decoder architecture that combines convolutional feature extraction with Long Short-Term Memory sequence modeling so that spatial or frame-level structure is encoded by a CNN and temporal, sequential, or multimodal dependencies are reconstructed through an LSTM-coupled decoder. In the cited arXiv literature, this pattern appears in video anomaly detection, image restoration under severe corruption, and multimodal variational representation learning, with reconstruction error or latent structure serving as the central organizing principle (Tüfekci et al., 2022, Haque et al., 2018, Shi et al., 2024).

1. Architectural configurations

In the cited literature, the label “CNN-LSTM autoencoder” covers several distinct encoder–decoder organizations rather than a single canonical blueprint. One configuration extracts per-frame visual embeddings with a CNN and reconstructs the resulting feature sequence with an LSTM autoencoder; another uses a CNN encoder and a multilayer LSTM decoder to generate an image sequentially; a third fuses CNN image encodings and LSTM text encodings into a shared variational latent space (Tüfekci et al., 2022, Haque et al., 2018, Shi et al., 2024).

Configuration	CNN role	LSTM role
Feature-sequence autoencoder	Fine-tuned ResNet-34 maps each $224\times224$ frame to a 512-dimensional feature vector	A 2-layer LSTM encoder and 2-layer LSTM decoder reconstruct the clip feature sequence in reverse order
CNN encoder + LSTM decoder	Two convolutional layers with max-pooling compress a corrupted $28\times28$ image into a 100-dimensional latent vector	A 5-layer LSTM reconstructs the image row by row using the latent vector, current corrupted row, and previous output
Convolutional-recurrent variational autoencoder	CNN encodes video frames resized to $200 \times 120 \times 3$	A bidirectional 2-layer LSTM encodes text embeddings; the decoder reconstructs both image and text embeddings from a shared latent code

In the driver-drowsiness formulation, the CNN backbone is explicitly defined as

$$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$

so each frame $I(t)$ becomes

$F(t) = B(I(t)).$

Those features are L2-normalized and passed to the LSTM autoencoder, which minimizes the clip-level reconstruction objective

$$L_{con} = \sum_{t=1}^{N} \left\lVert F(t) - \hat{I}(N-t+1) \right\rVert^2.$$

In the image-restoration formulation, the encoder compresses the corrupted image into a latent “thought vector” $v$, and the decoder uses that vector together with the corrupted image to reconstruct the clean image: $$v=f_{encoder}\left({image}_{corrupted}\right), \qquad {image}_{cleaned}=f_{decoder}\left(v+{image}_{corrupted}\right).$$ The paper describes “direct attention” not as a separate soft-attention module but as direct access by the decoder to the corrupted image during reconstruction.

In the multimodal CRVAE formulation, the CNN and LSTM branches are concatenated after batch normalization into a 14,048-dimensional joint vector, reduced to 4,000 neurons and then to a final 2,000-neuron latent layer. That final latent layer is split into 1,000 units for the latent mean and 1,000 units for the latent log standard deviation, making the model variational rather than purely deterministic.

2. Reconstruction targets and latent objectives

A defining technical distinction across CNN-LSTM autoencoders is the object being reconstructed. In the driver-drowsiness model, the system does not reconstruct raw pixels; it reconstructs the CNN feature sequence extracted from a short clip, so the autoencoder operates in a learned 512-dimensional representation space rather than image space (Tüfekci et al., 2022). In the image-restoration model, the decoder reconstructs the cleaned image itself from a latent vector plus the corrupted input, and the output is generated row by row (Haque et al., 2018). In the CRVAE model, reconstruction is multimodal: both the image tensor and the text embedding tensor are reconstructed from a shared latent variable (Shi et al., 2024).

These differences induce different latent semantics. In the feature-sequence setting, the latent state is a compact representation of normal temporal regularities over a clip. In the image-restoration setting, the latent vector captures global digit identity and supports sequential generation of missing rows. In the CRVAE setting, the latent code is explicitly probabilistic and is intended to capture shared semantics across a paired frame and caption.

The CRVAE loss is a weighted multimodal reconstruction objective,

$$L = \text{ImageLoss} + \lambda \cdot \text{TextLoss},$$

with $28\times28$0. The image branch uses pixel-wise mean squared error,

$28\times28$1

and the text branch reconstructs continuous embedding tensors with

$28\times28$2

The paper states that the encoder produces $28\times28$3 and $28\times28$4, samples from a Normal distribution over the latent space during training, and then uses the latent means $28\times28$5 as the final outputs for clustering after training.

The image-restoration paper reports mean squared loss with the written form

$28\times28$6

and also applies L2 regularization,

$28\times28$7

The authors later note that the use of RMSE encourages smoother predictions, which they use to explain the smoother reconstructed digit strokes.

3. Anomaly detection as a one-class reconstruction problem

One important operational regime for CNN-LSTM autoencoders is one-class anomaly detection. In the driver-drowsiness study, only clips labeled as normal are used during training, while both normal and drowsy clips are evaluated at test time; the underlying premise is that drowsy driving should yield higher reconstruction loss because it departs from the learned normal temporal manifold (Tüfekci et al., 2022). The decision rule is explicit: low reconstruction loss indicates likely normal behavior, and high reconstruction loss indicates likely anomalous or drowsy behavior. The threshold is selected to maximize ROC AUC on the test set.

A notable methodological contribution in that work is clip-confidence assignment. Normal clips are labeled according to thresholds of $28\times28$8, $28\times28$9, or $200 \times 120 \times 3$0 of frames being normal, corresponding to low-confident, medium-confident, and high-confident normal clips. The paper states a trade-off: low-confidence training improves the ability to catch anomalies and tends to improve recall, but can increase false positives; high-confidence training improves reliability of positive predictions and tends to improve precision, but can reduce recall. The contrast between Model$200 \times 120 \times 3$1 and Model$200 \times 120 \times 3$2 is used to explain shifts in anomaly-score distributions.

The same basic anomaly-detection logic also appears in the LSTM-only insider-threat model LAC, where each employee’s day-wise time-stamped activity sequence is reconstructed and anomalous employees are identified by significant reconstruction loss (Paul et al., 2020). That paper does not use CNNs, but it illustrates the same broader autoencoding principle: learn routine sequential structure, then use poor reconstruction as the anomaly signal. A plausible implication is that the anomaly-detection interpretation of CNN-LSTM autoencoders is strongest when the sequential component encodes routine rather than class labels.

4. Application domains and reported empirical behavior

The cited literature uses CNN-LSTM autoencoder designs in substantially different domains, and the empirical behavior depends on what is being reconstructed.

Domain	Representative setup	Reported outcome
Driver drowsiness detection	ResNet-34 feature extractor + 2-layer LSTM autoencoder on NTHU-DDD	$200 \times 120 \times 3$3, accuracy $200 \times 120 \times 3$4
Image denoising and restoration	CNN encoder + 5-layer LSTM decoder with direct attention on severely corrupted MNIST	Performed better at generating missing part of the images than convolutional autoencoder
Multimodal news clustering	CRVAE with CNN image branch and LSTM text branch	On COVID-19 data, image loss $200 \times 120 \times 3$5 (EN) and $200 \times 120 \times 3$6 (CN)

For driver drowsiness, the experiments are conducted on NTHU Driver Drowsiness Dataset, which contains 36 subjects, infrared-camera videos at 30 fps, and drowsy behaviors including nodding, yawning, sleepiness, and slow blinking. Clips are created with 48 frames, sampling rate 2, window stride 23, and each clip is about 3 seconds; about 7000 test clips are obtained, with roughly 52% anomalous. Because the official test set is unavailable, the published training set of 18 subjects is split into 12 subjects for training and 6 subjects for validation, while the published validation set of 4 subjects is used as the test set. The method is benchmarked against the state-of-the-art anomaly detection method from Kopuklu et al.; on NTHU-DDD, “Ours” reaches AUC $200 \times 120 \times 3$7 and accuracy $200 \times 120 \times 3$8, compared with $200 \times 120 \times 3$9 and $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$0 after post-processing for the compared method (Tüfekci et al., 2022).

For image restoration, the corruption process is intentionally severe: the lower half of each $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$1 MNIST image is made blank or black and then salt-and-pepper noise or white noise is added. The main baseline is a CNN-CNN encoder-decoder. The reported qualitative outcome is that the CNN-CNN model removes noise well but struggles to generate the missing lower half accurately, whereas the CNN-LSTM model removes noise and reconstructs the missing digit portion much more accurately, producing smoother and cleaner digit strokes (Haque et al., 2018).

For multimodal news analysis, CRVAE is trained on paired <frame, caption> samples from COVID-19 and Winter Olympics 2022 videos. The latent means are clustered with K-means to produce thematic groupings, and the paper reports that reconstructed images are clearer than those from CVAE-only models. On COVID-19 data, the reported image losses are 0.3645 and 0.1461 for Pure CVAE, 0.1653 and 0.0955 for Dense CVAE, and 0.0408 and 0.0323 for CRVAE on English and Chinese videos respectively (Shi et al., 2024).

This suggests that CNN-LSTM autoencoders are not tied to a single output modality. They can reconstruct feature trajectories, pixel arrays, or paired image–text representations, provided that convolution supplies local or frame-level abstraction and recurrence supplies sequential or cross-modal structure.

5. Preprocessing, optimization, and evaluation practice

The training pipeline for CNN-LSTM autoencoders is highly application-specific. In the driver-drowsiness model, frames are resized to $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$2, random horizontal flip is applied, normalization uses mean and deviation for ResNet-34 fine-tuning, and CLAHE is used to improve contrast, especially for night-condition images. Several CLAHE settings are tested: no CLAHE, limit 10 grid 8, limit 5 grid 8, and limit 5 grid 16; the best setting is CLAHE limit $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$3, grid size $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$4, with the reported best AUC in the CLAHE study. Fine-tuning of ResNet-34 uses learning rate $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$5, batch size $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$6, Adam, and weight decay $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$7. The LSTM autoencoder uses learning rate $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$8, batch size $$B(\cdot) : \mathbb{R}^{224\times 224 \times 3} \rightarrow \mathbb{R}^{512},$$9, and SGD (Tüfekci et al., 2022).

The image-restoration study uses TensorFlow on an Nvidia GTX 1080, with 500,000 iterations, batch size 100, and dropout 25%. The paper states that both Adadelta and Adam were used, and that Adam performed better than Adadelta. Convolutional layers use ReLU, the final fully connected decoder layer uses tanh, and the final output layer uses sigmoid because pixel values are normalized to $I(t)$0 (Haque et al., 2018).

The CRVAE study trains with image size $I(t)$1, batch size 16, AdamW optimizer, learning rate $I(t)$2, gradient clipping at 0.01, ReLU activations throughout, and 500 epochs on one NVIDIA RTX 3080 GPU with 10 GB memory. Training is reported to take around 30 minutes per video (Shi et al., 2024).

Evaluation practice also varies. The driver-drowsiness paper reports AUC under the ROC curve as the main metric and also reports accuracy, recall, precision, and F1, while emphasizing that AUC is the most reliable summary because accuracy alone is threshold-dependent and can hide false positive and false negative trade-offs. The image-restoration paper relies on loss curves and qualitative visual comparison rather than PSNR or SSIM. The CRVAE pipeline evaluates autoencoder performance with reconstruction losses and downstream clustering with average inter-cluster distance, average cross-cluster distance, and cluster robustness under centroid changes. This distribution of evaluation methods reflects the fact that CNN-LSTM autoencoders are used both as end-task predictors and as representation-learning modules.

6. Scope, boundary cases, and common confusions

A common confusion is to treat any CNN-LSTM system as a CNN-LSTM autoencoder. The cited literature draws a sharper boundary. The visual encryption and compression model is a CNN-based autoencoder with convolution, pooling, upsampling, and fully connected layers, but it contains no LSTM or recurrent component; it is therefore a CNN autoencoder reference rather than a CNN-LSTM autoencoder (Madani et al., 1 Apr 2025). Conversely, the insider-threat model LAC and the cyberbullying model based on a trustable LSTM-Autoencoder are autoencoders built around LSTM sequence reconstruction, but they do not include CNN layers in the proposed architecture (Paul et al., 2020, Akter et al., 2023).

A second confusion is to equate encoder–decoder structure with autoencoding. The image-captioning system based on VGG16 or ResNet50 plus an LSTM decoder is a CNN-LSTM encoder–decoder, but it is trained to predict caption tokens rather than reconstruct its own input, so it is not an autoencoder in the usual sense (Rampal et al., 2020). The CAZyme classifier based on CNN-LSTM with attention is likewise a sequence classifier; the same paper introduces a variational graph autoencoder, but that autoencoder is separate from the CNN-LSTM pipeline rather than integrated into it (Shi et al., 2022).

A third boundary case is the multi-speaker source-separation model, which uses an encoder–decoder CNN and LSTM in parallel and is described as an autoencoder-style separator. Its output semantics, however, are deep clustering embeddings and speaker masks rather than direct waveform reconstruction. The paper explicitly states that the network is best understood as taking a mixture spectrogram as input and outputting a structured latent representation whose final layer is interpreted as separation masks, not direct waveforms (Zegers et al., 2019).

Taken together, these cases clarify the scope of the term. A CNN-LSTM autoencoder, in the strict sense supported by the cited papers, is not merely any CNN-LSTM model, nor any autoencoder involving either CNNs or LSTMs. It is an encoder–decoder system in which convolutional processing and LSTM-based sequence modeling are both structurally integral to the learned reconstruction pathway.