Convolutional Autoencoder with Residual Connections

• CAER is a framework combining convolutional encoder–decoder architectures with explicit residual connections to preserve image details during restoration.
• It employs diverse skip mechanisms, including mirrored additive and concatenative strategies, to bypass bottlenecks and boost performance in tasks like denoising and inpainting.
• Different implementations, from optimization-derived residual updates to symmetric autoencoder designs, demonstrate CAER’s adaptability in maintaining structural fidelity.

Convolutional Autoencoder with Residual Connections (CAER) is an umbrella description for convolutional encoder–decoder restoration networks in which information bypasses the bottleneck through shortcut, skip, or residual mappings. In the literature, this framing encompasses symmetric convolutional autoencoders with mirrored encoder–decoder skips, residual encoder–decoder CNNs that learn image-domain correction terms, wavelet-conditioned autoencoders with concatenative shortcuts, and unrolled sparse-coding autoencoders whose encoder dynamics themselves have residual form. Across these variants, the common purpose of the residual pathway is to preserve structure that would otherwise be attenuated by compression, downsampling, or iterative approximation, while supporting supervised restoration objectives such as MSE, PSNR, and SSIM (Suganuma et al., 2018, Chen et al., 2017, Liang et al., 2023, Tolooshams et al., 2019).

1. Conceptual scope and representative formulations

The CAER label does not identify a single canonical network in these works. Rather, it denotes a family of restoration models built from a convolutional encoder–decoder backbone plus explicit residual information flow. In "Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search," the resulting models are called E-CAE, but they fit naturally into a CAER framing because the skip connections are used in a residual, encoder-to-decoder symmetric way (Suganuma et al., 2018). In "Low-Dose CT with a Residual Encoder-Decoder Convolutional Neural Network (RED-CNN)," the network is explicitly a residual encoder–decoder CNN for supervised LDCT-to-NDCT mapping (Chen et al., 2017). In "ResWCAE: Biometric Pattern Image Denoising Using Residual Wavelet-Conditioned Autoencoder," the model is a conditional residual autoencoder using both spatial and wavelet-domain information (Liang et al., 2023). In "Deep Residual Autoencoders for Expectation Maximization-inspired Dictionary Learning," the encoder can be interpreted as a recurrent neural network or as a deep residual network derived from FISTA unrolling (Tolooshams et al., 2019).

Model	Encoder–decoder organization	Residual mechanism
E-CAE	Symmetric CAE; decoder mirrored from encoder	Element-wise addition from encoder to mirrored decoder layer
RED-CNN	5 convolutional + 5 deconvolutional layers	Shortcut connections and residual mapping
Res-WCAE	Image encoder + wavelet encoder + decoder	Concatenation of encoder features into decoder stages
CRsAE	Unrolled FISTA encoder + tied linear decoder	Residual correction inside iterative updates

A central consequence of this heterogeneity is that “residual connection” is not a single implementation pattern. Depending on the model, it may mean additive cross-hourglass fusion, global residual prediction, concatenative encoder–decoder coupling, or residual correction within an optimization-inspired recurrence.

2. Architectural organization

A recurrent architectural theme in CAER-like systems is symmetry between analysis and synthesis paths. E-CAE restricts the search space to CAEs whose encoder half is mirrored by the decoder half. The encoder is represented as a sequence of convolutional layers, each followed by ReLU, with optional skip connections and, depending on the task, optional downsampling. The decoder is then automatically constructed to be symmetric to the encoder, and a final output layer is added to produce one filter for grayscale or three filters for color, both of size $3\times 3$. The paper emphasizes that specification of the encoder part of a CAE solely determines its entire architecture. For inpainting, the search space is further constrained so that a layer uses either skip connection or downsampling, but not both, and the network has a symmetric hourglass form. For denoising, downsampling is omitted and each convolution is zero-padded so spatial dimensions are preserved (Suganuma et al., 2018).

RED-CNN adopts a stricter paired design. Its architecture is symmetrical, with 5 convolutional layers in the encoder and 5 deconvolutional layers in the decoder, for a total of 10 layers. The pairing follows a FILO rule: the first convolutional layer corresponds to the last deconvolutional layer, and the last convolutional layer corresponds to the first deconvolutional layer. The network uses 96 filters for all layers except the last layer, which has 1, kernel size $5\times 5$ for all layers, stride 1, no padding, and no pooling. The omission of pooling is deliberate because pooling would destroy image details critical for CT diagnosis (Chen et al., 2017).

Res-WCAE generalizes the single-encoder CAER template to a dual-branch encoder. The image encoder processes the noisy grayscale fingerprint image $\bm{J}(M,N)$ through four downsampling convolution layers: $3\times 3$ convolution with 32 filters, stride 2, ReLU; $3\times 3$ convolution with 64 filters, stride 2, ReLU; $3\times 3$ convolution with 128 filters, stride 2, ReLU; and $3\times 3$ convolution with 256 filters, stride 2, ReLU. In parallel, the input is decomposed using a level-3 Symlets 2D wavelet transform, and the resulting subimages are passed through a 3-layer CNN with 16, 32, and 64 filters, each followed by ReLU. The bottleneck concatenates the compressed image representation with the compressed wavelet representation, and the decoder reconstructs the denoised fingerprint image using transpose convolutions with 128, 64, 32, and 1 filters, with sigmoid at the final layer; the paper also states that the final decoder layer uses bilinear interpolation to upsample feature maps in the decoder blocks (Liang et al., 2023).

CRsAE departs from heuristic encoder design and derives the CAER structure from the convolutional generative model $y = Hx + v$. Its encoder is obtained by unrolling $T$ iterations of FISTA for the CSC objective $$\min_x \frac{1}{2\sigma^2}\|y-Hx\|_2^2+\lambda\|x\|_1$$, while the decoder is a tied linear reconstruction layer $5\times 5$0. The encoder uses $5\times 5$1 in its gradient step, and the decoder uses the same $5\times 5$2 for synthesis, making it a tied-weight autoencoder with an explicit dictionary-learning interpretation (Tolooshams et al., 2019).

3. Residual pathways and information preservation

The residual mechanism is the defining feature of CAER, but the literature implements it in materially different ways. E-CAE uses mirrored additive skips across the hourglass. If a layer in the encoder has a skip connection, then the output feature maps obtained after ReLU of the encoder layer are passed to and are element-wise added to the output feature maps obtained before ReLU of the counterpart decoder layer. This is a residual-style connection across the hourglass, and the authors note that these skip connections are used only as standard components, with no special gates or learned fusion operators. In their architectural analysis, almost all of the best networks have a skip connection at the first layer, which they interpret as evidence that the input image itself contains crucial information for accurate restoration (Suganuma et al., 2018).

RED-CNN combines local shortcut connections with explicit residual learning at the restoration target. If $5\times 5$3 is the input image or patch and $5\times 5$4 is the output, the residual mapping is

$5\times 5$5

and reconstruction from the residual is

$5\times 5$6

The network therefore learns the residual between LDCT and NDCT rather than directly learning the full mapping. In this formulation, shortcut connections are used both to ease optimization and to preserve structural details (Chen et al., 2017).

Res-WCAE uses a different residual semantics. The output of each downsampling block in the image encoder is concatenated with the output of the corresponding upsampling block in the decoder. This is not an addition-based ResNet residual; it is a concatenation-based encoder–decoder skip connection, similar in spirit to U-Net/CAER designs. The stated purpose is preservation of fine-grained ridge structure, edge sharpness, minutiae-like structures, and precise spatial alignment in fingerprint images (Liang et al., 2023).

CRsAE internalizes residual structure within the encoder dynamics. Its FISTA recurrence includes the extrapolated variable

$5\times 5$7

followed by the residual correction

$5\times 5$8

and the shrinkage update $5\times 5$9, where the two-sided thresholding operator is

$\bm{J}(M,N)$0

Here the identity path is the carried-forward iterate, the correction is the gradient step, and the nonlinearity is soft-thresholding expressed through two-sided ReLU (Tolooshams et al., 2019).

A common misconception is that residual connections in autoencoders are invariably addition-based and ResNet-like. These papers show a broader taxonomy: additive mirrored skips, concatenative encoder–decoder fusion, global residual prediction, and optimization-derived residual correction all satisfy the CAER intuition while imposing different inductive biases.

4. Objectives, search procedures, and optimization

CAER-like models in this literature are trained for supervised restoration rather than generic self-reconstruction, but the optimization mechanisms differ sharply. E-CAE couples a standard $\bm{J}(M,N)$1 reconstruction objective with neural architecture search. The training loss inside each child model is

$\bm{J}(M,N)$2

where $\bm{J}(M,N)$3 is the clean ground truth, $\bm{J}(M,N)$4 is the corrupted input, $\bm{J}(M,N)$5 is the training set, and $\bm{J}(M,N)$6 is the CAE with parameters $\bm{J}(M,N)$7. Search is performed with a simple $\bm{J}(M,N)$8 evolutionary strategy on a directed acyclic graph phenotype over an $\bm{J}(M,N)$9 grid. Each node is encoded by layer type $3\times 3$0 and connection $3\times 3$1, the architecture is defined by a unique path from input to output, and only the nodes on this path are functioning nodes. The fitness is validation PSNR. During search, each child is trained for $3\times 3$2 iterations with mini-batch size $3\times 3$3 and learning rate $3\times 3$4 using ADAM; after search, the best architecture is fine-tuned for an additional 500k iterations, with the learning rate reduced by a factor of 10 at 200k and 400k iterations. The implementation uses PyTorch on four P100 GPUs, with $3\times 3$5, $3\times 3$6, $3\times 3$7, $3\times 3$8, $3\times 3$9, level-back $3\times 3$0, and candidate layers with $3\times 3$1 and $3\times 3$2 (Suganuma et al., 2018).

RED-CNN uses a conventional supervised MSE objective on paired LDCT/NDCT patches,

$3\times 3$3

optimized with Adam. Because full CT images are large and training data are limited, training is patch-based: patch size $3\times 3$4, sliding interval 4 pixels, and around $3\times 3$5 training patch pairs after extraction and augmentation. The augmentation includes rotation by 45 degrees, horizontal and vertical flipping, and scaling by factors 2 and 0.5. The reported implementation uses Caffe, GTX 1080 GPU hardware, and a learning rate of $3\times 3$6 slowly decreased to $3\times 3$7 (Chen et al., 2017).

Res-WCAE adds a regularized objective to the reconstruction loss. The paper uses an $3\times 3$8 error term plus a KL divergence regularizer, written in intended form as

$3\times 3$9

The stated role of the KLD term is to improve generalization and avoid overfitting to a single instance. All models are trained from scratch with mini-batch size 32, learning rate 0.001, and maximum iterations 200 (Liang et al., 2023).

CRsAE organizes training around an approximate EM interpretation of convolutional dictionary learning. The encoder computes the MAP sparse code, the dictionary update is implemented through the reconstruction loss

$3\times 3$0

and the regularization parameter is updated through

$3\times 3$1

Training therefore proceeds in two stages: backpropagation through the whole autoencoder to update $3\times 3$2, and backpropagation through the encoder only to update $3\times 3$3, followed by filter normalization $3\times 3$4 (Tolooshams et al., 2019).

5. Application domains and empirical results

The strongest quantitative case for plain CAER-style restoration is given by E-CAE. The model is evaluated on inpainting and denoising. For inpainting, CelebA, Stanford Cars, and SVHN are resized to $3\times 3$5, with center block masks, random pixel masks with 80% pixels missing, and half-image masks. On CelebA, the paper reports 29.9 dB for the center mask, 27.8 dB for the pixel mask, and 21.1 dB for the half mask, compared with CE at 28.5, 22.9, and 19.9 dB and SII at 19.4, 22.8, and 13.7 dB. On Cars with a pixel mask, E-CAE reaches 19.5 dB versus 15.6 dB for CE and 18.9 dB for SII. On SVHN with a pixel mask, E-CAE reaches 40.4 dB versus 30.5 dB for CE and 33.0 dB for SII. For denoising on BSD500/BSD200-style grayscale data with Gaussian noise at $3\times 3$6, E-CAE attains 28.23, 26.17, and 24.83 dB, respectively, exceeding BM3D and RED at all reported noise levels and exceeding MemNet except that the paper notes a single exception where MemNet slightly wins at $3\times 3$7. The authors also emphasize that CE and SII can rely on masks at inference and only restore missing regions, whereas E-CAE outputs a complete image and is evaluated over the entire image (Suganuma et al., 2018).

RED-CNN demonstrates CAER behavior in medical imaging rather than natural-image restoration. In the simulated abdominal case reported in Table I, LDCT yields PSNR 34.3094, RMSE 0.0193, and SSIM 0.8276, whereas RED-CNN yields PSNR 39.1959, RMSE 0.0097, and SSIM 0.9339. Over the testing set in Table II, RED-CNN reports PSNR $3\times 3$8, RMSE $3\times 3$9, and SSIM $3\times 3$0, which are the best among the reported methods. The paper further states that RED-CNN best suppresses noise while preserving structure on real clinical low-dose CT, improves visibility of low-contrast lesions, and is slightly better than KAIST-Net while being faster in training and testing. After training, the average execution time is 3.68 s (Chen et al., 2017).

Res-WCAE specializes the CAER idea to biometric denoising under severe AWGN. The dataset is SOCOFing, with 6,000 fingerprint images from 600 African subjects, converted to grayscale, resized to $3\times 3$1, and split into 70% training, 15% validation, and 15% testing. The degradation model is

$3\times 3$2

The model handles noise levels from $3\times 3$3 to $3\times 3$4, with a specifically evaluated case at $3\times 3$5. At that noise level, the reported metrics are: AWGN baseline PSNR 7.92, SSIM 0.45, MSE 0.17; dense neural network PSNR 10.03, SSIM 0.35, MSE 0.12; autoencoder PSNR 12.77, SSIM 0.69, MSE 0.06; and Res-WCAE PSNR 17.88, SSIM 0.79, MSE 0.02. The paper also reports an improved PSNR relative to the noisy input, with average improvement around 7.5 dB over a wide range of noise levels, and claims better performance than U-Finger and Fpd-m-net, especially under heavy noise (Liang et al., 2023).

CRsAE demonstrates that CAER-like structure can be used for interpretability-driven dictionary learning rather than only end-to-end denoising. On natural images, the learned filters become Gabor-like edge detectors, especially when the encoder is deeper, the encoder and decoder weights are tied, and $3\times 3$6 is trained with the EM-inspired loss. On extracellular neural recordings, the model learns realistic spike templates. The paper reports about 5× faster performance than a state-of-the-art CDL algorithm based on convex optimization (Sporco), and about 900× faster spike sorting than continuous basis pursuit on long recordings (Tolooshams et al., 2019).

6. Interpretive themes, limitations, and ongoing significance

Several technical themes recur across these CAER formulations. First, simple objectives can be sufficient when the architecture is well matched to the task. E-CAE repeatedly stresses that no adversarial loss or perceptual loss is needed, and attributes its performance to the architectural search rather than to sophisticated loss engineering (Suganuma et al., 2018). RED-CNN likewise uses MSE and Adam in a comparatively shallow 10-layer encoder–decoder, while explicitly noting that deeper networks do not always yield better results for low-level image restoration (Chen et al., 2017). This directly challenges the assumption that restoration quality necessarily depends on adversarial training, very deep stacks, or highly specialized losses.

Second, residual pathways are closely tied to domain-specific structure preservation. In E-CAE, frequent first-layer skip connections suggest that low-level input information is crucial for accurate restoration (Suganuma et al., 2018). In RED-CNN, shortcuts are justified by structural preservation and lesion detectability in CT (Chen et al., 2017). In Res-WCAE, concatenative skips are motivated by ridge continuity, edge sharpness, and minutiae-like detail in fingerprints (Liang et al., 2023). In CRsAE, residual updates are not merely heuristic but arise from proximal optimization and retain an interpretable sparse-coding semantics (Tolooshams et al., 2019). This suggests that CAER is best understood as a design principle for controlling information loss, not as a fixed topology.

Third, the literature also defines clear limits. E-CAE searches only over symmetric CAEs with standard convolutions and optional skip connections, not arbitrary graphs; many genotype nodes are non-functioning; inpainting and denoising use different architectural constraints; and the reported performance depends on extensive fine-tuning after search. The authors also state that the method optimizes pixel-level reconstruction accuracy rather than perceptual realism, so the results are best interpreted in an MSE/PSNR sense, not necessarily as the most visually photorealistic generators (Suganuma et al., 2018). RED-CNN acknowledges that training can be time-consuming and that performance depends on the training noise distribution (Chen et al., 2017). Res-WCAE, in the provided account, is evaluated only on SOCOFing and mainly under synthetic AWGN, and the method is specialized for fingerprint-like biometric patterns (Liang et al., 2023). CRsAE is more constrained than a generic CAER because its tied weights, two-sided ReLU, and dual-loss training are imposed by the dictionary-learning objective rather than by general reconstruction practice (Tolooshams et al., 2019).

A plausible implication is that CAER performance depends less on the abstract presence of “residual connections” than on the exact coupling between encoder, decoder, and target domain. Additive mirrored fusion, global residual prediction, wavelet-conditioned concatenation, and optimization-derived residual correction all instantiate the same broad idea, but they do so with different priors about where information is lost and how it should be recovered. Within that broader interpretation, CAER serves as a unifying framework for restoration models that combine convolutional encoding–decoding with explicit bypass mechanisms to preserve fidelity, stabilize training, or enforce interpretability.