Guided Deep Decoder (GDD) Architecture

• The paper introduces GDD, a hybrid framework that couples untrained deep decoders with encoder–decoder guidance to enable unsupervised inverse imaging.
• It employs multi-scale attention gates (URU and FRU) to integrate features from degraded and guidance images, achieving state-of-the-art performance in tasks like super-resolution and PET denoising.
• GDD adapts to diverse inverse problems without supervised training, making it effective for applications such as hyperspectral fusion and medical image restoration.

The Guided Deep Decoder (GDD) is a family of hybrid neural architectures that combine underparameterized, untrained convolutional image priors with multi-scale guidance from a secondary input, enabling unsupervised solutions to a diverse range of inverse problems including image fusion, denoising, compressive sensing, and PET–MR image restoration. Originating from the intersection of deep image prior theory and encoder–decoder-based attention, the architecture operates without reliance on supervised sample pairs or large labeled datasets and instead leverages structural or semantic cues present in guidance images or pretrained models (Uezato et al., 2020, Onishi et al., 2021, Daniels et al., 2020).

1. Core Architectural Framework

The canonical GDD architecture is a two-stream neural network composed of:

• Guidance Encoder–Decoder: A U-Net-style subnetwork processes the auxiliary guidance image $G$ (for example, an MR image or high-resolution RGB image), extracting multi-scale feature maps via a sequence of convolutional downsampling (encoder) and upsampling (decoder) layers, with skip connections providing spatial context at various scales. The encoder features are denoted $\Gamma_k$, and decoder features $\Xi_k$, $k=1, ..., K$ (Uezato et al., 2020, Onishi et al., 2021).
• Deep Decoder: An untrained, convolution-only upsampling module parameterized by random input tensor $Z$, mapping it to output image $X = f_\theta(Z)$ through a cascade of bilinear upsampling, $3\times3$ convolutions, channel normalization, and nonlinearities. This subnetwork is strictly a decoder (no encoder), with the design informed by the underparameterization principle (“Deep Decoder” [Heckel & Hand, 2019]) (Daniels et al., 2020, Uezato et al., 2020).
• Feature Refinement Units (FRUs and URUs): At each scale, the two subnetworks interact via channel-wise gating/attention. The Upsampling Refinement Unit (URU) injects encoder features $\Gamma_k$ into the corresponding deep decoder layer by channel-wise multiplication after nonlinear projection. The Feature Refinement Unit (FRU) similarly modulates with decoder-path features $\Xi_k$. Thus, the guidance image acts exclusively through these attention gates, with no direct imposition of guidance image structure onto the reconstruction (Uezato et al., 2020, Onishi et al., 2021).

2. Mathematical Formulation and Optimization

Let $Y$ denote the primary (degraded, noisy, or low-resolution) image, $G$ the guidance image, and $X$ the desired fused or restored output. The network output is $f_{\theta,\phi}(Z, G)$ with parameters $\theta$ (deep decoder) and $\phi$ (guidance encoder–decoder). For general image fusion or restoration, the unsupervised optimization is:

$$\theta^\star, \phi^\star = \arg\min_{\theta, \phi} \ \mathcal{L}(f_{\theta,\phi}(Z, G), Y, G),$$

where $\mathcal{L}$ is a task-adapted loss, typically $\ell_2$ for denoising, spectral–spatial fidelity for fusion, or a combination thereof (Uezato et al., 2020, Onishi et al., 2021).

In the case of hybrid (learned–unlearned) priors (Daniels et al., 2020), the output is a linear combination: $$H(z, \theta, \alpha, \beta) = \alpha G_\varphi(z) + \beta DD(\theta)$$ where $G_\varphi$ is a pretrained GAN generator, $DD$ is the untrained deep decoder, and $\alpha, \beta$ are learned scalar coefficients. The loss is

$$\min_{z, \theta, \alpha, \beta} \|\mathsf{A} H(z, \theta, \alpha, \beta) - y\|_2^2$$

for linear inverse problems $\mathsf{A}x + \eta = y$.

3. Attention Mechanisms and Feature Integration

GDD implements multi-scale guidance through channel-wise attention gates, formalized as:

• URU gating (encoder feature integration):

$$\widetilde F_k = F_k \odot \sigma(\mathrm{LReLU}(\mathrm{Conv}_{1\times1}(\Gamma_k)))$$

• FRU gating (decoder feature integration):

$$F_k^{out} = \widetilde F_k \odot \sigma(\mathrm{LReLU}(\mathrm{Conv}_{1\times1}(\Xi_k)))$$

where $\sigma$ is the sigmoid function, $\odot$ is channel-wise multiplication. This restricts structural and semantic guidance to modulate deep decoder features, preventing direct transfer of high-frequency artifacts or imposition of unwanted shapes from $G$ (Uezato et al., 2020, Onishi et al., 2021).

4. Application Domains

4.1 Image Fusion and Super-Resolution

GDD was established as a general-purpose prior for image pair fusion in tasks such as hyperspectral–RGB super-resolution, pansharpening, and flash/no-flash denoising (Uezato et al., 2020). Task-specific unsupervised losses combine data fidelity with spectral or gradient matching. Empirical results on CAVE (HS super-resolution), WorldView-2 (pansharpening), and flash/no-flash datasets demonstrate state-of-the-art reconstruction metrics (RMSE, ERGAS, SSIM, Q8, QNR), outperforming classical CNNs and prior deep image prior methods without external training data. Ablations reveal that removing URU produces over-smoothed images (attenuated edge structures), while omission of FRU reduces semantic alignment for small objects.

4.2 Medical Imaging—PET Denoising

The MR-Guided Deep Decoder (“MR-GDD”) applies the GDD paradigm to 3D PET image denoising, leveraging anatomical details from registered MR images (Onishi et al., 2021). The network processes 3D PET–MR volumes via a U-Net-style MR encoder–decoder and a deep decoder for PET, integrated through anatomically guided FRUs/URUs. The unsupervised $\ell_2$ loss between low-count PET and output suffices as the deep decoder architecture enforces strong implicit regularization. Quantitative evaluation (Monte Carlo PET simulation, preclinical nonhuman primate, and human amyloid datasets) confirms that MR-GDD achieves the highest PSNR (27.92±0.44 dB) and SSIM (0.886±0.007) compared to Gaussian filtering, image-guided filtering, Deep Image Prior (DIP), and MR-DIP, indicating superior denoising with minimal loss of spatial resolution or quantitative fidelity.

4.3 Hybrid Learned–Unlearned Inverse Recovery

Combining deep decoders with pretrained GAN priors offers an adaptive representation for inverse problems such as compressive sensing and image super-resolution (Daniels et al., 2020). The hybrid GDD model reconstructs images as a linear mixture of a GAN-generated semantic base and a deep decoder residual, tuned via the learned mixing coefficients. On in-distribution data (e.g., faces), the hybrid significantly improves PSNR (by 1–2 dB over deep decoder, >10 dB over GAN alone), while for out-of-distribution cases (e.g., birds with face-trained GAN), the model down-weights the GAN contribution to default to the unlearned prior. This adaptivity reduces the intrinsic representation error of GAN-only or decoder-only models.

5. Optimization and Implementation

GDD models employ differentiable optimization of network parameters initialized randomly, leveraging Adam or L-BFGS for small-batch stochastic gradient descent. No pretraining or external labeled data is required. For deep decoder branches, early stopping is avoided by underparameterization, which empirically prevents overfitting. For MR-GDD, L-BFGS quasi-Newton was used, while Adam with learning rate $1{\times}10^{-3}$ to $1{\times}10^{-4}$ is typical for other fusion domains (Onishi et al., 2021, Uezato et al., 2020). Computation is tractable: in medical 3D settings, modern GPUs suffice for rapid training and inference.

6. Limitations and Directions for Extension

GDD’s principal limitations arise from the nature of its attention-based integration and its unsupervised paradigm. Effective operation presumes correct alignment of guidance and target images; degradation in registration (e.g., PET/CT misalignment, patient motion) can impair gating fidelity (Onishi et al., 2021). As guidance is currently exploited only for denoising/restoration, extension to simultaneous structural corrections—such as partial-volume correction in PET—is highlighted as a future prospect. Validation on diverse clinical populations, especially with asymmetrical or pathological tracer uptake, remains outstanding. For hybrid models, computational cost is roughly doubled relative to single-branch methods, and empirical performance in extreme subsampling is contingent on the relevance of learned priors.

7. Impact, Empirical Summary, and Comparative Table

GDD unifies neural implicit priors and multi-scale guidance for unsupervised inverse imaging. It consistently surpasses handcrafted priors, Deep Image Prior, and stand-alone deep decoders, matching or outperforming supervised architectures in several benchmarks without labeled training data (Uezato et al., 2020, Onishi et al., 2021, Daniels et al., 2020). The following table summarizes typical reported metrics across distinct imaging tasks:

Application	Best GDD Variant	Reference Metric (Mean±Std)	Notable Comparator (Metric)
HS Super-Resolution (CAVE)	GDD	SSIM = 0.9869	MHF (supervised, competitive)
Pansharpening (WorldView-2)	GDD	Q8 = 0.9469; QNR = 0.9517	Best other QNR = 0.9492
PET Denoising (Simulation)	MR-GDD	PSNR=27.92±0.44dB; SSIM=0.886±0.007	MR-DIP (27.65±0.42dB; 0.879±0.007)
Compressive Sensing (CelebA)	Hybrid GAN+DD (GDD)	PSNR ≈ 27dB (m=5%)	Deep Decoder (≈26dB), GAN (≈15dB)

These results illustrate GDD’s capacity to exploit cross-modal structure and multi-scale cues in the absence of large-scale paired datasets. A plausible implication is that GDD may serve as a general unsupervised prior applicable to broad classes of imaging inverse problems.