Semantic & Uncertainty-Aware ESRGAN

• The paper introduces SU-ESRGAN, a framework for satellite imagery that integrates semantic consistency and Monte Carlo dropout-based uncertainty estimation.
• It preserves the ESRGAN architecture with RRDB blocks while adding a DeepLabv3 segmentation loss to maintain class-specific details.
• Per-pixel uncertainty maps guide decision-making by highlighting low-confidence areas, which is crucial for applications like disaster response and urban planning.

Semantic and Uncertainty-Aware ESRGAN (SU-ESRGAN) is a super-resolution framework for remote sensing that extends ESRGAN by introducing semantic consistency and pixel-wise uncertainty estimation into adversarial image reconstruction (Ramkumar, 1 Aug 2025). In the reported formulation, it is presented as the first SR framework designed for satellite imagery to integrate the ESRGAN, segmentation loss via DeepLabv3 for class detail preservation, and Monte Carlo dropout to produce pixel-wise uncertainty maps. Its motivation is that GAN-based super-resolution can generate realistic textures yet still lack semantic consistency and per-pixel confidence, which limits credibility in applications such as disaster response, urban planning, and agriculture. The framework inherits the perceptual and adversarial design of ESRGAN, whose original contribution was to improve SRGAN through Residual-in-Residual Dense Blocks (RRDBs), a relativistic discriminator, and a revised perceptual loss aimed at stronger brightness consistency and texture recovery (Wang et al., 2018).

1. ESRGAN foundation and architectural inheritance

ESRGAN is the immediate architectural basis for SU-ESRGAN. The original ESRGAN replaces standard residual blocks with the Residual-in-Residual Dense Block and removes BatchNorm layers, applies residual scaling $\gamma = 0.2$ and small parameter initialization $(\text{MSRA} \times 0.1)$, uses a relativistic average GAN discriminator, and computes perceptual loss on VGG19 features before activation (Wang et al., 2018). The generator operates on a low-resolution input image $I_{lr} \in \mathbb{R}^{H \times W \times 3}$, begins with a shallow feature extractor $3 \to 64$ using a $3 \times 3$ convolution and LeakyReLU$(0.2)$, passes features through a trunk of RRDB blocks, applies a long skip connection, performs two successive PixelShuffle upsampling stages, and reconstructs the final super-resolved image with a $64 \to 3$ convolution.

In the detailed specification, the ESRGAN baseline uses $N=16$ RRDB blocks, with $N=23$ for a deeper model. Each RRDB receives a 64-dimensional feature and returns a 64-dimensional feature. The upsampling path consists of two low-resolution feature-space blocks:

• Conv Upsample_1: $64 \to 256$, kernel $(\text{MSRA} \times 0.1)$0 $(\text{MSRA} \times 0.1)$1 PixelShuffle$(\text{MSRA} \times 0.1)$2 $(\text{MSRA} \times 0.1)$3 LeakyReLU
• Conv Upsample_2: $(\text{MSRA} \times 0.1)$4, kernel $(\text{MSRA} \times 0.1)$5 $(\text{MSRA} \times 0.1)$6 PixelShuffle$(\text{MSRA} \times 0.1)$7 $(\text{MSRA} \times 0.1)$8 LeakyReLU

The RRDB itself is defined as three cascaded DenseBlocks wrapped by two outer residual shortcuts. Each DenseBlock is a 5-layer dense network with growth rate $(\text{MSRA} \times 0.1)$9, followed by a $I_{lr} \in \mathbb{R}^{H \times W \times 3}$0 compression convolution back to 64 channels. The block-level forward form is given as

$I_{lr} \in \mathbb{R}^{H \times W \times 3}$1

with $I_{lr} \in \mathbb{R}^{H \times W \times 3}$2 in the specification.

This baseline matters because SU-ESRGAN does not replace ESRGAN’s core perceptual SR machinery. Instead, it preserves the RRDB-based generator and relativistic adversarial formulation while augmenting them with semantic regularization and uncertainty estimation. A central implication of the reported motivation is that ESRGAN’s strength in producing sharp, high-frequency detail is also the source of a domain-specific risk: hallucinated structure can be visually plausible yet semantically incorrect when reconstructing satellite or drone imagery (Ramkumar, 1 Aug 2025).

2. Semantic consistency mechanisms

The implemented SU-ESRGAN introduces semantic awareness through a pre-trained DeepLabv3 segmentation network $I_{lr} \in \mathbb{R}^{H \times W \times 3}$3 (Ramkumar, 1 Aug 2025). For each pixel $I_{lr} \in \mathbb{R}^{H \times W \times 3}$4, the predicted class index in the super-resolved image and in the ground-truth high-resolution image is defined as

$I_{lr} \in \mathbb{R}^{H \times W \times 3}$5

The semantic consistency term is then the average $I_{lr} \in \mathbb{R}^{H \times W \times 3}$6 distance between these label maps:

$I_{lr} \in \mathbb{R}^{H \times W \times 3}$7

The reported full generator objective becomes

$I_{lr} \in \mathbb{R}^{H \times W \times 3}$8

where $I_{lr} \in \mathbb{R}^{H \times W \times 3}$9 weights semantic fidelity so that segmentation loss and perceptual loss are on comparable scales.

The accompanying technical specification also describes a more explicit semantic-fusion blueprint derived from ESRGAN’s internal feature hierarchy (Wang et al., 2018). In that specification, a semantic segmentation map $3 \to 64$0 is either precomputed or jointly predicted, and then fused at multiple generator depths via Spatial Feature Transform (SFT) or simple concatenation. At selected layers, modulation parameters are computed from the segmentation features:

$3 \to 64$1

and the feature modulation is

$3 \to 64$2

The same specification proposes a semantic consistency loss based on pixel-wise KL divergence between softmax probability vectors:

$3 \to 64$3

These two semantic formulations should be distinguished. The SU-ESRGAN paper summary reports DeepLabv3 with an “argmax-L1” loss, whereas the ESRGAN extension blueprint proposes multi-depth semantic conditioning and KL-based soft consistency. The reported authors note that one could replace the “argmax-L1” formulation with soft cross-entropy or Dice loss on DeepLabv3 softmax outputs, but they found the simple argmax-based form effective (Ramkumar, 1 Aug 2025). This suggests a design space in which semantic awareness can act either as an output-level regularizer or as an internal conditioning signal.

3. Uncertainty modeling and predictive confidence

A defining feature of SU-ESRGAN is per-pixel uncertainty estimation. In the reported model, dropout with probability $3 \to 64$4 is kept active in the RRDB layers even at inference time, enabling Monte Carlo dropout as an epistemic uncertainty mechanism (Ramkumar, 1 Aug 2025). For an input $3 \to 64$5, $3 \to 64$6 stochastic forward passes are drawn:

$3 \to 64$7

The predictive mean and variance at each pixel are estimated as

$3 \to 64$8

Uncertainty maps are often visualized through the standard deviation $3 \to 64$9. High $3 \times 3$0 indicates low confidence, for example at fine edges, while low $3 \times 3$1 corresponds to stable predictions.

The technical extension blueprint expands this idea to joint aleatoric and epistemic uncertainty (Wang et al., 2018). In that specification, the generator predicts both a mean image $3 \times 3$2 and a log-variance map $3 \times 3$3, with the last generator convolution outputting 3 channels for $3 \times 3$4 and 1 channel for $3 \times 3$5. The heteroscedastic aleatoric term is

$3 \times 3$6

Epistemic uncertainty is modeled by inserting Monte Carlo dropout with $3 \times 3$7 in selected RRDB convolutions and drawing stochastic samples $3 \times 3$8. The predictive variance is estimated approximately as

$3 \times 3$9

The corresponding full SU-ESRGAN loss in that blueprint is

$(0.2)$0

The paper’s reported uncertainty behavior is operationally significant: uncertainty maps consistently highlight regions of fine structural detail, including roads, rooftops, and vessel outlines, indicating where SR predictions should be treated with caution (Ramkumar, 1 Aug 2025). A plausible implication is that uncertainty is not merely an auxiliary visualization but a decision-support signal for downstream remote-sensing analysis.

4. Losses, optimization, and data regimes

The original ESRGAN combines content, perceptual, and adversarial learning in a two-stage schedule (Wang et al., 2018). In the technical summary, the content term is

$(0.2)$1

and the perceptual term uses VGG19 features before activation:

$(0.2)$2

The full ESRGAN generator loss in that specification is

$(0.2)$3

with $(0.2)$4 and $(0.2)$5.

The training schedule is explicitly divided into two stages. Stage 1 is PSNR-oriented pretraining of the generator using only $(0.2)$6, with learning rate $(0.2)$7, decay by a factor of $(0.2)$8 every $(0.2)$9 minibatch updates, batch size 16, patch size $64 \to 3$0, and data augmentation by random horizontal flip and $64 \to 3$1 rotations. Stage 2 is GAN fine-tuning, initialized from stage 1, with alternating updates of generator and discriminator, learning rate $64 \to 3$2 halved at iterations $64 \to 3$3, Adam$64 \to 3$4, and one discriminator update per generator update. The reported training data for ESRGAN are DIV2K (800), Flickr2K (2,650), and OST ($64 \to 3$5) high-quality 2K images.

The SU-ESRGAN training pipeline is domain-specific and remote-sensing oriented (Ramkumar, 1 Aug 2025). Training uses the UC Merced Land Use dataset (2,100 RGB images) and AID (10,000 RGB images). After bicubic down-sampling to $64 \to 3$6 from $64 \to 3$7, the union is split 64% train / 16% val / 20% test, corresponding to 7,744 / 1,936 / 2,420 images. Both baseline ESRGAN and SU-ESRGAN are trained for 30 epochs on a Tesla P100 GPU with early stopping of patience 10. The generator and discriminator learning rates are initialized around $64 \to 3$8, while fine-tuning on drone imagery uses a reduced rate of $64 \to 3$9 to avoid overfitting.

Cross-domain fine-tuning is carried out on UAVid and the Aerial Maritime Drone Dataset, each split 60/20/20. The reported motivation is to test how well a satellite-trained model adapts to differences in flight altitude and imaging perspective, which is described as crucial for real-world UAV deployment (Ramkumar, 1 Aug 2025).

5. Evaluation protocols and empirical behavior

The ESRGAN extension specification proposes a broad evaluation suite that separates reconstruction fidelity, perceptual quality, semantic fidelity, and calibration (Wang et al., 2018). Reconstruction fidelity is assessed with PSNR and SSIM on the Y channel. Perceptual quality is assessed with Ma’s score and NIQE, combined into the perceptual index

$N=16$0

For semantic fidelity, the specification proposes mIoU between $N=16$1 and $N=16$2. For uncertainty calibration, it proposes Expected Calibration Error (ECE), Negative Log-Likelihood (NLL) under a Gaussian model,

$N=16$3

and sharpness-vs-calibration curves in the form of reliability diagrams.

The reported SU-ESRGAN experiments evaluate held-out satellite/aerial performance using PSNR, SSIM, LPIPS, and FID (Ramkumar, 1 Aug 2025). On that test set, Baseline ESRGAN obtains PSNR 25.99 dB, SSIM 0.696, LPIPS 0.2672, and FID 68.62. SU-ESRGAN obtains PSNR 25.01 dB, SSIM 0.696, LPIPS 0.3172, and FID 75.23. The authors state that SU-ESRGAN’s PSNR and SSIM are comparable to the baseline, while LPIPS and FID degrade slightly.

The qualitative interpretation reported alongside these numbers is that SU-ESRGAN produces less hallucinated detail and fewer semantic errors, with the example that buildings remain building-shaped, but at the cost of mildly blurrier edges. This directly frames the system as prioritizing semantically constrained reconstruction over purely texture-driven perceptual sharpness.

A common misconception is that adding semantic supervision and uncertainty estimation should necessarily improve all image-quality metrics. The reported results do not support that interpretation. Instead, they indicate that semantic regularization can preserve class detail while still incurring a perceptual-versus-fidelity trade-off in LPIPS and FID, even when PSNR and SSIM remain comparable.

6. Cross-domain adaptation, limitations, and projected development

The cross-domain results are central to SU-ESRGAN’s remote-sensing relevance (Ramkumar, 1 Aug 2025). After fine-tuning, the model trained on UAVid reports PSNR 23.78, SSIM 0.637, LPIPS 0.2448, and FID 3.75. Fine-tuning on the Aerial Maritime Drone Dataset reports PSNR 24.77, SSIM 0.742, LPIPS 0.1769, and FID 154.15. The reported interpretation is asymmetrical: the extreme drop in FID on UAVid is described as symptomatic of mode collapse rather than true quality gain, whereas fine-tuning on the more similar maritime dataset raises SSIM by $N=16$4 and lowers LPIPS by $N=16$5, but increases FID, signaling richer, more diverse outputs that remain slightly distribution-shifted from real maritime images.

These findings are tied to the datasets’ imaging geometry. UAVid is described as dense urban with oblique views and lower altitude, while the Aerial Maritime dataset is described as higher altitude and more top-down. The latter aligns more closely with satellite vantage points, and adaptation is therefore reported as stronger. The stated conclusions are the necessity of domain-aware fine-tuning schedules, the value of semantic loss in preserving class-specific details across domains, and the warning that blindly optimizing for FID can mask collapse when domain gaps are large.

The future directions named for SU-ESRGAN are specific. They include replacing the argmax-L1 segmentation loss with soft cross-entropy or Dice, jointly learning segmentation and super-resolution in a multi-task framework, integrating more advanced uncertainty estimation such as evidential networks to separate aleatoric from epistemic uncertainty, extending the framework to multi-spectral and SAR data, and developing curriculum-based domain adaptation that progressively shifts from satellite to drone scales (Ramkumar, 1 Aug 2025). These proposals suggest that SU-ESRGAN is best understood not as a single fixed architecture, but as a family of ESRGAN-based remote-sensing SR systems in which semantic preservation, uncertainty quantification, and domain adaptation are treated as first-class design constraints.