Deep Perceptual Relevancy Loss

• Deep Perceptual Relevancy Loss Function (DPRLF) is a family of loss functions that measure differences in learned feature spaces to mirror human sensory perceptions.
• It employs weighted sums of L₁/L₂ differences from pretrained networks, using CSF-derived and human-judgment weights to prioritize perceptually salient features.
• DPRLF has practical applications in medical imaging, image compression, generative modeling, and audio enhancement, yielding superior perceptual quality despite slight declines in pixel-based metrics.

The Deep Perceptual Relevancy Loss Function (DPRLF) is a family of loss functions for deep neural networks, constructed to align optimization objectives with human perceptual relevance rather than pixel-wise or pointwise similarity. DPRLFs operate by measuring differences between samples in a learned or handcrafted feature space, employing weightings and feature selections motivated by models of the human visual system (HVS), human auditory perception, or human-judgment data, depending on the modality. The framework has been instantiated for images, audio signals, and 3D point clouds, and forms the basis for state-of-the-art approaches in medical imaging, compression, generative modeling, speech enhancement, and computational imaging.

1. Mathematical Foundations and Core Formulations

A canonical DPRLF is expressed as a weighted sum of squared L₂ or L₁ differences between deep features extracted from a fixed (pretrained or custom) neural backbone. In the D-PerceptCT approach for LDCT enhancement, DPRLF is defined as

$$\mathcal{L}_{\mathrm{DPRLF}} = \lambda_{\mathrm{low}} \|\phi_{\mathrm{low}}(I_{\mathrm{pred}}) - \phi_{\mathrm{low}}(I_{\mathrm{gt}})\|_2^2 + \lambda_{\mathrm{mid}} \|\phi_{\mathrm{mid}}(I_{\mathrm{pred}}) - \phi_{\mathrm{mid}}(I_{\mathrm{gt}})\|_2^2 + \lambda_{\mathrm{high}} \|\phi_{\mathrm{high}}(I_{\mathrm{pred}}) - \phi_{\mathrm{high}}(I_{\mathrm{gt}})\|_2^2$$

where $\phi_*$ denotes feature activations at different semantic levels (e.g., VGG-16 layers), and the weights $\lambda_*$ reflect perceptual importance, often derived from the HVS contrast sensitivity function (CSF) (Nabila et al., 18 Nov 2025). Other modalities instantiate DPRLF as the mean squared difference in a feature space learned by a Wavenet (PESQ approximator for audio (Elbaz et al., 2017)), an LSTM with FIR-based pre-emphasis filters for audio modeling (Wright et al., 2019), or a 3D convolutional autoencoder for point cloud geometry (Quach et al., 2021).

Feature-space selection and weighting mechanisms vary. In (Tariq et al., 2018) channel selection and normalized weighting are derived from direct measurements of frequency and orientation selectivity scores that capture CSF-like and orientation-tuning properties in intermediate layers, restricting the loss to the subset of channels most aligned with human perception. In some image compression settings (Patel et al., 2019), learned per-channel weights $w^l$ (obtained from human similarity judgments) scale the per-location differences, and unit feature normalization is employed to match deep Learned Perceptual Image Patch Similarity (LPIPS) design.

2. Perceptual Motivation and Connection to Human Sensitivity

DPRLF differs fundamentally from pixel- or voxel-based losses by exploiting the structure of feature representations tuned, either via supervised learning or carefully designed autoencoders, to reflect aspects of human perception. In the visual domain, frequency weighting via the CSF up-weights mid-band spatial frequencies that are maximally discriminable for the human observer and down-weights coarse or fine details with less perceptual salience (Nabila et al., 18 Nov 2025, Tariq et al., 2018). In audio, perceptual losses account for frequency masking, loudness asymmetries, or objective standards (e.g. PESQ, A-weighting), compelling the network to prioritize perceptual artifacts over spectral divergence (Elbaz et al., 2017, Wright et al., 2019).

General guidance for DPRLF configuration includes:

• Choice of feature space: Pretrained image classification (e.g., VGG-16 (Deng et al., 2019, Nabila et al., 18 Nov 2025)), custom-trained autoencoders (e.g., TDF voxel features (Quach et al., 2021)), or perceptual proxy networks (e.g., Wavenet estimator of PESQ (Elbaz et al., 2017)).
• Weighting of features/frequencies: Fixed (CSF-derived), human-judgment-learned, or optimized during training.
• Subset selection: Restriction to the most perceptually effective channels improves alignment with human quality scores (see PE-based selection in (Tariq et al., 2018)).
• Preprocessing: Range normalization, input scaling, or task-specific calibration.

3. Practical Implementations and Modalities

The design and integration of DPRLF depend on the application and data modality:

Modality	Feature Extractor / Comparator	Weighting / Selection	Example Papers
Medical Images	VGG-16 pretrained on ImageNet, selected layers	CSF-based $\lambda_*$	(Nabila et al., 18 Nov 2025, Deng et al., 2019)
Natural Images	VGG-16, AlexNet, GoogleNet, custom (cf. LPIPS)	Human-judgment or PE	(Patel et al., 2019, Tariq et al., 2018, Dosovitskiy et al., 2016)
Audio	Pre-emphasis filtering (A-weighting, HP, FD), Wavenet proxies	Human loudness/PESQ	(Elbaz et al., 2017, Wright et al., 2019)
3D Point Clouds	3D Conv autoencoder (binary/TDF voxel)	Feature selection	(Quach et al., 2021)

Feature extraction is usually performed on frozen networks; gradients flow only through the generative or enhancement network. The final loss is summed or blended with auxiliary terms such as MSE, Charbonnier, or adversarial losses, with a coefficient (often $\alpha \approx 1$) that balances pixel and perceptual fidelity, exploiting the perceptual weights to encode the desired tradeoff (Nabila et al., 18 Nov 2025).

4. Empirical Benefits and Comparative Performance

Ablation studies consistently show that incorporating DPRLF leads to outputs with markedly improved perceptual metrics, as judged by LPIPS, DISTS, MOS, or human 2AFC preference—even when PSNR and SSIM decrease slightly (Nabila et al., 18 Nov 2025, Patel et al., 2019, Deng et al., 2019). For example, D-PerceptCT shows a reduction in PSNR (44.27 dB → 42.97 dB) after switching from MSE-only to DPRLF, but achieves a 73% reduction in LPIPS (0.0388 → 0.0104) and a substantial drop in DISTS, directly evidencing a perceptual fidelity gain (Nabila et al., 18 Nov 2025). In audio, A-weighted or PESQ-proxy DPRLFs yield MUSHRA listening scores that significantly outperform unweighted losses (Wright et al., 2019, Elbaz et al., 2017).

In object detection tasks on compressed images, regularization with a DPRLF maintains downstream accuracy at substantially reduced bitrates compared to non-perceptual codecs (Patel et al., 2019). For phase retrieval under extreme Poisson noise, DPRLF-trained networks recover sharper edges and semantically critical cues, outperforming MSE and NPCC loss designs (Deng et al., 2019).

5. Hyperparameterization and Feature Weight Assignment

The assignment of weights to feature bands or channels is central to DPRLF design:

• In (Nabila et al., 18 Nov 2025), $\lambda_{\mathrm{low}}=0.35$, $\lambda_{\mathrm{mid}}=0.50$, $\lambda_{\mathrm{high}}=0.15$ reflect CSF measurements, with no adaptation during training.
• Channel weighting in PE-based DPRLFs is performed by normalizing the product of CSF-weighted frequency sensitivity and orientation selectivity, limiting the active channels typically to the top 10% by PE score (Tariq et al., 2018).
• In LPIPS-style DPRLFs (Patel et al., 2019), channel-wise weights $w^l$ are learned directly from human similarity judgment triplets for optimal perceived similarity estimation.
• For audio, fixed filter coefficients are chosen to model human perceptual curves, e.g., A-weighting FIR + LP filter (Wright et al., 2019), or learned nonlinear mappings to objective perceptual scores (PESQ) (Elbaz et al., 2017).

Hyperparameters for auxiliary loss terms (pixel-wise loss, cycle-consistency, style/content mix) are determined empirically for stability and optimal performance (Zhu et al., 2023, Nabila et al., 18 Nov 2025).

6. Design Considerations and Limitations

Careful consideration is required in the selection of feature layers and their associated weights:

• Layer specificity: Mid-level layers contain optimal tradeoffs between spatial localization and semantic invariance (Dosovitskiy et al., 2016, Tariq et al., 2018).
• Potential artifacts: At extremely low SNRs or photon counts, DPRLFs relying on fixed feature spaces (e.g., VGG-16 relu2_2) may transmit artifacts reflective of non-smooth feature map sensitivities (Deng et al., 2019). Composite or blended losses are sometimes necessary.
• Calibration: For strictly quantitative or scientific imaging, blend DPRLF with MSE or NPCC and apply post-hoc calibration.
• Task adaptation: The perceptual model and weighting must be customized for each modality and target observer (e.g., HVS vs. auditory system).

7. Applications and Impact Across Domains

DPRLFs are now integrated in multiple architectures and domains:

• Medical Imaging: Achieve better diagnostic saliency and preservation of anatomic fine details in LDCT and CBCT translation tasks by guiding models toward HVS-aligned features rather than smoothing noise at the expense of clinically relevant structures (Nabila et al., 18 Nov 2025, Zhu et al., 2023).
• Image Compression: Deep perceptual metric-driven video and image codecs exhibit superior visual quality and higher object recognition accuracy than MSE- or SSIM-trained alternatives (Patel et al., 2019).
• Generative Models: Feature-based similarity metrics (DeePSiM, contextual-PE) yield visually plausible, high-detail inference in autoencoders, VAEs, and GANs, avoiding the blurring typical of standard pixel-wise losses (Dosovitskiy et al., 2016, Tariq et al., 2018).
• Audio Enhancement: Losses incorporating human-centric frequency emphasis (e.g., A-weighting, PESQ proxies) result in perceptual gains confirmed by formal listening studies (Elbaz et al., 2017, Wright et al., 2019).
• 3D Data: Perceptual losses in the latent TDF space of a trained 3D autoencoder track subjective point cloud quality better than classical pointwise or occupancy-based metrics (Quach et al., 2021).

In all settings, the central mechanism of DPRLF—optimizing in a feature space attuned to human relevance rather than coordinate-wise similarity—has proven to be a robust avenue for producing outputs judged superior by both learned perceptual metrics and direct human evaluation.