Perceptual Similarity Metrics in Imaging

• Perceptual similarity metrics are algorithmic proxies for human visual judgment that quantify similarity using feature spaces aligned with semantic content.
• They integrate classical methods like SSIM with learned representations to capture invariances such as color shifts and geometric perturbations.
• These metrics drive practical applications in image generation, reconstruction, compression, adversarial defense, and multimodal evaluation.

Perceptual similarity metrics are algorithmic proxies for the human judgment of similarity between complex signals, most notably images. Unlike traditional pixel-wise distances such as $\ell_2$, PSNR, or SSIM, which are sensitive to pointwise deviations but insensitive to semantic content or invariant phenomena (e.g., tiny shifts, global color shifts), perceptual metrics compare visual or multimodal inputs in a feature space constructed to better align with the human visual system’s representations and decision boundaries. These metrics now underpin a range of applications including image generation, reconstruction, compression, adversarial robustness, and unified multimodal evaluation.

1. From Classical to Learned Perceptual Metrics

Classical perceptual similarity metrics such as SSIM and its multiscale variant MS-SSIM are founded on local comparisons of luminance, contrast, and structure in small patches. Given two image patches $x$ and $y$, the SSIM index is computed as

$$\mathrm{SSIM}(x, y) = [l(x, y)]^\alpha [c(x, y)]^\beta [s(x, y)]^\gamma$$

with closed-form expressions for the luminance, contrast, and structure terms based on local statistics. MS-SSIM extends this to multiple resolution levels, pooling local detail with global perceptual cues (Snell et al., 2015).

However, evaluations and large-scale psychophysical studies show that SSIM and MS-SSIM, while outperforming raw pixel error on many distortion types, fail under more complex, semantic, or geometric perturbations (Wickrema et al., 14 Jun 2025). Recent advances leverage