UMD SAD: Saliency-Guided Adversarial Defense

• UMD SAD is a saliency-based adversarial defense method that combines neural saliency estimation with block-wise JPEG compression to selectively preserve important image regions.
• It integrates pre-trained saliency models like PiCANet with adaptive quantization, achieving significant performance improvements in metrics such as Earth Mover’s Distance and Pearson’s correlation coefficient.
• The modular design allows flexible substitution of saliency estimators and JPEG quality settings, enhancing robustness against various adversarial attacks.

The Saliency-based Adversarial Defense (SAD) is a targeted, block-wise image denoising front-end designed to counteract adversarial perturbations in deep learning models. Rather than applying uniform filtering or compression across the input, SAD leverages a neural saliency map to selectively preserve information in visually important regions while more aggressively cleaning less salient regions. This approach aims to robustly remove adversarial noise while optimizing the retention of perceptual fidelity, as measured against ground-truth human saliency annotations. SAD utilizes pre-trained saliency estimators such as PiCANet, integrates directly with JPEG compression workflow, and demonstrates significant performance improvements on metrics such as Earth Mover’s Distance, Pearson’s correlation coefficient, and KL-divergence when defending against attacks such as FGSM, evaluated across standard saliency datasets (Tran et al., 2020).

1. Pipeline and Algorithmic Workflow

SAD consists of the following ordered algorithmic steps:

1. Input: Accepts an image $x$, potentially already perturbed by an adversarial attack (e.g., FGSM, DeepFool).
2. Saliency Estimation: Computes a pixel-wise saliency map $S(x) \in [0,1]^{W\times H}$ using a pre-trained model (PiCANet on DUTS-TR). The result is scaled to 8-bit precision via $S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$.
3. Block-wise Aggregation: Divides $x$ and $S_8(x)$ into non-overlapping $8\times 8$ spatial blocks $W_{ij}$ and computes mean block saliency $$Sal_{ij} = \frac{1}{64}\sum_{(u,v)\in W_{ij}} S_8(x)(u,v)$$.
4. Compression-Level Selection: Selects a quality factor $C_{ij}$ from a discrete list $Q = [Q_0,\ldots,Q_{K-1}]$ using $S(x) \in [0,1]^{W\times H}$0 and $S(x) \in [0,1]^{W\times H}$1.
5. Block-wise JPEG Re-compression: Re-compresses $S(x) \in [0,1]^{W\times H}$2 at quality $S(x) \in [0,1]^{W\times H}$3, producing $S(x) \in [0,1]^{W\times H}$4.
6. Output Assembly: Reassembles $S(x) \in [0,1]^{W\times H}$5 into the cleaned image $S(x) \in [0,1]^{W\times H}$6, retaining high fidelity where saliency is high and aggressively denoising elsewhere.

The core defense operator can be formalized as:

$S(x) \in [0,1]^{W\times H}$7

The method’s modularity allows replacements of both the saliency estimator and the compression scheme, provided block structure compatibility is maintained.

2. Formal Specification and Pseudocode

SAD’s implementation is efficiently described via the following pseudocode:

$S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$8

The saliency mapping is provided by PiCANet with no modifications except for utilization of its output as a guidance signal for block-wise JPEG quantization.

3. Quantitative Evaluation: Metrics and Results

Evaluation of SAD centers on comparison of model saliency output $S(x) \in [0,1]^{W\times H}$8 against ground-truth map $S(x) \in [0,1]^{W\times H}$9 using the following established metrics:

• Earth Mover’s Distance (EMD) ↓:

$S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$0, flow-constrained.

• Pearson’s Correlation Coefficient (CC) ↑:

$S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$1

• Normalized Scanpath Saliency (NSS) ↑:

$S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$2

• KL-Divergence (KLD) ↓:

$S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$3

• Similarity Score (SIM) ↑:

$S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$4

Quantitative outcomes are summarized below for representative adversarial (FGSM) settings and SAD with $S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$5:

Dataset + Backend	Setting	EMD↓	CC↑	NSS↑	KLD↓	SIM↑
SALICON + BASNet	Original	83.61	0.4218	0.3761	10.67	0.4060
	FGSM	79.91	0.4088	0.3648	11.36	0.3892
	FGSM + SAD	79.90	0.4123	0.3667	11.31	0.3907
ECSSD + BASNet	Original	48.07	0.9120	1.9792	1.5060	0.8844
	FGSM	45.52	0.8435	1.8291	3.2077	0.8041
	FGSM + SAD	45.93	0.8615	1.8741	2.8253	0.8249
ECSSD + CPD	Original	48.28	0.9044	1.9648	1.2533	0.8743
	FGSM	45.37	0.8253	1.7933	3.0017	0.7846
	FGSM + SAD	45.29	0.8306	1.8130	3.0791	0.7902
SALICON + CPD	Original	84.03	0.4644	0.4047	9.4618	0.4308
	FGSM	77.18	0.4409	0.3803	10.85	0.3956
	FGSM + SAD	76.42	0.4387	0.3776	11.02	0.3916

Across all tables, SAD recovers a significant portion of the saliency-metric drop induced by FGSM. Under DeepFool, these metrics show minimal degradation, so SAD has little net effect in such cases.

4. Saliency-Guided Denoising: Principles and Context

SAD’s key innovation is leveraging explicit visual saliency to guide spatially variable denoising under adversarial threat. Instead of applying the same compression or quantization level globally (as in generic JPEG or bit-depth compression), SAD ensures that semantically important regions, as determined by neural saliency, retain maximal fidelity. This targeted approach minimizes distortion in regions likely to be important for either human interpretation or downstream network prediction, while maximizing noise suppression in less critical areas. Global denoising approaches often degrade important information; SAD’s selective strategy addresses this by aligning with perceptual regions.

This mechanism does not depend on the specific choice of saliency estimator or quantization scheme, as long as the estimator is robust and output can be interpreted in a block-wise, monotonic fashion. PiCANet was chosen for robustness and saliency quality, but alternative estimators remain compatible provided they offer competitive detection of visual regions of interest.

5. Advantages and Limitations

Advantages:

• Targeted Cleaning: High-saliency preservation ensures that key perceptual features are minimally degraded.
• Saliency Metric Recovery: On standard datasets with attacks such as FGSM, SAD closes up to 60–80% of the loss in EMD, CC, NSS, KLD, and SIM compared to applying a global defense.
• Modularity: The block size for compression, saliency model, and JPEG quality schedule are all replaceable, providing broad flexibility.

Limitations:

• Hyperparameter Sensitivity: Effectiveness is sensitive to the choice and ordering of the JPEG quality list $S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$6, and the $S_8(x)(u,v) = \lfloor 255\cdot S(x)(u,v) \rfloor$7 block structure.
• Saliency Model Robustness: If the saliency estimator is vulnerable to adversarial manipulation, critical region selection may fail. Using deeper or ensemble saliency models may ameliorate this risk.
• Limited Efficacy on Localized Attacks: Small-magnitude or highly localized attacks (e.g., DeepFool inside saliency peaks) are less mitigated by SAD; more aggressive or hybrid denoising strategies may be required for these cases.

A plausible implication is that the success of SAD is closely tied to the reliability of the saliency prediction pipeline, especially under adversarial conditions.

6. Comparative Perspective and Usage Context

SAD is applicable as a pre-processing front-end for vision systems at risk of adversarial manipulation, where perceptual fidelity and robustness to attack are both operational requirements. Its design integrates smoothly with JPEG-based image pipelines, making adoption practical in settings with legacy encoding infrastructure. While SAD demonstrates pronounced benefits in saliency preservation under global attack scenarios, performance against sophisticated or highly localized attacks remains bounded by saliency estimation quality and the compressive nature of JPEG.

Compared to global or naive denoising schemes, SAD provides a nuanced and adaptive capability, preserving critical content while degrading adversarial patterns. The compatibility with arbitrary saliency models and block-wise quantization schedules positions SAD as a flexible, defense-agnostic framework in adversarial robustness research (Tran et al., 2020).