Feature Space Reservation (FSR)

• FSR is a neural network defense that separates and recalibrates adversarially perturbed feature maps to recover latent class-relevant information.
• It employs a light-weight separation net with Gumbel-Softmax-based mask generation to differentiate robust from non-robust activations.
• FSR integrates seamlessly into adversarial training pipelines, offering improved robustness with marginal computational overhead across benchmark datasets.

Feature Separation and Recalibration (FSR) is a neural network defense mechanism developed to improve adversarial robustness by explicitly processing intermediate feature maps to disentangle and recalibrate non-robust activations, rather than simply deactivating them. FSR targets the accumulation of adversarial perturbations in the feature space of deep neural networks and seeks to preserve discriminative signals that are often lost by conventional feature deactivation methods (Kim et al., 2023).

1. Motivation and Theoretical Foundations

Deep neural networks, when subjected to adversarial attacks, exhibit a compounding effect where small input-space perturbations ($\|\delta\|_\infty \leq \epsilon$) amplify through the network, corrupting intermediate feature maps $F \in \mathbb{R}^{C \times H \times W}$. Classical defenses such as Feature Denoising (FD), Channel Activation Suppression (CAS), and Channel-wise Importance-based Feature Suppression (CIFS) suppress or entirely deactivate those perturbed (“non-robust”) activations, resulting in improved robustness but with a significant loss of potentially discriminative features.

Empirical evidence and prior analyses (e.g., Ilyas et al. 2019) indicate that non-robust activations, while sensitive to adversarial perturbations, still encode residual class-correlated information. FSR builds on this insight, introducing a paradigm that (a) separates feature maps into robust and non-robust components, and (b) recalibrates the non-robust portion to recover salvaged discriminative cues, enhancing prediction fidelity under attack (Kim et al., 2023).

2. Feature Separation and Masking

Formally, for a given feature map $F \in \mathbb{R}^{C \times H \times W}$, FSR implements feature separation as follows:

• Robust features: $F_r = M_r \odot F$
• Non-robust features: $F_n = M_n \odot F$

where $M_r$, $M_n \in [0,1]^{C \times H \times W}$ are element-wise complementary masks ($M_n = 1 - M_r$), and $\odot$ denotes the Hadamard product.

The separation masks are produced by a light-weight “Separation Net” $S_{\theta_s}$, with $F \in \mathbb{R}^{C \times H \times W}$0. $F \in \mathbb{R}^{C \times H \times W}$1 is normalized via the sigmoid function $F \in \mathbb{R}^{C \times H \times W}$2, and then a soft binary mask $F \in \mathbb{R}^{C \times H \times W}$3 is sampled using a two-class Gumbel-Softmax:

$F \in \mathbb{R}^{C \times H \times W}$4

where $F \in \mathbb{R}^{C \times H \times W}$5 and temperature $F \in \mathbb{R}^{C \times H \times W}$6. For $F \in \mathbb{R}^{C \times H \times W}$7, $F \in \mathbb{R}^{C \times H \times W}$8 approximates a hard binary mask; for $F \in \mathbb{R}^{C \times H \times W}$9, it is differentiable, supporting end-to-end training.

3. Recalibration of Non-robust Activations

After separation, FSR targets the non-robust activations $F \in \mathbb{R}^{C \times H \times W}$0 for recalibration using a dedicated Recalibration Net $F \in \mathbb{R}^{C \times H \times W}$1, constructed as three Conv$F \in \mathbb{R}^{C \times H \times W}$2BatchNorm$F \in \mathbb{R}^{C \times H \times W}$3ReLU blocks (the final block omits ReLU):

$F \in \mathbb{R}^{C \times H \times W}$4

$F \in \mathbb{R}^{C \times H \times W}$5

Here, $F \in \mathbb{R}^{C \times H \times W}$6 is the output of the recalibration network. Only the non-robust region $F \in \mathbb{R}^{C \times H \times W}$7 is refined, allowing the block to restore latent class-salient information that conventional suppression schemes would discard.

4. Integration into Adversarial Training Pipelines

FSR is architecturally modular and can be inserted after arbitrary intermediate layers $F \in \mathbb{R}^{C \times H \times W}$8 of a backbone CNN. During training, adversarial examples $F \in \mathbb{R}^{C \times H \times W}$9 are generated (e.g., via PGD, FGSM). For each designated layer $F_r = M_r \odot F$0, the separated features $F_r = M_r \odot F$1 are computed using the Gumbel-Softmax-generated mask $F_r = M_r \odot F$2, recalibrated, then recombined for subsequent layer propagation:

$F_r = M_r \odot F$3

The modified feature map $F_r = M_r \odot F$4 feeds into the next network block.

The composite training objective combines:

• Classification Loss: $F_r = M_r \odot F$5 on final logits.
• Separation Loss: $F_r = M_r \odot F$6 (Equation 6 in text), using auxiliary heads $F_r = M_r \odot F$7 for $F_r = M_r \odot F$8 and labels $F_r = M_r \odot F$9 (ground truth), $F_n = M_n \odot F$0 (most likely incorrect class).
• Recalibration Loss: $F_n = M_n \odot F$1 (Equation 7), applied to $F_n = M_n \odot F$2.

Full objective:

$F_n = M_n \odot F$3

with typical choices $F_n = M_n \odot F$4, $F_n = M_n \odot F$5. Any adversarial training method (PGD, TRADES, MART) can be accommodated by substituting the appropriate $F_n = M_n \odot F$6.

5. Experimental Performance and Robustness

FSR was empirically validated on CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet using ResNet-18, VGG-16, and WideResNet-34-10. Evaluation attacks included FGSM, PGD-20, PGD-100, C&W ($F_n = M_n \odot F$7, $F_n = M_n \odot F$8), black-box methods TI-FGSM, DI-FGSM, NAttack, and AutoAttack. Results are summarized in the following table for ResNet-18 on CIFAR-10:

Method	Ensemble Acc. (%)	AutoAttack Acc. (%)
AT (Baseline)	45.5	44.1
AT + FSR	48.3	46.4
FD	45.8	44.6
CAS	46.5	44.2
CIFS	47.3	43.9

FSR achieved consistent absolute gains, with up to $F_n = M_n \odot F$9 improvement seen on SVHN/PGD-20 when added to TRADES.

A plausible implication is that FSR’s recalibration mechanism enables the retention and repair of class-relevant signals, extending robustness beyond what is achievable by strict deactivation methods such as FD, CAS, or CIFS. Consistency was observed across white-box, black-box, and ensemble AutoAttack protocols.

6. Comparative Analysis with Deactivation Approaches

Unlike prior methods that irreversibly suppress or zero out feature activations associated with adversarial vulnerability, FSR selectively recalibrates these regions. Feature Denoising (FD) applies blanket denoising filters, while CAS and CIFS score and deactivate channels, risking loss of residual discriminatory content. FSR’s auxiliary heads and mask-based partitioning facilitate discriminative learning within both robust and recalibrated non-robust subspaces (Kim et al., 2023).

Information preserved by recalibration, rather than discarded, appears critical. The performance differentials in Table 1 reflect this, with FSR outperforming all compared baselines under identical adversarial training and network settings.

7. Practical Implementation and Computational Efficiency

Implementation details include stochastic gradient descent (SGD) with momentum (0.9) and weight decay $M_r$0, learning rates of $M_r$1 (CIFAR/Tiny) and $M_r$2 (SVHN), decayed at epochs 75 and 90 over 100 epochs. Batch size is 128. FSR modules are inserted after block 4 in ResNet-18/VGG16 and block 3 in WideResNet; they consist of two three-block convolutional subnets (Separation and Recalibration) and small MLP-based auxiliary heads.

Overhead is marginal: $M_r$3 additional FLOPs and parameters (ResNet-18+FSR: 1.11 GFLOPs$M_r$41.15 GFLOPs, 11.17 M$M_r$512.43 M params), translating to 114 s$M_r$6120 s per epoch (CIFAR-10). This supports deployment in research and practical settings seeking robustness gains with minimal resource expenditure (Kim et al., 2023).

---

FSR represents a feature-space defense strategy that forgoes unconditional suppression in favor of integration and recalibration of adversarially distorted activations, demonstrating effectiveness across benchmark datasets, models, and both white- and black-box attack scenarios (Kim et al., 2023).