AMRM-Pure: Semantic-Preserving Adversarial Purification

Abstract: Adversarial purification is a defense technique that employs generative models to remove adversarial perturbations. Current methods often rely on powerful generators, typically diffusion models, and focus on reducing the gap between adversarial and clean samples in the feature space, while overlooking semantic correlation within a single sample. To address this issue, we explore adversarial purification from the perspective of preserving semantic relationships among image patches. We employ an Attentive Mask Reconstruction Model (AMRM), which shows superior performance. Our theoretical and experimental analysis reveals that AMRM is highly sensitive to adversarial noise, as such noise significantly distorts patch relationships. Based on this observation, we propose AMRM-Pure, a purification framework that denoises adversarial inputs by preserving patch-level semantics, and formulate this process as a tractable optimization problem with respect to the input. To further enhance robustness, we finetune AMRM-Pure with classification loss to strengthen semantic consistency. We apply our insight to two AMRM architectures, including Mask Autoencoder (MAE) and MaskDiT. Extensive experiments confirm the effectiveness of our method, establishing new state-of-the-art performance across multiple benchmarks.

• The paper proposes an adversarial purification method that minimizes intra-sample attention matrix variation to restore semantic integrity.
• It employs iterative PGD-based denoising to align adversarial examples’ attention maps with clean samples, resulting in improved reconstruction loss.
• Robust variants with classifier loss fine-tuning achieve state-of-the-art performance across benchmarks, outperforming traditional defense approaches.

Semantic-Preserving Adversarial Purification via Attention Matrix Variation Minimization

Motivation and Context

Adversarial robustness of DNNs remains a central issue, with adversarial training achieving strong defense but suffering from computational burden and notable robustness-generalization trade-offs. Adversarial purification, particularly generative model-based approaches, has emerged as an alternative, yet most prior work focuses on feature-space alignment between adversarial and clean samples using powerful generators such as diffusion models, while neglecting intra-sample semantic relationships, especially among image patches. This paper investigates the impact of adversarial perturbations on patch-level semantics and proposes a principled denoising framework exploiting inter-patch attention sensitivity, yielding substantial improvements in purification accuracy across benchmarks.

Theoretical Analysis of Patch-Level Semantic Disruption

The authors empirically and theoretically dissect how adversarial noise distorts intra-sample semantics in attentive masked reconstruction models (AMRM), exemplified by Masked Autoencoders (MAE). Even visually imperceptible perturbations induce significant changes in the internal attention matrices, as illustrated by the degraded MAE reconstructions of adversarially perturbed images—an effect not observed in pixel-level visualizations.

Figure 1: Demonstrates that adversarial perturbations destroy patch-level semantics—clean and adversarial images look identical, but MAE’s reconstructions diverge dramatically, and AMRM-Pure recovery aligns with the clean baseline.

Rigorous theoretical analysis establishes that the latent feature shifts induced by adversarial attacks result in an amplified variation of the attention matrices (AMV), with small perturbations yielding disproportionately large semantic distortions due to the structural properties of attention layers. A further theorem shows that the adversarial reconstruction loss in MAE is lower-bounded by the sum of the clean loss and AMV, confirming that attention disruption directly correlates with semantic misalignment and degraded denoising capability. These theoretical claims are supported by quantitative comparisons of reconstruction loss and AMV trends under attacks.

Figure 2: MAE reconstruction loss and attention matrix variation evolve in lockstep under AutoAttack and during purification, confirming the correlation and sensitivity established in theory.

Visualization of attention weights reveals the mechanism: for clean samples, masked patches are reconstructed by attending to semantically similar regions, whereas adversarial samples allocate high importance to irrelevant, distant patches; denoised samples via AMRM-Pure restore correct attention focus.

Figure 3: Visualizes attention weights for clean, adversarial, and denoised samples, showing adversarial noise misdirects attention across patches, while AMRM-Pure restores semantic locality.

AMRM-Pure: Attention Matrix Variation Minimization for Purification

Given the acute sensitivity of AMV to adversarial noise, AMRM-Pure is formulated as an optimization-based purification pipeline. The denoising objective is to minimize the distance between attention matrices of denoised adversarial inputs and corresponding clean samples. Since clean attention is unavailable during inference, the framework exploits reconstruction loss minimization (tractable surrogate for AMV minimization), iteratively refining adversarial samples via PGD to align their semantics with the clean distribution.

Figure 4: Schematizes AMRM-Pure pipeline: iterative denoising minimizes reconstruction loss, indirectly reducing attention variation and patch semantic distortion.

Two instantiations are advocated: AMRM-Pure$_{\text{MAE}}$ (MAE) and AMRM-Pure$_{\text{MaskDiT}}$ (MaskDiT, a diffusion-based AMRM). The method's convergence is theoretically characterized and empirically validated: increasing purification iterations reduces both AMV and reconstruction loss, progressively restoring patch-level semantics.

Robust AMRM-Pure: Fine-Tuned Classifier-Integrated Purification

Building on recent findings that classifier loss fine-tuning enhances purification robustness, robust variants (RAMRM-Pure$_{\text{MAE}}$, RAMRM-Pure$_{\text{MaskDiT}}$) are introduced. These use a two-stage protocol: adversarial examples are generated to attack the purifier-classifier system, then purification models are fine-tuned with classification loss on these examples. Empirical analysis using AMV as a metric demonstrates that robust purification models better preserve inter-patch semantics and achieve superior denoising performance.

Figure 5: Purification iterations progressively reduce attention matrix variation for RAMRM-Pure$_{\text{MAE}}$, demonstrating improved semantic consistency.

Empirical Results and Contradictory Claims

Comprehensive evaluations on CIFAR-10, CIFAR-100, SVHN, and ImageNet establish that the MaskDiT-based robust purification variant (RAMRM-Pure$_{\text{MaskDiT}}$) achieves the highest robust accuracy across all benchmarks and attack types (e.g., 75.83% robust accuracy on CIFAR-10 under $\ell_\infty$ attacks), surpassing both classical adversarial training—even those employing extensive extra data—and state-of-the-art diffusion-based purification (DiffPure, ADBM). The non-attentive CNN-based reconstruction baseline exhibits dramatically inferior robustness, supporting the claim that attention-driven patch-level semantics are essential for effective denoising—a notable contradiction to prior pixel-level approaches.

Practical and Theoretical Implications

By demonstrating that attention matrix variation directly governs MAE robustness and that AMRM-Pure optimizes this metric for purification, the authors substantiate a shift in the adversarial defense paradigm toward intra-sample semantic structure preservation. Practically, inference time of AMRM-Pure scales favorably on large datasets; fine-tuned robust purification models exhibit transferability across classifiers, outperforming DiffPure and related baselines even under adaptive, black-box, and unseen threat models. Theoretically, the formulation connects robust purification to weakly convex optimization landscapes in masked autoencoders, providing analytical guarantees on convergence and efficacy.

Figure 6: MaskDit loss, AMV, and perturbation trends are tightly coupled, extending the theoretical results to diffusion-based AMRMs.

Figure 7: Generalizes attention weight restoration for several samples, confirming cross-domain semantic preservation for denoised images.

Conclusion

AMRM-Pure introduces a principled adversarial purification framework exploiting the sensitivity of patch-level attention matrices to adversarial noise, validated both theoretically and empirically across AMRM architectures. Robust fine-tuning via classifier loss produces purification models with superior accuracy, transferability, and resistance to adaptive and unseen attacks. These results challenge pixel-level defense paradigms, demonstrating that inter-patch semantic preservation is key for robust generative denoising, and opening avenues for semantic-level adversarial defense mechanisms in future AI systems.