Randomized Smoothing of All Shapes and Sizes (2002.08118v5)

Abstract: Randomized smoothing is the current state-of-the-art defense with provable robustness against $\ell_2$ adversarial attacks. Many works have devised new randomized smoothing schemes for other metrics, such as $\ell_1$ or $\ell_\infty$; however, substantial effort was needed to derive such new guarantees. This begs the question: can we find a general theory for randomized smoothing? We propose a novel framework for devising and analyzing randomized smoothing schemes, and validate its effectiveness in practice. Our theoretical contributions are: (1) we show that for an appropriate notion of "optimal", the optimal smoothing distributions for any "nice" norms have level sets given by the norm's Wulff Crystal; (2) we propose two novel and complementary methods for deriving provably robust radii for any smoothing distribution; and, (3) we show fundamental limits to current randomized smoothing techniques via the theory of Banach space cotypes. By combining (1) and (2), we significantly improve the state-of-the-art certified accuracy in $\ell_1$ on standard datasets. Meanwhile, we show using (3) that with only label statistics under random input perturbations, randomized smoothing cannot achieve nontrivial certified accuracy against perturbations of $\ell_p$-norm $\Omega(\min(1, d^{\frac{1}{p} - \frac{1}{2}}))$, when the input dimension $d$ is large. We provide code in github.com/tonyduan/rs4a.

• The paper demonstrates that optimal smoothing distributions possess level sets forming Wulff Crystals, significantly advancing the theoretical underpinnings of robust certification.
• The paper introduces two novel methods for computing robust radii, extending efficacy beyond the traditional ℓ2 norm to include ℓ1 and ℓ∞ perturbations.
• The paper shows practical performance gains on datasets like CIFAR-10 and ImageNet by integrating tailored distributions with stability training and semi-supervised learning techniques.

Randomized Smoothing of All Shapes and Sizes: Insights and Implications

The research presented in "Randomized Smoothing of All Shapes and Sizes" proposes a novel framework for devising and analyzing randomized smoothing schemes, an area crucial for enhancing the robustness of machine learning models against adversarial attacks. This work significantly advances the theoretical understanding and practical application of randomized smoothing by addressing its limitations and exploring new possibilities across various norms.

Theoretical Contributions

The paper provides several key theoretical insights into randomized smoothing.

1. Wulff Crystals and Optimal Smoothing Distributions: It establishes that for "nice" norms, the optimal smoothing distributions have level sets forming Wulff Crystals. This connection to Wulff Crystals—a concept from physics involving equilibrated crystal structures—suggests that smoothing distributions can be optimized by aligning their geometric properties with those of the norm in question. This insight allows for more efficient design and analysis of randomized smoothing strategies.
2. New Methods for Deriving Robust Radii: Two novel methods are proposed for calculating robust radii from any given smoothing distribution. These methods significantly enhance the ability to certify robustness across different norm types, such as $\ell_1$ and $\ell_\infty$, beyond the traditionally studied $\ell_2$ norm. These techniques underscore the flexibility of the approach and offer a framework for exploring robustness certificates in a broader scope.
3. Fundamental Limits via Banach Space Cotypes: The paper defines fundamental constraints on current randomized smoothing techniques using Banach space cotypes. This theoretical boundary delineates the limitations of achieving nontrivial certified accuracy for high-dimensional inputs under various $\ell_p$-norm perturbations. It elucidates that as the input dimension increases, existing smoothing techniques face significant challenges, especially beyond $\ell_2$ perturbations, thus pinpointing areas for further investigation in advanced smoothing models.

Practical Implications and Performance Enhancements

The practical efficacy of the proposed framework is evidenced by its improvement over the state-of-the-art certified accuracies for $\ell_1$ on standard datasets such as CIFAR-10 and ImageNet. The results demonstrate that by choosing appropriate smoothing distributions—specifically those aligning with the geometry of the Wulff Crystal—substantial gains in robust accuracy can be achieved.

• For $\ell_1$ adversaries, adopting uniform or specifically tailored Wulff Crystal distributions enables models to attain robustness guarantees that other techniques cannot match efficiently.
• The research further incorporates stability training and semi-supervised learning techniques to enhance model performance, suggesting practical pathways for integrating more data and training strategies.

Speculating Future Directions

While this research markedly enriches the understanding and capability of randomized smoothing, it also highlights the inherent challenges in scaling robustness guarantees for high-dimensional data and under alternative norms ($\ell_1, \ell_\infty$). Future research may focus on:

• Developing more efficient computational techniques to leverage the full potential of Wulff Crystal-based distributions for real-world applications.
• Exploring extensions of randomized smoothing beyond classical norm settings, potentially involving distributionally robust optimization methods that consider complex perturbation models.
• Investigating hybrid approaches that can balance the expressive power and computational requirements, particularly as input dimensions continue to grow.

The blending of geometric insights with statistical robustness in this work not only forwards adversarial defense but also opens avenues for a more principled development of robust machine learning models. The research provides a rich platform for further theoretical exploration and practical integration into diverse AI systems.