Distance-Based Randomized Smoothing

• Distance-based first-order randomized smoothing is a defense framework that certifies discrete sequence classifiers against adversarial insertions, deletions, and substitutions.
• It employs randomized deletion sampling and first-order linear approximations to derive certified robustness guarantees based on edit-distance metrics.
• Empirical evaluations, such as in malware detection, demonstrate a strong trade-off between clean accuracy and certified robustness radius.

Distance-based first-order randomized smoothing encompasses a class of certified defenses for discrete sequence classifiers, wherein robustness guarantees are derived against adversarial edits measured by a sequence metric such as Levenshtein (edit) distance. Unlike smoothing for continuous domains with $\ell_p$-norm constraints, these methods specifically address the adversarial threat model of insertions, deletions, and substitutions in discrete or variable-length data, notably source code and raw binary sequences. A principal instantiation of this approach is the RS-Del mechanism, which constructs robustness certificates via the randomized application of deletion edits, leveraging first-order linear bounds to relate adversarial edit distance to prediction invariance.

1. Formalism of Randomized Deletion Smoothing

Let $\Omega$ denote a finite alphabet and $\mathcal{X}=\Omega^*$ the space of all finite sequences. Given a base classifier $f:\mathcal{X}\rightarrow\{0,1,\dots,K-1\}$, randomized smoothing constructs a smoothed classifier $\bar f$ via the sampling of random deletions:

• For sequence $x\in\mathcal{X}$ with length $|x|$, index positions by $[x]=\{1,\dots,|x|\}$.
• A deletion edit $E\subseteq[x]$ selects the positions to survive; the resulting subsequence is $z = \text{apply}(x,E)$.
• The deletion distribution $G(x)$ independently deletes each position $i$ with probability $p\in(0,1)$, such that:

$$\Pr[G(x)=E] = \prod_{i=1}^{|x|}\;p^{\,1_{\{i\notin E\}}}(1-p)^{\,1_{\{i\in E\}}}$$

• The smoothing distribution $\phi(x)$ over edited sequences is:

$$\Pr[\phi(x)=z] = \sum_{E\subseteq[x]}\Pr[G(x)=E]\cdot1_{\{\text{apply}(x,E)=z\}}$$

• Smoothed class probabilities are given by:

$p_y(x) = \Pr_{z\sim\phi(x)}[f(z)=y]$

with optional class-specific thresholds $\eta_y$. The predicted class is $\hat y = \arg\max_y \{p_y(x) - \eta_y\}$.

This framework enables robustness certification against edit-based adversaries, notably those bounded by Levenshtein distance.

2. Certification of Edit-Distance Robustness

Robustness certification is achieved by establishing that the class prediction $\hat y$ remains invariant under any input $x'$ within an edit distance $r$ of $x$, where edits include insertions, deletions, and substitutions.

Main theorem (edit-distance guarantee):

• Let the winning class be $y = \bar f(x)$ with confidence $\mu_y = p_y(x)$.
• Define

$$\nu_y = \begin{cases} \tfrac12+\eta_y-\min_{y'\neq y}\eta_{y'}, & \eta_y\ge\min_{y'\neq y}\eta_{y'},\;K>2 \ 1 +\eta_y-\min_{y'\neq y}\eta_{y'}, & \eta_y<\min_{y'\neq y}\eta_{y'},\;K>2 \ \frac{1+\eta_y-\min_{y'\neq y}\eta_{y'}}{2}, & K=2 \end{cases}$$

• The certified radius is

$$R = \left\lfloor\frac{\log(1+\nu_y-\mu_y)}{\log p}\right\rfloor$$

guaranteeing $\bar f(x')=y$ for all $x'$ with $d_{Lev}(x',x)\leq R$.

The relationship between the smoothed confidence and edit distance is further quantified using longest common subsequence (LCS) bounds:

$$p_y(x') \geq p^{|x'|-|x|}\left(\mu_y -1 + p^{\tfrac12(d_{LCS}(x',x)+|x|-|x'|)}\right)$$

For all $x'$ within Levenshtein distance $r$, a conservative lower bound is $\rho(x;\mu_y)\geq\mu_y-1+p^r$, from which $R$ is derived by requiring $\rho(x;\mu_y)\geq\nu_y$.

3. First-Order Approximation and Monotonicity

Certificate computation harnesses two monotonicity properties under adversarial edits, constituting what may be termed a first-order bound (Editor's term):

• Each symbol deletion reduces the smoothed class probability $p_y(x)$ by a factor of $p$ per deletion.
• Insertions and substitutions degrade the LCS overlap, incurring additional exponentiation in the bound as $p^{\,\#\text{insertions}}$.

These linear-in-the-exponent relationships approximate the Neyman-Pearson lemma, which is otherwise computationally intractable for discrete sequence editing threat models.

4. Sampling-Based Certification Algorithm

Certification is performed via Monte-Carlo approximation, significantly more scalable than exhaustive enumeration, which is intractable ($2^{|x|}$ subsets):

1. Draw $n_{\text{pred}}$ samples $z_i\sim\phi(x)$, tally base-model votes to estimate $\hat\mu_y$ for all $y$.
2. Predict $\hat y = \arg\max_y(\hat\mu_y - \eta_y)$.
3. Draw $n_{\text{bnd}}$ further samples to obtain $(1-\alpha)$ lower confidence bound (LCB) on true $\mu_{\hat y}$.
4. Compute largest $R$ satisfying $\rho(x;\text{LCB})\geq\nu_{\hat y}$; abstain if $\text{LCB}<\eta_{\hat y}$, otherwise certify radius $=R$.

Computational complexity is $O((n_{\text{pred}}+n_{\text{bnd}})\cdot T_f)$, where $T_f$ is the cost of one base-model inference on a sequence of expected length $(1-p)|x|$. Empirical practice suggests $n_{\text{pred}},n_{\text{bnd}}\approx 10^3$--$10^4$ yield reliable certificates.

5. Empirical Evaluation: Malware Detection Case Study

RS-Del was evaluated on the MalConv binary classifier for raw-byte Windows executables, input length truncated/padded to 2MB, with deletion probability $p=99.5\%$. Key results include:

• Certified accuracy at edit-distance radius $128$ bytes: $91\%$ (fraction of files both correctly classified and provably robust at that radius).
• Trade-off profile on Sleipnir2 dataset: At $p=99.5\%$, clean accuracy was $96.5\%$, with median certified radius $137$ bytes ($\approx0.06\%$ of file).
• Increasing $p$ above $99.5\%$ increases certificate radius but decreases clean accuracy.
• RS-Del achieves strictly higher certified accuracy versus certificate radius compared to randomized-ablation smoothing (which exclusively targets Hamming-distance substitutions), highlighting effectiveness for the larger Levenshtein threat model.

6. Comparison With Related Smoothing Frameworks

The RS-Del mechanism represents a notable extension of randomized smoothing to discrete domains, directly addressing adversarial edits quantified by edit distance, in contrast to prior work limited to $\ell_p$-norm or Hamming distance. Importantly, deleting alone suffices to confer robustness to deletions, insertions, and substitutions due to the role of longest common subsequence. This suggests randomized smoothing can be adapted to variable-length discrete data with appropriately chosen edit distributions and threat models.

Randomized smoothing in continuous regimes yields robustness to $\ell_p$ adversaries; RS-Del generalizes this logic to sequence classifiers, substituting deletion operations for additive Gaussian noise. The first-order nature of the certificate reflects a linear approximation suitable for practical large-scale certification. Comparison with alternative frameworks reveals that the RS-Del mechanism offers enhanced certified accuracy particularly in domains where edit-based adversaries are realistic and impactful, such as malware detection.

7. Limitations and Practical Considerations

The RS-Del certificate's efficacy depends critically on the choice of deletion probability $p$ and base classifier reliability. Higher $p$ values yield larger certified radii but may reduce clean accuracy. The Monte-Carlo nature of the algorithm enables scalability on sequences of practical length but does so with only a first-order bound, not a tight Neyman-Pearson certificates obtainable in specific continuous settings.

Empirical results demonstrate strong performance in malware detection, but the approach may be constrained by domain-specific requirements such as sequence alphabet size and application threat models. A plausible implication is that while randomized smoothing via deletion supports edit-distance certification, the trade-off between clean accuracy and certified robustness must be closely managed in deployment, and domain adaptation may be necessary for non-binary or highly structured sequence data.