Certified Semantic Smoothing (CSS)

• CSS is a rigorous statistical framework that certifies per-component robustness for complex models subjected to semantic and structured perturbations.
• It employs Gaussian randomized smoothing, one-sided binomial tests, and Holm’s sequential correction to guarantee stable, per-pixel or per-point predictions.
• Empirical evaluations on segmentation and point cloud tasks demonstrate scalable certification with high accuracy and controlled false positive rates.

Certified Semantic Smoothing (CSS) is a rigorous statistical framework designed to provide provable robustness guarantees for complex machine learning models subjected to semantic, structured, or large-scale perturbations. CSS generalizes and extends classical randomized smoothing—originally developed for $\ell_p$-norm bounded input noise—to settings where each input yields thousands of correlated outputs (as in semantic segmentation), or where the threat model involves transformations not directly expressible via norm balls, such as semantic editing or adaptive adversarial attacks. The approach leverages randomized smoothing, abstention, and multiple-testing correction mechanisms to produce certified per-component (e.g., per-pixel or per-point) robustness guarantees, with direct control over global false certification rates, and provides the first practical and scalable certificates for real-world segmentation and related structured prediction tasks.

1. Formal Definition and Objectives

Let the base model $f: \mathbb{R}^{N\times m} \to \mathcal{Y}^N$ map an input $x = (x_1, ..., x_N)$ to a per-component prediction $f_i(x) \in \mathcal{Y}$. CSS constructs a smoothed predictor

$f_T(x) \in (\mathcal{Y} \cup \{\varnothing\})^N$

where each output $f_{T,i}(x)$ is either a concrete class label or indicates abstention ($\varnothing$). The core objectives of CSS are:

• Per-component certified robustness: For every component $i$, certify a radius $r_i$ such that, under any admissible perturbation $\delta$ with $\|\delta\|_2 \le r_i$, the class predicted for $x_i$ remains unchanged unless abstained;
• Statistical guarantees under multiple hypothesis testing: Control the family-wise error rate (FWER) over all $N$ tests, ensuring that the probability of any false certification is at most $\alpha$;
• Scalability: The method should function efficiently for large $N$ (e.g., $10^6$ pixels), neither blowing up in computational cost nor losing statistical power as the number of outputs increases (Fischer et al., 2021).

2. Algorithmic and Statistical Foundations

Randomized Smoothing for Structured Outputs

CSS employs Gaussian randomized smoothing: given input $x$, generate $n$ i.i.d. samples $\epsilon_j \sim \mathcal{N}(0, \sigma^2 I)$ and compute, for each component $i$ and class $c$,

$$n_{i,c} = \#\{\text{samples } j : f_i(x + \epsilon_j) = c\}$$

Let $\hat c_{A,i} = \arg\max_c n_{i,c}$ (most frequent label) and $n_i = n_{i,\hat c_{A,i}}$. The empirical "probability of stability" for $i$ is $$\hat p_{A,i} \approx P_{\epsilon}(f_i(x + \epsilon) = \hat c_{A,i})$$.

Robustness Certification and Per-Component Radius

If $$P_{\epsilon}\bigl(f_i(x+\epsilon) = \hat c_{A,i}\bigr) \geq T$$ for threshold $T \in (0.5, 1)$, the classic Gaussian smoothing result yields

$r_i = \sigma \Phi^{-1}(T)$

where $\Phi^{-1}$ is the standard normal quantile function. In practice, the true $p_i$ is unknown; instead, CSS performs a one-sided binomial test for $H_{0,i}: p_i \leq T$ and computes a $p$-value: $$\text{pval}_i = P(\mathrm{Binomial}(n, T) \geq n_i)$$ A small value indicates evidence $p_i > T$ (Fischer et al., 2021).

Multiple-Testing Correction

Correcting for the simultaneous certification of $N$ outputs, CSS implements the Holm sequential procedure:

1. Sort the $p$-values $\text{pval}_{(1)} \le ... \le \text{pval}_{(N)}$.
2. For $j=1$ to $N$, if $\text{pval}_{(j)} \leq \frac{\alpha}{N-j+1}$, reject $H_{0,(j)}$; otherwise, stop.

This controls the FWER at $\leq \alpha$ and is more powerful than naive Bonferroni (Fischer et al., 2021).

Abstention Protocol

CSS will abstain on component $i$ if the available evidence does not suffice for certification (i.e., if the binomial test is inconclusive). This design ensures that single insecure components (pixels/points) do not lower the global guarantee and provides a mechanism to balance robustness radius versus coverage.

3. Theoretical Guarantees

Per-Component Robustness Theorem

Let

$$\hat c_{A,i} = \arg\max_c P_\varepsilon(f_i(x+\varepsilon)=c)$$

If $$P_{\varepsilon}(f_i(x+\varepsilon)=\hat c_{A,i}) \geq T$$, then for every $\delta$ with $\|\delta\|_2 \le r_i = \sigma \Phi^{-1}(T)$, the model is robust at $i$: $f_i(x+\delta) = \hat c_{A,i}$ If a component does not meet the threshold (i.e., in the abstain set), no claim is made for that component (Fischer et al., 2021).

Simultaneous Inference under Multiple Testing

With per-component binomial tests at level $\alpha$ and Holm's correction, the procedure ensures (with probability $\geq 1-\alpha$) that no certified (i.e., non-abstained) prediction is a false positive. All retained (non-abstained) outputs are robust to $\ell_2$ perturbations within their certified radii (Fischer et al., 2021).

4. Practical Implementation and Empirical Performance

CSS employs Monte Carlo (MC) sampling for each output, with typical $n$ in $100$–$1000$ range. Using $HRNetV2$ on Cityscapes, with $\|\delta\|_2 \le 0.5$, certified per-pixel accuracy reaches $\approx88\%$ with mean IoU $\approx0.60$; abstention rate is $\approx10\%$. At higher perturbations ($\|\delta\|_2 \le 1.0$), certified accuracy is $\approx78\%$ with $20\%$ abstention. The procedure is scalable: e.g., certification for $1024 \times 2048$ images completes within $30$–$1000$ seconds per image on a single GPU (Fischer et al., 2021).

Multiple-testing correction using Holm's method adds negligible computational overhead ($\ll0.1$s/image) while improving certified coverage by up to $2\%$ compared to Bonferroni.

On point cloud segmentation (ShapeNet, PointNetV2), CSS certifies $\approx62\%$ accuracy with $32\%$ abstention for $\|\delta\|_2 \le 0.25$; surface normals can increase metrics by $\sim10\%$.

The abstention mechanism is critical for preventing error propagation: increasing $T$ (threshold) or reducing $n$ (samples) trades off certification radius versus retained coverage.

5. Comparative Perspective and Extensions

CSS represents the first statistically sound, scalable framework for certifying the $\ell_2$-robustness of segmentation models in settings with thousands to millions of outputs and intricate dependencies between decisions. The methodology generalizes seamlessly to non-image domains, including 3D point clouds (Fischer et al., 2021).

Other CSS-inspired frameworks for structured data include randomized smoothing for ownership watermark verification in LLMs (embedding plus permutation smoothing) (Qiao et al., 17 Oct 2025) and smoothing for patch-robust segmentation using demasked views and majority voting (Yatsura et al., 2022).

Contrasting with prior smoothing for classification, which certifies a single label, CSS enables structured prediction tasks—where every component requires an individual, error-corrected certificate—without exponentially growing the global failure risk. Empirical results confirm that the framework closes the gap between theoretical robustness guarantees and deployment-scale tasks, providing certifiably robust segmentation on challenging datasets (Cityscapes, Pascal Context, ShapeNet) with nontrivial accuracy and tight FWER control (Fischer et al., 2021).

6. Limitations, Trade-Offs, and Future Directions

CSS’s coverage-abstention tradeoff is fundamental: aggressive robustness requirements increase abstention, potentially reducing overall semantic coverage. High MC sampling budgets are often needed for tight confidence bounds, impacting computational efficiency, though improvements such as more efficient statistical tests or improved sampling strategies could ameliorate these costs.

Extending CSS beyond $\ell_2$ (e.g., to semantic or structured perturbations not reducible to vector norms) remains an area of active research, where alternative smoothing distributions, problem-specific noise models, or compositional robustness via multi-domain smoothing may further increase coverage and radius. Future work could explore tighter and faster confidence calibration, joint training objectives to increase certifiable radii, and adaptation of CSS to complex input manifolds in vision, language, or multi-modal tasks.

---

Key Reference:

• "Scalable Certified Segmentation via Randomized Smoothing" (Fischer et al., 2021)