Higher-Order Certified Robustness for Regression

Abstract: Randomized smoothing has emerged as a scalable technique for certifying the adversarial robustness of classifiers. However, its application to regression remains under-explored and faces unique challenges. Existing regression certificates rely on probabilistic acceptance regions and fail to exploit the local geometry of the function. In this work, we present a novel framework for certified robust regression that addresses these limitations. We derive a prediction-centered certificate that guarantees the stability of the smoothed model's prediction and ensures practical computability at test time. We investigate several alternatives for constructing these certificates by explicitly incorporating means, variances, and gradients. In particular, we demonstrate on the MNIST rotation task that utilizing gradient information yields significantly tighter robustness certificates compared to the current state-of-the-art, alpha-smoothing.

• The paper introduces a prediction-centered certificate for regression that leverages local gradient information to tighten robustness guarantees.
• It employs variational calculus to derive analytical worst-case formulations, enabling efficient computation of certified radii under various constraint regimes.
• Empirical evaluations on MNIST and UTKFace show significantly improved certified radii and high robustness, surpassing earlier zeroth-order methods.

Higher-Order Certified Robustness for Regression: A Technical Analysis

Motivation and Background

Recent advances in randomized smoothing have enabled scalable, distribution-agnostic certification of adversarial robustness for classification networks. However, the extension of these guarantees to regression settings—where outputs are continuous-valued—remains unsatisfactory. Existing certificates for robust regression, such as $\alpha$-smoothing and RS-Reg, reduce model behavior to probabilistic inclusion in acceptance regions, discarding geometric information about function variation. This typically yields loose certificates and constrains guarantees to properties of the base predictor rather than the deployed smoothed regressor. The paper "Higher-Order Certified Robustness for Regression" (2607.05536) addresses these substantial technical limitations by developing a geometric, prediction-centered certifier for regression under randomized smoothing that explicitly leverages higher-order statistics, particularly local gradients.

Framework: Prediction-Centered Certificates with Higher-Order Information

The foundational goal is to certify that for any adversarial input perturbation $\|\delta\|_2 \leq R$, the change in the smoothed regression output $|g(z+\delta) - g(z)|$ is bounded by a prescribed tolerance $\epsilon$. This centers certification guarantees directly on the deployed (smoothed) prediction $g(z)$, rather than on the less meaningful base model output $f(z)$ or a probabilistic acceptance region.

Randomized smoothing is realized by defining $g(x) = E_e [f(x + e)]$ with $e \sim \mathcal{N}(0, \sigma^2 I)$. To bound the adversarial shift $|g(z+\delta) - g(z)|$, the analysis considers all possible base functions $f$ consistent with empirically estimable constraints (mean, variance, gradient norm), and characterizes the worst-case such $\|\delta\|_2 \leq R$0 using variational calculus.

Crucially, three constraint regimes are considered:

• Variance-only (unbounded): Constrains the scale of possible $\|\delta\|_2 \leq R$1 by variance $\|\delta\|_2 \leq R$2.
• Variance and gradient (unbounded): Adds a local sensitivity constraint using gradient $\|\delta\|_2 \leq R$3.
• Mean, variance, gradient, and bounds (bounded): Further imposes pointwise bounds (e.g., $\|\delta\|_2 \leq R$4) and considers the mean as an explicit parameter.

The core technical contribution is to solve, via calculus of variations, the nested maximization over $\|\delta\|_2 \leq R$5 and $\|\delta\|_2 \leq R$6 for each constraint regime, leading to analytical or efficiently computable certificates.

Analytical Results: Structure of the Worst-case Function

In the variance-only setting, the optimal $\|\delta\|_2 \leq R$7 is shown to be proportional to the likelihood ratio between the nominal and shifted smoothing distributions:

$\|\delta\|_2 \leq R$8

with scaling chosen to saturate the variance constraint. This yields an explicit closed-form for the certified radius:

$\|\delta\|_2 \leq R$9

Incorporating the gradient constraint, the optimal $|g(z+\delta) - g(z)|$0 is a sum of exponential and linear terms along the direction of $|g(z+\delta) - g(z)|$1, and the worst-case $|g(z+\delta) - g(z)|$2 aligns with $|g(z+\delta) - g(z)|$3. The resulting adversarial shift bound is:

$|g(z+\delta) - g(z)|$4

which is strictly tighter than the variance-only result.

For the bounded setting (where $|g(z+\delta) - g(z)|$5 is constrained to a box, and the mean matters), the problem reduces, via symmetry and alignment theorems, to a one-dimensional optimization over a function $|g(z+\delta) - g(z)|$6 defined on the projected subspace determined by $|g(z+\delta) - g(z)|$7. The optimal $|g(z+\delta) - g(z)|$8 is a clipped affine transformation of the likelihood ratio:

$|g(z+\delta) - g(z)|$9

where $\epsilon$0 and the multipliers are determined via dual optimization. This reduction makes high-dimensional regression certification tractable in practice.

Estimation and Practical Certification

Variance and gradient norm are estimated as U-statistics over Gaussian-perturbed samples. Confidence intervals are constructed using asymptotic normality, and a union bound ensures that the joint certificate (simultaneously valid for variance and gradient) achieves the prescribed statistical guarantee. While finite-sample validity does not match the exact binomial bounds in classification, the convergence analysis demonstrates high empirical tightness and coverage for realistic $\epsilon$1.

Empirical Evaluation

Quantitative experiments highlight several technically significant outcomes.

• Synthetic Function Certification: Across distinct representative functions (quadratic, piecewise linear, bounded), the higher-order certifier ((C,G)) achieves consistently larger mean radii than $\epsilon$2-smoothing under optimal parameter search, with soundness (true worst-case radius is always above the certificate).
• MNIST Rotation (Periodic Regression): On the canonical MNIST rotation angle estimation task, the method with all constraints ((E,C,G)+M) achieves a mean certified radius of $\epsilon$3 pixels (at $\epsilon$4), outperforming both the variance-only version (mean $\epsilon$5 pixels) and $\epsilon$6-smoothing (mean $\epsilon$7 pixels, but with 6% abstain rate).

Figure 1: CDFs of certified radii for the best configuration on MNIST rotation and aperiodic UTKFace age-estimation; higher-order methods substantially dominate $\epsilon$8-smoothing.

• Aperiodic Regression (UTKFace Age Estimation): Similar trends are observed; at $\epsilon$9, the higher-order certifier yields a mean certified radius 2.63× larger than $g(z)$0-smoothing (2.15 vs. 0.82).
• Certified Accuracy: For practical robustness, the method maintains high certified accuracy (95% absolute at $g(z)$1) and low mean error on certified points, supporting both utility and reliability.
• Tightness and Soundness: PGD-based adversarial search shows true radii are, on average, 3.4× larger than the certified guarantee, with <2% rate of empirical unsoundness, reflecting conservative but credible estimation.

  Figure 2: Tightness ratio distribution (optimization-based vs. certified radii) at $g(z)$2; only two samples fall below the soundness threshold (dotted line).

  

  Figure 3: Variance and squared gradient norm estimator convergence; CI bands confirm $g(z)$3 concentration, supporting interval validity for practical sample sizes.

  

  Figure 4: Certified radius convergence for a test point; empirical estimates approach the theoretical value as $g(z)$4 increases.

  

  Figure 5: Single-point convergence on UTKFace; all relevant statistics (variance, gradient norm, radius) converge to their population references as $g(z)$5 grows.

  

  Figure 6: Trade-off between certified radius and smoothed MAE across smoothing levels for UTKFace, quantifying the cost of increased robustness.

Implications and Future Directions

This work demonstrates that leveraging higher-order statistics—particularly gradients—enables significantly tighter, prediction-centered certified robustness for regression compared to prior zeroth-order/probabilistic approaches. By providing analytic forms for worst-case function behavior under smoothing, the methodology resolves the anchoring mismatch of earlier regression certificates and more fully exploits the structure of the smoothed landscape.

The theoretical implications are clear: geometric constraints propagate through smoothing to produce worst-case shifts driven by the gradient, and in turn, the multi-dimensional optimization collapses to the gradient-aligned direction. For practical deployment, this enables reliable robustness guarantees for safety-critical regression tasks such as pose estimation, sensor fusion, and medical prediction.

Key technical limitations include (1) reliance on asymptotic normality for moment estimation, not providing strict finite-sample guarantees, and (2) exponential scaling with input dimension in the worst-case behavior, as dictated by the curse of dimensionality for smoothing-based certificates [kumar2020curse, (Levine et al., 2020)]. Nevertheless, empirical error rates are well-controlled at practical sample sizes and dimensions.

Looking forward, further directions include:

• Extensions to heteroscedastic smoothing, vector-valued or manifold-valued outputs, and regression under different threat models;
• Integration with robust training objectives for end-to-end optimization of both base model and certificate sharpness;
• Theoretical characterization of minimal moment assumptions for non-asymptotic finite-sample validity in high-dimensional regimes.

Conclusion

"Higher-Order Certified Robustness for Regression" develops an analytically tractable, prediction-centered certification framework for smoothed regression models, establishing that higher-order statistics—specifically gradient information—yield substantially stronger guarantees than previous zeroth-order approaches. This work both advances the theoretical foundation of robust regression certification and delivers a practical, empirically validated method for certifying non-trivial robustness radii in high-dimensional regression models (2607.05536).