--- title: Anytime-Valid Certified Robustness url: https://www.emergentmind.com/topics/anytime-valid-certified-robustness type: topic --- # Anytime-Valid Certified Robustness Searching arXiv for the cited papers and closely related randomized smoothing work. arXiv search query: 2606.27698 OR 2606.27694 OR 2605.20270 OR randomized smoothing Cohen 2019 Anytime-valid certified robustness denotes a family of certification procedures in which robustness guarantees remain valid under optional stopping, adaptive sampling, and continuous monitoring. Across the settings considered in recent work, the common structure is a nonnegative martingale or supermartingale—typically an e-process or test martingale—combined with Ville’s inequality to obtain time-uniform control. In randomized smoothing for image classifiers, this yields confidence sequences for class probabilities and certified radii that can be updated after every noisy evaluation [2606.27694]. In automatic speech recognition (ASR), the same statistical machinery is used to certify token existence and adversarial exclusion, then to certify a sentence through a rank-based tournament over filtered candidates, producing a certified transcript $\hat Y$ and robustness radius $R$ under additive Gaussian noise [2606.27698]. In deployment-time control for RLVR-trained LLMs, an e-process per release threshold yields anytime-pathwise certificates that the verifier-measured failure rate among released outputs remains below a contractual risk budget $\alpha$ up to a vanishing-in-$N_T$ slack [2605.20270]. ## 1. Definition and problem setting Certified robustness, in the classifier setting, means that for a classifier $F$ and norm $p$, a sample $x$ is certified robust at radius $r$ if $F(x)=F(x')$ for all $x'$ in the ball $B_p(x,r)=\{x':\|x-x'\|_p\le r\}$ [2606.27694]. Randomized smoothing instantiates this by defining a smoothed classifier $$ g(x)=\arg\max_c P(f(x+\epsilon)=c), \qquad \epsilon\sim N(0,\sigma^2 I), $$ and then mapping lower and upper probability bounds to an $\ell_2$ radius through Gaussian smoothing formulas [2606.27694]. The same notion is generalized in sequence and deployment settings, but the certified object changes. For ASR, the final output is a certified transcript $\hat Y$ together with a robustness radius $R$ such that, with probability at least $1-\alpha$, $\hat Y$ is invariant to any perturbation with $\ell_2$ norm less than $R$ [2606.27698]. For RLVR deployment, the certified object is not a per-input perturbation radius but a per-deployment guarantee on selective risk: at every time $T$, the verifier-measured failure rate among released outputs is bounded by $\alpha$ plus a slack that shrinks with the number of acted rounds $N_T$ [2605.20270]. A central motivation is that fixed-horizon procedures are poorly matched to adaptive deployments. Classical randomized smoothing requires large, precommitted sample sizes and suffers from the peeking problem if one stops early; offline conformal-risk methods depend on exchangeability; online-conformal methods control long-run averages rather than pathwise risk; and standard ASR evaluation is difficult in deployment because oracle transcripts are unavailable [2606.27694] [2605.20270] [2606.27698]. Anytime-valid procedures replace fixed-horizon guarantees with uniform-in-time guarantees that tolerate monitoring and data-dependent stopping. ## 2. Statistical foundations: e-values, test martingales, and confidence sequences The technical core is a nonnegative e-value or e-process. In the sequential setting, if $E_t$ is an e-value at time $t$, wealth is accumulated as $$ W_t=\prod_{s=1}^t E_s. $$ By Ville’s inequality, $$ P(\exists t: W_t\ge 1/\alpha)\le \alpha, $$ so threshold crossing yields valid rejection at any time and therefore optional-stopping validity [2606.27698]. The same logic appears in the image-certification setting, where an e-process $\{E_t\}$ is a nonnegative supermartingale under the null with $E[E_t]\le 1$ for all $t$, and tests based on thresholding are anytime-valid [2606.27694]. In RLVR deployment, a Ville-type e-process is maintained for each release threshold on a Bonferroni grid, evaluated against the RLVR filtration, yielding pathwise guarantees on the realized stream without exchangeability and without pooling across deployments [2605.20270]. A second common ingredient is inversion. In ASR, likelihood-ratio martingales $$ L_T(p)=\prod_{t=1}^T \frac{\mathrm{Bernoulli}(W_{w,t};p)}{\mathrm{Bernoulli}(W_{w,t};0.5)} $$ are inverted to obtain anytime-valid lower and upper confidence sequence bounds $\underline p_{w,T}$ and $\overline p_{w,T}$ for token-inclusion probabilities $p_w$ [2606.27698]. In sequential randomized smoothing, the set of nonrejected nulls at time $t$ forms a confidence set $$ C_t=\{p_0\in[0,1]: \max_{\tau\le t} W_\tau(p_0)<1/\alpha\}, $$ and the anytime-valid lower confidence bound is $\underline p_t=\inf C_t$; mapping $\underline p_t$ through $r_t=\sigma\Phi^{-1}(\max\{\underline p_t,1/2\})$ yields a time-uniform certified radius [2606.27694]. In RLVR deployment, the inversion is threshold-oriented rather than probability-oriented: the certified set is the set of thresholds whose e-processes have crossed their Bonferroni-adjusted boundaries, and deployment uses the most permissive certified threshold [2605.20270]. | Setting | Certified object | Sequential mechanism | |---|---|---| | Randomized smoothing | lower confidence bound on $p$ and radius $r_t=\sigma\Phi^{-1}(\max\{\underline p_t,1/2\})$ | mixture e-processes over Bernoulli trials | | ASR | token existence/exclusion, winning sentence $\hat Y$, end-to-end radius $R$ | Two-Sided Atomic Audit and Rank-Based Tournament | | RLVR deployment | per-deployment selective risk $R_T^{act}$ | e-process per threshold with max-certified-threshold rule | These constructions share a common interpretation: evidence is accumulated multiplicatively, stopping is triggered by wealth crossing, and the resulting certificate remains valid under peeking. This suggests that anytime-valid certified robustness is best understood not as a single robustness definition but as a sequential certification paradigm whose certified quantity depends on the prediction object and threat model. ## 3. Randomized smoothing as the canonical continuous-radius instance The most direct instantiation is sequential randomized smoothing for image classification. For a fixed input $x$ and target class $c_A$, Bernoulli trials are defined by $$ X_i=\mathbf 1\{f(x+\epsilon_i)=c_A\}, \qquad \epsilon_i \ \text{iid}\sim N(0,\sigma^2 I), $$ with success probability $p=P(X_i=1)$ [2606.27694]. Classical randomized smoothing estimates $p$ by Monte Carlo and then uses fixed-sample binomial confidence intervals such as Clopper–Pearson. The limitations described in recent work are that classic RS requires fixed horizons, often $N=10{,}000$ noisy evaluations per input, and invalidates coverage under peeking [2606.27694]. The anytime-valid alternative replaces fixed-sample confidence intervals with mixture e-processes. For a point null $H_0:p=p_0$ and alternative $H_1:p=q$, the pointwise likelihood-ratio e-value is $$ E_i=\frac{q^{X_i}(1-q)^{1-X_i}}{p_0^{X_i}(1-p_0)^{1-X_i}}, $$ and the associated wealth process is $$ W_t(p_0;q)=\frac{q^{h_t}(1-q)^{t-h_t}}{p_0^{h_t}(1-p_0)^{t-h_t}}, $$ where $h_t=\sum_{i=1}^t X_i$ [2606.27694]. Since $q$ is unknown, the method integrates over a prior $Q$, yielding a mixture e-process $\overline W_t(p_0)$. With a Beta prior this has closed form through Beta functions, and the paper further uses a mixture of truncated Beta components to focus mass on regions most relevant for certification, such as the robust region $[0.5,1]$ [2606.27694]. A notable contribution is the use of a lightweight meta-learner to predict image-specific priors. The meta-learner $M_\theta$ takes a penultimate-layer embedding $\phi(x)$, clean-image softmax summary, and a small Phase I glimpse of noisy evaluations, and outputs mixture weights $w_k$, Beta parameters $(\beta_k,\gamma_k)$, and optionally truncation bounds $[a_k,b_k]$ [2606.27694]. To guard against prior misspecification, the final e-process includes a fixed small weight $w_{\mathrm{safety}}=0.01$ on the KT prior; by convexity of e-values, validity is preserved [2606.27694]. The certification pipeline has two phases. Phase I identifies the target class and features, then discards those samples from the e-process to maintain supermartingale validity. Phase II updates the mixture e-process sequentially, inverts it to obtain $\underline p_t$, and computes $r_t$ [2606.27694]. Because the procedure is anytime-valid, it supports precision-based stopping, plateau exit, target-radius stopping, an adversarial/UCB exit for quick rejection, and a bankruptcy exit for compute-saving conservative rejection [2606.27694]. Empirically, the paper reports a 20-fold reduction in sample complexity compared to traditional methods while maintaining rigorous statistical guarantees, with viable certifications frequently completing in fewer than $500$ samples [2606.27694]. Relative to a KT prior baseline, meta-learning reduces samples by $8$–$15\%$ on average and tightens radii by up to about $4\%$ on ImageNet; for non-robust samples, early rejection policies yield up to $33\times$–$45\times$ speedups with only minor accuracy/radius penalties [2606.27694]. The paper also emphasizes an “anytime-valid tax”: anytime-valid intervals can be slightly wider than fixed-horizon Clopper–Pearson intervals for the same $N$ [2606.27694]. A common misconception is therefore that anytime-validity is a free improvement over fixed-sample certification; the reported formulation instead trades some finite-$N$ tightness for optional-stopping validity and adaptive compute allocation. ## 4. Hierarchical certification for structured outputs in automatic speech recognition For ASR, majority-class certification collapses in sequence spaces because probability mass fragments across many outputs. The proposed solution is hierarchical aggregation: certify atomic content first, then certify a sentence assembled from those atomic decisions [2606.27698]. Under additive white Gaussian noise $\epsilon\sim N(0,\sigma^2 I)$, for a token $w$ in the vocabulary $\mathcal X$, the inclusion probability is $$ p_w=P(w\in f(x+\epsilon)). $$ A token existence certificate rejects the null $H_{\mathrm{pos}}: p_w\le 0.5$, while an adversarial exclusion certificate rejects $H_{\mathrm{neg}}: p_w\ge 0.5$ [2606.27698]. The Two-Sided Atomic Audit is a sequential, two-sided e-value or martingale audit over tokens discovered in an initial sample budget. With betting fraction $\lambda\in(0,2]$ and token indicator $W_{w,t}\in\{0,1\}$, the updates are $$ E_{\mathrm{pos},t}(w)=E_{\mathrm{pos},t-1}(w)\cdot \bigl(1+\lambda(W_{w,t}-0.5)\bigr), $$ $$ E_{\mathrm{neg},t}(w)=E_{\mathrm{neg},t-1}(w)\cdot \bigl(1+\lambda(0.5-W_{w,t})\bigr), $$ initialized at $1$ [2606.27698]. Tokens crossing $1/\alpha_{\mathrm{atomic}}$ become part of the certified vocabulary $V_{\mathrm{cert}}$ or the excluded vocabulary $V_{\mathrm{excl}}$. Confidence sequences derived from likelihood-ratio martingales then map to token-level radii $$ r_w=\sigma\Phi^{-1}(\underline p_{w,T}) \quad \text{for inclusion}, $$ and $$ r_w=\sigma\Phi^{-1}(1-\overline p_{w,T}) \quad \text{for exclusion}, $$ with $R_{\mathrm{atomic}}=\min_{w\in V_{\mathrm{cert}}\cup V_{\mathrm{excl}}} r_w$ [2606.27698]. Sentence certification is handled by a Rank-Based Tournament. New noisy transcripts are filtered through $V_{\mathrm{cert}}$, and the top-$K$ most frequent unique filtered sequences form the candidate set. At each step, the candidate with minimum WER to the current filtered sample is declared the winner, and candidate e-values are updated as $$ E_{i,t}=E_{i,t-1}\cdot \bigl(1+\lambda(\mathbf 1(i=i^*)-1/K)\bigr). $$ The average wealth $$ \overline E_{\mathrm{tourn},t}=\frac{1}{K}\sum_i E_i $$ is a nonnegative martingale; stopping when it crosses $1/\alpha_{\mathrm{tourn}}$ yields a sentence-level certificate without paying a multiplicity penalty linear in $K$ [2606.27698]. The winner is $\hat Y=\arg\max_i E_i$, and the sentence-level radius is $$ R_{\mathrm{tourn}}=\frac{\sigma}{2}\bigl(\Phi^{-1}(\underline p_{\mathrm{winner}})-\Phi^{-1}(\overline p_{\mathrm{runner\text{-}up}})\bigr). $$ The end-to-end radius is $R=\min(R_{\mathrm{atomic}},R_{\mathrm{tourn}})$, with coverage at least $1-(\alpha_{\mathrm{atomic}}+\alpha_{\mathrm{tourn}})$ [2606.27698]. The empirical setting comprises LibriSpeech and Common Voice 17.0 English; Whisper-Large-v3, Whisper-Small, HuBERT-Large, and Wav2Vec2-Large; AWGN at $\mathrm{SNR}\in\{10,5,0,-5\}$ dB; and budgets $N_1=50$, $N_C=1000$, $K=5$, $N_3$ up to $250$, with $\alpha=0.01$, $\lambda_{\mathrm{inst}}=0.5$, and $\lambda_{\mathrm{tourney}}=0.2$ [2606.27698]. Reported outcomes include certification recall at $-5$ dB SNR ranging from $40.5$–$74.0\%$ versus approximately $0$–$2\%$ for baselines, and a Whisper-Large-v3 WER reduction from $0.273$ to $0.126$, approximately a $54$–$55\%$ relative reduction [2606.27698]. Certified radius is reported to correlate inversely with WER and to remain informative when baselines’ recall collapses to zero; anytime-stopping yields up to $28\%$ compute savings versus fixed-budget on autoregressive models and is $17$–$20\%$ faster than ROVER for CTC models [2606.27698]. The ASR formulation also clarifies an important distinction. The robustness claim is with respect to an AWGN smoothing model and is mapped to $\ell_2$ perturbations; the paper explicitly notes that real-world perturbations may be non-Gaussian or psychoacoustic, and robustness may not fully reflect human perception [2606.27698]. Another limitation is multiplicity: atomic decisions use per-token thresholds at $1/\alpha_{\mathrm{atomic}}$, giving local guarantees but not family-wise error control across the vocabulary [2606.27698]. ## 5. Deployment-time selective-risk certification for RLVR-trained LLMs A second major line of work extends anytime-valid certified robustness from perturbation robustness to deployment-time risk control. In this setting, a specialist LLM is fine-tuned with reinforcement learning from verifiable rewards on operator-local data, a deterministic verifier returns a binary signal $V_t\in\{0,1\}$ at each round, and a gate releases or abstains [2605.20270]. The operator requires that, on this deployment stream and at every time, the verifier-measured failure rate among released outputs does not exceed a contractual budget $\alpha$, up to a small slack that vanishes with the number of released outputs [2605.20270]. The central statistic is the gated excess-risk increment $$ X_t(q):=A_t(q)\bigl((1-V_t)-\alpha\bigr)\in\{-\alpha,0,1-\alpha\}, $$ where $A_t(q)=\mathbf 1\{S_t\le q\}$ is the release rule induced by threshold $q$ on a predictable calibrated score $S_t$ [2605.20270]. Under the unsafe null at threshold $q$, the conditional mean of $X_t(q)$ is nonnegative. On epoch $j$, the e-process is $$ E_{j,t}(q):=\prod_{s=\tau_j}^t \bigl(1-\lambda_{j,s}(q)X_s(q)\bigr), $$ with predictable betting fraction $\lambda_{j,t}(q)$ clipped to $[0,(1-\alpha)^{-1}]$ [2605.20270]. Proposition-level validity states that under the unsafe null and predictability, $\{E_{j,t}(q)\}$ is a nonnegative supermartingale, so Ville’s inequality gives $$ P\Bigl(\sup_{t\in I_j}E_{j,t}(q)\ge 1/\delta\Bigr)\le \delta. $$ CSA maintains such an e-process for each threshold in a finite grid $Q=\{q^{(1)}<\cdots