What Was That Again? Certified Robustness for Automatic Speech Recognition

Abstract: Automatic Speech Recognition systems are notoriously both sensitive to adversarial and benign perturbations. While this has been repeatedly demonstrated using reference datasets, detecting such behaviors in deployed systems is incredibly challenging, due to the absence of oracle knowledge of the true transcription. We demonstrate that employing a certification-inspired mechanism can significantly decrease WER, increase recall, and decrease the Spearman correlation between confidence and WER. We achieve this through a dual-gate diagnostic pipeline: a Two-Sided Atomic Audit that accumulates statistical wealth to certify both token existence and adversarial exclusion, and a Rank-Based Tournament that selects the winning sequence. Our evaluations across four diverse architectures demonstrate up to a 55% relative reduction in Word Error Rate, while also providing granular word- and sentence-level certifications to enhance acoustic security.

• The paper introduces a dual-gate certification framework that decomposes ASR robustness into token-level atomic tests and rank-based sequence tournaments using martingales.
• It empirically demonstrates significant reductions in word error rates and improved recall across multiple ASR architectures, especially in high-noise environments.
• The methodology offers anytime-valid E-value tests and actionable trust scores, supporting efficient real-time certification in safety-critical applications.

Certified Robustness in Sequence-to-Sequence Automatic Speech Recognition

Motivation and Context

Automatic Speech Recognition (ASR) systems, predominantly neural sequence-to-sequence architectures, are highly susceptible to both adversarial and benign perturbations in their input space. This fragility poses critical security risks in numerous real-world deployments, especially when ground-truth transcripts are unavailable for auditing system reliability. Conventional certified robustness frameworks, such as Randomized Smoothing (RS), have demonstrated efficacy in classification tasks but fail to scale to high-dimensional, non-categorical sequence outputs typical of ASR. The sequence space expands combinatorially under noise, leading to rapid degeneration in majority-class probability mass and rendering standard RS certificates vacuous.

This paper introduces a dual-gate certification framework leveraging E-value tournaments for anytime-valid certification of ASR outputs. The approach decomposes sequence certification into atomic (token-level) and structural (sequence-level) gates applying martingale-based wealth accumulation and competitive hypothesis testing. The framework achieves granular word and sentence certification, addresses fundamental combinatorial bottlenecks in sequence certification, and empirically yields substantial reductions in Word Error Rate (WER) and improved recall across diverse ASR architectures.

Methodology

Dual-Gate Certification Pipeline

The certification strategy is composed of two hierarchical stages:

1. Token-Level Atomic Certification: A preliminary candidate vocabulary is discovered through noisy sampling. Each token’s inclusion/exclusion is certified via wealth-compounding martingales, testing null hypotheses ($p_w \leq 0.5$ for inclusion, $p_w \geq 0.5$ for exclusion) using E-values, ensuring statistical validity at any sampling point by Ville’s inequality.
2. Rank-Based Structural Tournament: Filtered noisy transcriptions restricted to certified tokens are aggregated; the top-K most frequent filtered sequences form the candidate pool. Sequence-level certification is performed via competitive betting martingales, selecting the dominant sequence and certifying its robustness using confidence sequence inversion mapped to safety radii.

This hierarchical decomposition allows the certified radius to be defined as $R = \min(R_\text{atomic}, R_\text{tourn})$, which bounds perturbations under which the output transcription remains invariant with high probability.

Statistical Foundations and Anytime Validity

The use of E-values provides an expectation-constrained hypothesis testing setting immune to peeking and multiple comparison problems, allowing optimal stopping. This contrasts with frequentist lower-bound estimation (e.g., Clopper-Pearson) which is prone to computational inefficiency and overconservatism when monitoring sample statistics online. The martingale construction for both atomic and tournament gates is anytime-valid, yielding rigorous control over Type-I error unconstrained by sample size or peeking.

The shift from sequence alignment algorithms (e.g., ROVER, confusion networks) to martingale-based tournaments fundamentally reduces combinatorial multiplicity issues; certification evidence accumulation does not scale quadratically in sequence length, but rather with candidate sequence dominance, enabling viable sentence certification in high-noise regimes.

Empirical Evaluation

Experiments evaluated four ASR architectures—Whisper-large-v3, Whisper-small, HuBERT-large, and Wav2Vec2-large—on LibriSpeech and Common Voice datasets under varying Signal-to-Noise Ratio (SNR) conditions. The dual-gate pipeline was benchmarked against established baselines: Naive Cohen Randomized Smoothing and ROVER sequence alignment approaches.

Correlation Between Certified Radius and WER

There exists a strong positive correlation between certified radius and observed WER. This is observable across datasets and models, especially in production settings where ground truth is unavailable; the certified radius acts as a proxy for transcription stability and reliability.

Figure 1: Observed WER as a function of Certified Radius, evidencing a robust correlation on LibriSpeech and Common Voice.

Comparative Certification Performance

The dual-gate framework achieves superior recall and significant WER reductions at all SNR levels, especially in high-noise environments where traditional baselines collapse.

Figure 2: Relative Certification performance across approaches, demonstrating sustained certification recall for the tournament method.

At SNR of -5dB, baseline recall approaches zero, while tournament recall remains stable (40–74%). For Whisper-large-v3, WER improves from 0.273 (raw) to 0.126 (certified) signifying a 54% reduction. Notably, the tournament method provides actionable trust scores for every sample, not merely a certified subset, maintaining informative power even in extreme noise.

Robustness Across SNR and Model Architectures

Certified transcriptions exhibit consistently lower WER compared to unfiltered noisy outputs, with pronounced benefits as SNR decreases.

Figure 3: SNR versus WER showing superior robustness of certified transcriptions (solid lines) vs. uncertified noisy outputs (dashed lines).

Computational Efficiency

Martingale-based pipelines are more time-efficient for CTC ASR models and provide substantial compute savings for auto-regressive models via anytime-stopping rules, supporting practical deployment in real-time (RTF) scenarios.

Figure 4: SNR versus RTF, establishing competitive scaling of tournament certification across ASR architectures.

Linguistic Fragility Analysis

Operational audits reveal content-heavy tokens (nouns, verbs, proper nouns) are significantly less robust to certification compared to functional words (conjunctions, pronouns, determiners). The tournament mechanism substantially improves certified accuracy of content words, reinforcing the value of linguistic-aware certification strategies for mission-critical ASR.

Theoretical and Practical Implications

This research advances sequence-to-sequence certified robustness by circumventing sequence alignment multiplicity limitations and introducing competitive martingale-based aggregation for non-categorical outputs. The methodology is extensible to other sequence domains (machine translation, code synthesis) where combinatorial output spaces pose challenges for certification. The practical deployment of certified radius trust metrics is validated for real-time safety-critical applications, even under adversarial conditions.

The limitations regarding strict global Family-Wise Error Rate control in the atomic gate are acknowledged, with the tournament acting as a robust secondary filter. The framework provides dynamic safety markers rather than vacuous worst-case guarantees, improving both practical transcribability and diagnostic interpretability.

Conclusion

The dual-gate certification mechanism employing E-value tournaments offers a rigorous, efficient, and scalable path to certified robustness in ASR. Demonstrably outperforming existing sequence certification approaches on multiple axes—accuracy, recall, computational cost, and actionable trust-scoring—the framework sets foundations for robust deployment in adversarial and noisy environments. The methodology is structurally and statistically extensible to broader sequence prediction domains within AI, informing future development of anytime-valid, linguistically-aware certification systems.