VoxWatermark: A Large-Scale Benchmark for Audio Watermark Detection under Perturbations

Abstract: With the rapid deployment of speech generation systems in open environments, providing verifiable source attribution and copyright accountability for audio content has become critical. A gap in current research is the lack of a unified benchmark that systematically compares different watermark injection methods under realistic distribution shifts. To address this, we build VoxWatermark by applying 10 watermarking methods (4 neural and 6 traditional) with unified injection and annotation on multilingual, multi-source corpora, and introducing no-box, black-box, and white-box perturbations to simulate real recording and transmission conditions. Based on this benchmark, we propose AudioWMD as a robust baseline detector for large-scale, multi-method, cross-distribution settings. Results show that injection-method diversity and distribution shifts affect detection stability, while validating the effectiveness and scalability of AudioWMD. Dataset and code are publicly available.

• The paper introduces VoxWatermark, a large-scale dataset with over 91 million samples and varied watermarking schemes under realistic perturbations.
• It proposes AudioWMD, a two-stage, stability-based detection method that enhances robustness against adversarial white-box and black-box attacks.
• Empirical findings demonstrate improved cross-domain generalization and attack-specific performance, highlighting the need for comprehensive evaluation in audio watermarking.

Comprehensive Benchmarking for Audio Watermark Detection: The VoxWatermark Dataset and AudioWMD

Introduction

The increasing prevalence of high-fidelity TTS systems in practical deployment has amplified the demand for reliable and scalable audio watermark detection solutions. However, existing evaluation protocols and datasets are limited, focusing primarily on watermark robustness and perceptual quality while neglecting systematic and detector-agnostic evaluation under realistic and adversarial perturbations. The paper "VoxWatermark: A Large-Scale Benchmark for Audio Watermark Detection under Perturbations" (2606.15187) directly addresses this gap by introducing VoxWatermark, a large-scale, open-source benchmark for evaluating watermark detection under diverse perturbation regimes, as well as proposing AudioWMD, a robust and scalable watermark detection framework suitable for cross-method and cross-distribution deployment.

Construction of the VoxWatermark Benchmark

The VoxWatermark dataset is constructed to provide both breadth and realism. It contains over 91 million audio samples encompassing more than 126,000 hours of diverse clean, watermarked, and perturbed audio sourced from multilingual and multi-accent corpora. Watermarks are injected using 10 distinct schemes, including both classical signal processing (e.g., LSB, QIM, DSSS, Patchwork, Phase Coding, Echo Hiding) and neural network-based approaches (e.g., AudioSeal, WavMark, Timbre, Perth).

This heterogeneity in both cover content and watermarking algorithms ensures that model evaluation is robust against simple overfitting to a particular watermark variant, language, or speaker identity. The dataset uniquely incorporates extensive perturbation families designed to simulate actual recording, broadcast, and adversarial conditions. Perturbations are hierarchically categorized by attacker knowledge:

• No-box: Standard DSP distortions such as codecs (Opus, EnCodec), environmental noise, dynamic range compression, and time-frequency modifications—requiring no system internals.
• Black-box: Adversarial perturbations that exploit only query access, including HopSkipJumpAttack (HSJA) and Square Attack.
• White-box: Gradient-based removal and forgery crafted with full knowledge of the watermark decoder and bitstream.

These systematic variations expose the limitations of detectors trained and evaluated under narrow, method-specific conditions.

AudioWMD: Stability-Aware Watermark Detection

To address the limitations of classical single-pass acoustic classifiers, the paper proposes AudioWMD, a two-stage detection pipeline based on stability analysis under stochastic transformations. The core intuition is that authentic watermarks induce stable predictive signals across small perturbations, whereas non-watermarked or adversarially manipulated audio exhibits greater prediction variance. The AudioWMD pipeline operates as follows:

1. Query Generation: Each input audio is used to generate $K$ variants via stochastic, minor, but plausible perturbations (e.g., pitch/time shift, noise addition).
2. Signal Extraction: Each variant passes through a CNN-based base classifier, yielding a distribution of $K$ sigmoid outputs.
3. Meta-Feature Aggregation: These $K$ scores are reduced to a 5-dimensional feature vector encoding mean, variance, range, positive occupancy, and decision-flip statistics.
4. Meta-Classification: A logistic regression meta-classifier consumes this feature vector for the final detection decision.

This architecture enables robust performance across unseen watermark injection methods and distribution shifts, even in conditions where attackers attempt to evade detection by exploiting fixed classifier weaknesses.

Figure 1: Architecture of the proposed AudioWMD framework, illustrating the stochastic query generation and meta-feature aggregation pipeline.

Figure 2: Overview of AudioWMD: stochastic transformation, base detection, and aggregation for robust watermark classification.

Experimental Protocol and Evaluation

VoxWatermark splits the data into in-domain (Clean English/Chinese) and OOD (25 languages, diverse accents) sets, also partitioned by seen/unseen watermarking methods not exposed during training. No-box perturbations for testing are selected to cover noise, temporal, and codec-related degradations, and a rigorous separation of training and evaluation ensures no contamination.

AudioWMD is compared against a strong baseline (WMD), which employs a ConvNeXt-V2 CNN evaluated in a single-pass regime. Evaluation metrics include AUROC, accuracy, precision, recall, and F1 for both "clean" and "watermarked" classes over all attack conditions.

Empirical Findings

Cross-Domain Generalization: AudioWMD achieves consistently higher OOD AUROC—Test 1: 0.638, Test 2: 0.632—compared to WMD (0.571, 0.579, respectively). This demonstrates improved cross-lingual and cross-accent robustness, especially under method and distribution shifts.

No-box Robustness: Both models degrade under generic perturbations to near-chance AUROCs, reflecting the intrinsic difficulty of reliable watermark extraction post-transformation. However, AudioWMD retains a modest robustness edge in AUROC under these conditions.

White-box Attack Resistance: AudioWMD exhibits substantially higher white-box attack resilience (AUROC/F1: up to 0.77/0.57 vs. WMD's 0.41/0.36), indicating that stability-based approaches can mitigate strong gradient-based adversarial removals and forgeries.

Black-box Attack Specificity: Performance is highly attack-dependent. AudioWMD is robust to some attacks (e.g., Square), but highly susceptible to others (e.g., HSJA in the spectrogram domain, where TPR drops below 0.04), indicating that attack diversity in benchmarking is essential to diagnosis.

Theoretical and Practical Implications

By formalizing the detection scenario via query-time statistical stability, this work advances the methodology for robust detection in open-world settings, where watermark details, corpus provenance, and perturbation history are all variable. The clear performance gap under adaptive attacks exposes both the need for architectures with multi-modal (time/frequency) invariance and the value of defensive ensembles that combine direct and stability-aware detection.

Practically, the release of VoxWatermark is expected to catalyze rapid progress towards standardized, reproducible evaluation and the development of attack-agnostic detectors suitable for deployment in high-stakes digital provenance pipelines. The dataset and codebase, made openly available, create a foundation for further work on adversarial training, self-supervised and unsupervised detection, and domain-adaptive defenses.

Future Directions

Future research should focus on:

• Expanding the diversity of perturbation operators, especially those that simultaneously affect perceptual quality and watermark recoverability.
• Developing and benchmarking models that integrate joint time-frequency analysis and ensemble decision-making tailored for adversarial robustness.
• Leveraging self-supervised representations to capture watermark signal without overfitting to fixed, known injection or decoding algorithms.
• Incorporating human perception models to align detector decision boundaries with perceptually significant watermark artifacts.

Conclusion

VoxWatermark establishes a new standard for large-scale, reproducible audio watermark detection research, uncovering limitations of existing baselines and providing evidence for the effectiveness of stability-aware meta-classification. The contradictory finding that robustness is highly attack-dependent and not uniformly improved by advanced modeling underscores the necessity of comprehensive, method-agnostic evaluation. As watermarking becomes integral to combating speech deepfakes and establishing media authenticity, benchmarks such as VoxWatermark will continue to play a central role in both theoretical research and practical deployment in AI-driven ecosystems.