VoxWatermark: Audio Watermark Benchmark

• VoxWatermark is a large-scale benchmark evaluating audio watermark detection under diverse perturbations, emphasizing detector stability over varied embedding methods.
• It integrates 10 watermarking methods across multilingual and multi-source corpora and organizes tests into no-box, black-box, and white-box attack regimes.
• The baseline detector AudioWMD employs multi-query stability modeling, demonstrating improved AUROC while revealing vulnerabilities under out-of-domain conditions.

Searching arXiv for papers on VoxWatermark and closely related audio watermarking work. [arXiv search] Query: "VoxWatermark audio watermark detection perturbations 2026" VoxWatermark is a large-scale benchmark for audio watermark detection under perturbations, introduced to evaluate whether watermark detectors remain reliable when the watermarking method is unknown, the language or acoustic domain shifts, and the audio is recorded, compressed, transmitted, transformed, or adversarially manipulated (Sedaghati et al., 13 Jun 2026). Its emphasis is detector-oriented rather than embedding-oriented: the benchmark targets source attribution and copyright accountability for speech generation systems deployed in open environments. The name also has a separate use in "The Watermark Shortcut: How Provenance Marking Sabotages Audio Deepfake Detection," where “VoxWatermark” denotes a shortcut phenomenon in which a detector learns the rule “watermark $\Rightarrow$ fake”; that usage refers to a detector failure mode rather than to the benchmark itself (Müller et al., 22 Jun 2026).

1. Conceptual scope and benchmark rationale

VoxWatermark was created to fill a gap in prior audio watermark evaluation. Earlier benchmarks such as AudioMarkBench and RAW-Bench mainly assess watermark robustness and perceptual quality, whereas VoxWatermark is built specifically for watermark detection and detector benchmarking (Sedaghati et al., 13 Jun 2026). The benchmark therefore asks a different question from most audio watermarking evaluations: not merely whether a watermark survives perturbation, but whether a detector can identify watermarked content stably across heterogeneous watermarking methods and cross-distribution conditions.

The design differs from prior work along six axes stated explicitly in the paper. It is detection-oriented rather than embedding-only; it covers 10 watermarking methods rather than a narrow subset; it is multilingual and multi-source; it uses unified injection and annotation; it organizes perturbations by access regime into no-box, black-box, and white-box settings; and it includes a baseline detector, AudioWMD, for large-scale cross-method evaluation (Sedaghati et al., 13 Jun 2026). This combination makes the benchmark a controlled paired dataset rather than a loose aggregation of watermarking outputs.

A central premise of VoxWatermark is that detection stability changes substantially when embedding paradigms vary. Traditional spread-spectrum, echo-based, quantization-based, phase-based, and least-significant-bit schemes induce different statistical traces from neural watermarking systems such as AudioSeal, WavMark, Timbre, and Perth. The benchmark is therefore intended to expose failure modes that are obscured when detectors are tested only on a single watermark family or a single language domain.

2. Corpus construction and data composition

The benchmark is large-scale, comprising 91,090K samples and approximately 126,513.89 hours of audio, including clean, watermarked, and perturbed samples (Sedaghati et al., 13 Jun 2026). All source audio is standardized to 5-second clips at 16 kHz mono, which gives a uniform substrate for paired clean-versus-watermarked construction and for perturbation analysis.

The unwatermarked pool is built from four speech corpora:

Corpus	Samples	Languages
LibriSpeech	20,000	English
Common Voice	20,000	25 languages
VCTK	10,000	English
AISHELL-1	10,000	Mandarin

These four corpora total 60,000 clips and 83.4 hours in the standardized format (Sedaghati et al., 13 Jun 2026). The selection is explicitly motivated by diversity in languages, speaker attributes such as gender, age, and accent, and acoustic conditions. That heterogeneity is central to the benchmark’s cross-domain objective: VoxWatermark is meant to test detectors under multilingual and multi-source variation rather than under a single homogeneous speech distribution.

The paper’s experimental protocol further partitions this diversity into in-domain and out-of-domain regimes. The training pool is formed from 34k clean clips drawn from LibriSpeech, Common Voice (English/Chinese), and AISHELL-1; the final training split contains 61,200 samples, evenly divided into 30,600 clean and 30,600 watermarked examples (Sedaghati et al., 13 Jun 2026). Two explicit OOD settings are then defined: a cross-lingual test set from Common Voice excluding English and Chinese, and a cross-accent test set from VCTK. This makes language shift and accent shift first-class evaluation variables rather than incidental dataset properties.

3. Watermarking methods, payload conventions, and perturbation regimes

VoxWatermark includes 10 watermarking methods, divided into 6 traditional and 4 learning-based / neural methods (Sedaghati et al., 13 Jun 2026).

Category	Methods
Traditional	LSB, QIM, Patchwork, Echo Hiding, Phase Coding, DSSS
Learning-based / neural	AudioSeal, WavMark, Timbre, Perth

The benchmark fixes payload conventions to improve comparability. 16-bit random payloads are used for all methods except Timbre, which uses a 10-bit payload, and Perth, which uses an implicit signature rather than a standard explicit payload (Sedaghati et al., 13 Jun 2026). The same preprocessing, segmentation, payload conventions, and labeling protocol are then applied across methods, so detector comparisons are not confounded by heterogeneous construction pipelines.

The perturbation model is organized by attacker access. In the benchmark’s formalism, watermark removal for a watermarked sample $x_w$ is written as

$$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$

while watermark forgery for an unwatermarked sample $x_u$ is

$$\eta_{\text{forg}} = \arg\max_{\eta} \mathcal{D}(x_u + \eta), \quad \text{s.t. } \mathcal{Q}(x_u + \eta) \approx \mathcal{Q}(x_u).$$

Here $\mathcal{D}(\cdot)$ is the detector output and $\mathcal{Q}(\cdot)$ is a perceptual quality metric such as ViSQOL; detector output $0$ means “unwatermarked” and $1$ means “watermarked” (Sedaghati et al., 13 Jun 2026).

The no-box regime contains 17 perturbations intended to simulate recording, storage, and transmission distortions. These include time stretching with rate range $[0.7,1.5]$, Gaussian noise with SNR $x_w$0 dB, background noise with SNR $x_w$1 dB, echo with delay $x_w$2 s, spectral filtering with ratio $x_w$3, quantization with bit depth $x_w$4, EnCodec/Opus with bitrate $x_w$5 kbps, dynamic range compression/expansion with ratio $x_w$6, and phase/jitter with parameter range $x_w$7 (Sedaghati et al., 13 Jun 2026). The background noise pool includes industrial, military, environmental, and communication-channel noises such as Factory 1/2, Leopard, M109, F-16, Buccaneer, Babble, Volvo, and HF Channel.

The black-box regime assumes query access to the detector but no access to internals. VoxWatermark uses HopSkipJumpAttack (HSJA) as a decision-based attack on raw waveforms and time-frequency representations, and Square Attack as a score-based random-search attack restricted to the spectrogram domain (Sedaghati et al., 13 Jun 2026). Removal attacks in this setting are applied to 200 watermarked samples per watermarking method.

The white-box regime assumes access to the decoder $x_w$8 and the target bitstream $x_w$9. For neural methods such as Timbre and WavMark, the benchmark minimizes binary cross-entropy,

$$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$0

with $$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$1 (Sedaghati et al., 13 Jun 2026). For traditional methods, differentiable approximations such as differentiable QIM and differentiable Echo Hiding are used. Both forgery and removal modes are evaluated, optimized with Adam, on 200 unwatermarked samples for forgery and 200 watermarked samples for removal per method.

4. AudioWMD: the benchmark’s baseline detector

VoxWatermark introduces AudioWMD, a two-stage detector intended for large-scale, multi-method, cross-distribution settings (Sedaghati et al., 13 Jun 2026). Its core premise is that watermark detection can be cast as a stability-analysis problem: a robust watermark should induce detector responses that remain locally stable under mild stochastic transformations, whereas fragile or perturbed inputs should show higher variance or decision instability.

In Stage I, a base detector $$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$2 is trained on 16 kHz log-mel spectrograms using a BCE-with-logits objective,

$$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$3

The sigmoid score $$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$4 is interpreted as watermark confidence (Sedaghati et al., 13 Jun 2026).

In Stage II, AudioWMD makes $$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$5 queries per clip: one original sample and seven transformed variants drawn from a stochastic transform set $$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$6 that includes temporal shifts, pitch shifts, and low-level noise. From the eight scores $$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$7, it constructs a 5-dimensional meta-feature vector

$$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$8

where

$$\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),$$9

$x_u$0

$x_u$1

$x_u$2

and

$x_u$3

A logistic regression meta-classifier $x_u$4 then produces the final decision (Sedaghati et al., 13 Jun 2026).

The benchmark compares AudioWMD against WMD, a single-query baseline that uses a ConvNeXt-V2 backbone, converts audio to spectrograms, produces a watermark confidence score, and is trained with an asymmetric loss (Sedaghati et al., 13 Jun 2026). The comparison functions as the paper’s main architectural ablation: WMD represents a conventional single-shot acoustic classifier, whereas AudioWMD adds multi-query stability modeling.

5. Empirical results and detector behavior under distribution shift

The principal reported result is that detector performance deteriorates sharply under OOD conditions, even when validation performance is relatively strong (Sedaghati et al., 13 Jun 2026).

Model	Validation AUROC / Accuracy	Test Set 1 AUROC / Accuracy	Test Set 2 AUROC / Accuracy
AudioWMD	88.3 / 84.0	63.8 / 53.0	63.2 / 58.0
WMD	72.0 / 67.0	57.1 / 55.0	57.9 / 56.0

These OOD results are obtained under cross-lingual and cross-accent evaluation with unseen watermarking methods—Patchwork, Echo, and WavMark—outside the training set (Sedaghati et al., 13 Jun 2026). The paper’s interpretation is explicit: both methods degrade sharply on OOD data, but AudioWMD is consistently better in AUROC than WMD, which supports the claim that query-time stability modeling improves ranking robustness under linguistic and algorithmic shift.

The perturbation analysis shows that robustness is highly regime-dependent. Under no-box perturbations, both detectors degrade significantly and performance can approach chance; AudioWMD yields only modest average gains over WMD (Sedaghati et al., 13 Jun 2026). Under white-box perturbations, AudioWMD is substantially stronger: for T1, WMD records 48.63 / 45.0 in AUROC/F1 versus 77.15 / 53.0 for AudioWMD, and for T2, WMD records 41.18 / 36.0 versus 70.02 / 57.0 for AudioWMD (Sedaghati et al., 13 Jun 2026).

The black-box results are more heterogeneous. For T1, overall TPR is 73.96 for WMD and 50.26 for AudioWMD; under Square attack, WMD records 40.62 and AudioWMD 77.34; under HSJA on spectrogram, WMD records 96.09 and AudioWMD 3.91 (Sedaghati et al., 13 Jun 2026). For T2, overall TPR is 93.42 for WMD and 51.44 for AudioWMD; under Square attack, WMD records 84.15 and AudioWMD 73.17; under HSJA on spectrogram, WMD records 100.00 and AudioWMD 2.50 (Sedaghati et al., 13 Jun 2026). The benchmark therefore does not present a single monotone robustness ranking; instead, it demonstrates that attack space, query structure, and feature domain interact strongly.

From these results, the benchmark draws four broad conclusions. First, method diversity exposes weaknesses: detectors trained on a subset of watermark schemes do not generalize uniformly to unseen schemes. Second, distribution shift hurts stability: language and accent shifts reduce confidence and separability. Third, stability modeling helps but is not universal: AudioWMD improves performance when perturbations induce local score instability that can be captured by query statistics, but it can fail when adversarial manipulations exploit a representation space misaligned with the query transforms. Fourth, attack space matters: waveform-level queries and spectrogram-level attacks do not stress detectors in the same way (Sedaghati et al., 13 Jun 2026).

6. Position in the literature, adjacent systems, and unresolved issues

VoxWatermark sits within a broader literature on proactive watermarking for speech and voice generation. One of its included methods, WavMark, is a deep-learning audio watermarking framework for generated speech that encodes up to 32 bits per 1-second segment and reports an average BER of 0.48\% across ten attacks in an utterance-level setting using 10–20 second hosts (Chen et al., 2023). Adjacent work not explicitly defined as VoxWatermark but operating in the same problem space includes True, a temporal-aware robust watermarking method for speech and singing voice that reports an average PESQ of 4.63 and robust performance up to 500 bps (Li et al., 21 Apr 2025), and VoiceMark, a zero-shot voice cloning-resistant watermarking method that achieves 0.964 ACC / 0.112 FAR on CosyVoice, 0.979 / 0.070 on F5-TTS, and 0.957 / 0.141 on MaskGCT after zero-shot VC synthesis (Li et al., 27 May 2025).

A distinct but closely related line of work concerns detector interaction rather than watermark embedding. "The Watermark Shortcut: How Provenance Marking Sabotages Audio Deepfake Detection" shows that when synthetic speech is watermarked and bona-fide speech is not, an audio deepfake detector can learn a spurious shortcut, effectively “watermark $x_u$5 fake,” producing coupled failures in generalization, strip-to-evade, and mark-to-frame behavior (Müller et al., 22 Jun 2026). In a controlled white-box experiment, watermarking bona-fide speech raises Equal Error Rate from 16% to 75%, and the paper proposes watermark augmentation on both classes during training as a remedy. A plausible implication is that detector benchmarks and provenance benchmarks should keep watermark detection, deepfake detection, and class-label correlations analytically distinct, because a detector may otherwise exploit watermark presence as a label proxy rather than learn the intended task.

The benchmark itself has explicit and implicit limitations. Only a subset of watermarking methods is used during training, with others reserved for OOD evaluation; AudioWMD uses a relatively simple logistic-regression meta-head; the scope is confined to speech/audio rather than multimodal provenance; black-box robustness is uneven; and white-box attacks on traditional schemes depend on differentiable surrogates such as differentiable QIM and differentiable Echo Hiding (Sedaghati et al., 13 Jun 2026). The paper also notes, at least implicitly, that real deployments may involve more compositionally layered distortions than any fixed perturbation suite can exhaustively model.

Within those limits, VoxWatermark establishes a standardized detector-centric evaluation substrate for multilingual, multi-method audio watermark detection under realistic perturbations. Its main contribution is not a new watermark embedding algorithm, but a benchmarked claim: watermark detection is fragile under distribution shift, and stability-aware detection offers a partial but non-universal path toward robustness at scale (Sedaghati et al., 13 Jun 2026).