---
title: 'VoxWatermark: Audio Watermark Benchmark'
url: https://www.emergentmind.com/topics/voxwatermark
type: topic
---

# VoxWatermark: Audio Watermark Benchmark

Searching arXiv for recent papers on VoxWatermark and closely related audio watermarking work.
[arXiv search] Query: "VoxWatermark audio watermark detection perturbations 2026"
VoxWatermark is a large-scale benchmark for audio watermark detection under perturbations, introduced to evaluate whether watermark detectors remain reliable when the watermarking method is unknown, the language or acoustic domain shifts, and the audio is recorded, compressed, transmitted, transformed, or adversarially manipulated [2606.15187]. Its emphasis is detector-oriented rather than embedding-oriented: the benchmark targets source attribution and copyright accountability for speech generation systems deployed in open environments. The name also has a separate use in "The Watermark Shortcut: How Provenance Marking Sabotages Audio Deepfake Detection," where “VoxWatermark” denotes a shortcut phenomenon in which a detector learns the rule “watermark $\Rightarrow$ fake”; that usage refers to a detector failure mode rather than to the benchmark itself [2606.23335].

## 1. Conceptual scope and benchmark rationale

VoxWatermark was created to fill a gap in prior audio watermark evaluation. Earlier benchmarks such as AudioMarkBench and RAW-Bench mainly assess watermark robustness and perceptual quality, whereas VoxWatermark is built specifically for watermark detection and detector benchmarking [2606.15187]. The benchmark therefore asks a different question from most audio watermarking evaluations: not merely whether a watermark survives perturbation, but whether a detector can identify watermarked content stably across heterogeneous watermarking methods and cross-distribution conditions.

The design differs from prior work along six axes stated explicitly in the paper. It is detection-oriented rather than embedding-only; it covers 10 watermarking methods rather than a narrow subset; it is multilingual and multi-source; it uses unified injection and annotation; it organizes perturbations by access regime into no-box, black-box, and white-box settings; and it includes a baseline detector, AudioWMD, for large-scale cross-method evaluation [2606.15187]. This combination makes the benchmark a controlled paired dataset rather than a loose aggregation of watermarking outputs.

A central premise of VoxWatermark is that detection stability changes substantially when embedding paradigms vary. Traditional spread-spectrum, echo-based, quantization-based, phase-based, and least-significant-bit schemes induce different statistical traces from neural watermarking systems such as AudioSeal, WavMark, Timbre, and Perth. The benchmark is therefore intended to expose failure modes that are obscured when detectors are tested only on a single watermark family or a single language domain.

## 2. Corpus construction and data composition

The benchmark is large-scale, comprising **91,090K samples** and approximately **126,513.89 hours** of audio, including clean, watermarked, and perturbed samples [2606.15187]. All source audio is standardized to **5-second clips** at **16 kHz mono**, which gives a uniform substrate for paired clean-versus-watermarked construction and for perturbation analysis.

The unwatermarked pool is built from four speech corpora:

| Corpus | Samples | Languages |
|---|---:|---|
| LibriSpeech | 20,000 | English |
| Common Voice | 20,000 | 25 languages |
| VCTK | 10,000 | English |
| AISHELL-1 | 10,000 | Mandarin |

These four corpora total **60,000** clips and **83.4** hours in the standardized format [2606.15187]. The selection is explicitly motivated by diversity in languages, speaker attributes such as gender, age, and accent, and acoustic conditions. That heterogeneity is central to the benchmark’s cross-domain objective: VoxWatermark is meant to test detectors under multilingual and multi-source variation rather than under a single homogeneous speech distribution.

The paper’s experimental protocol further partitions this diversity into in-domain and out-of-domain regimes. The training pool is formed from **34k clean clips** drawn from LibriSpeech, Common Voice (English/Chinese), and AISHELL-1; the final training split contains **61,200 samples**, evenly divided into **30,600 clean** and **30,600 watermarked** examples [2606.15187]. Two explicit OOD settings are then defined: a cross-lingual test set from Common Voice excluding English and Chinese, and a cross-accent test set from VCTK. This makes language shift and accent shift first-class evaluation variables rather than incidental dataset properties.

## 3. Watermarking methods, payload conventions, and perturbation regimes

VoxWatermark includes **10 watermarking methods**, divided into **6 traditional** and **4 learning-based / neural** methods [2606.15187].

| Category | Methods |
|---|---|
| Traditional | LSB, QIM, Patchwork, Echo Hiding, Phase Coding, DSSS |
| Learning-based / neural | AudioSeal, WavMark, Timbre, Perth |

The benchmark fixes payload conventions to improve comparability. **16-bit random payloads** are used for all methods except **Timbre**, which uses a **10-bit payload**, and **Perth**, which uses an **implicit signature** rather than a standard explicit payload [2606.15187]. The same preprocessing, segmentation, payload conventions, and labeling protocol are then applied across methods, so detector comparisons are not confounded by heterogeneous construction pipelines.

The perturbation model is organized by attacker access. In the benchmark’s formalism, watermark removal for a watermarked sample $x_w$ is written as
$$
\eta_{\text{rem}} = \arg\min_{\eta} \mathcal{D}(x_w + \eta), \quad \text{s.t. } \mathcal{Q}(x_w + \eta) \approx \mathcal{Q}(x_w),
$$
while watermark forgery for an unwatermarked sample $x_u$ is
$$
\eta_{\text{forg}} = \arg\max_{\eta} \mathcal{D}(x_u + \eta), \quad \text{s.t. } \mathcal{Q}(x_u + \eta) \approx \mathcal{Q}(x_u).
$$
Here $\mathcal{D}(\cdot)$ is the detector output and $\mathcal{Q}(\cdot)$ is a perceptual quality metric such as ViSQOL; detector output $0$ means “unwatermarked” and $1$ means “watermarked” [2606.15187].

The **no-box** regime contains **17 perturbations** intended to simulate recording, storage, and transmission distortions. These include time stretching with rate range $[0.7,1.5]$, Gaussian noise with SNR $[5,40]$ dB, background noise with SNR $[5,35]$ dB, echo with delay $[0.1,0.9]$ s, spectral filtering with ratio $[0.1,0.6]$, quantization with bit depth $[4,64]$, EnCodec/Opus with bitrate $[1.5,256]$ kbps, dynamic range compression/expansion with ratio $[2.0,8.0]$, and phase/jitter with parameter range $[0.01,1000]$ [2606.15187]. The background noise pool includes industrial, military, environmental, and communication-channel noises such as Factory 1/2, Leopard, M109, F-16, Buccaneer, Babble, Volvo, and HF Channel.

The **black-box** regime assumes query access to the detector but no access to internals. VoxWatermark uses **HopSkipJumpAttack (HSJA)** as a decision-based attack on raw waveforms and time-frequency representations, and **Square Attack** as a score-based random-search attack restricted to the spectrogram domain [2606.15187]. Removal attacks in this setting are applied to **200 watermarked samples per watermarking method**.

The **white-box** regime assumes access to the decoder $\mathrm{Dec}$ and the target bitstream $\mathbf{w}$. For neural methods such as Timbre and WavMark, the benchmark minimizes binary cross-entropy,
$$
\mathcal{L}_{\text{BCE}} = -\sum_{i=1}^{n} \left[ w_i \log(\hat{w}_i) + (1-w_i)\log(1-\hat{w}_i) \right],
$$
with $\hat{w}_i = \mathrm{Dec}(x+\eta)$ [2606.15187]. For traditional methods, differentiable approximations such as differentiable QIM and differentiable Echo Hiding are used. Both **forgery** and **removal** modes are evaluated, optimized with **Adam**, on **200 unwatermarked samples** for forgery and **200 watermarked samples** for removal per method.

## 4. AudioWMD: the benchmark’s baseline detector

VoxWatermark introduces **AudioWMD**, a two-stage detector intended for large-scale, multi-method, cross-distribution settings [2606.15187]. Its core premise is that watermark detection can be cast as a stability-analysis problem: a robust watermark should induce detector responses that remain locally stable under mild stochastic transformations, whereas fragile or perturbed inputs should show higher variance or decision instability.

In **Stage I**, a base detector $f_\theta$ is trained on **16 kHz log-mel spectrograms** using a BCE-with-logits objective,
$$
\mathcal{L}_{\text{base}}(\theta) = -\frac{1}{N}\sum_{i=1}^{N} \Big[ y_i \log \sigma(f_\theta(x_i)) + (1-y_i)\log(1-\sigma(f_\theta(x_i))) \Big].
$$
The sigmoid score $s(x)=\sigma(f_\theta(x))\in[0,1]$ is interpreted as watermark confidence [2606.15187].

In **Stage II**, AudioWMD makes **$K=8$ queries** per clip: one original sample and seven transformed variants drawn from a stochastic transform set $\mathcal{T}$ that includes temporal shifts, pitch shifts, and low-level noise. From the eight scores $s^{(k)}$, it constructs a **5-dimensional meta-feature vector**
$$
\phi(x) = [\mu, \sigma_s, \Delta, \rho, r_{\text{flip}}]^\top,
$$
where
$$
\mu = \frac{1}{K}\sum_{k=1}^{K} s^{(k)},
$$
$$
\sigma_s = \sqrt{\frac{1}{K}\sum_{k=1}^{K}(s^{(k)}-\mu)^2},
$$
$$
\Delta = \max_k s^{(k)} - \min_k s^{(k)},
$$
$$
\rho = \frac{1}{K}\sum_{k=1}^{K}\mathbb{I}[s^{(k)} > 0.5],
$$
and
$$
r_{\text{flip}} = \frac{1}{K}\sum_{k=1}^{K} \mathbb{I}\Big[\operatorname{sign}(s^{(k)}-0.5)\neq \operatorname{sign}(s^{(1)}-0.5)\Big].
$$
A **logistic regression meta-classifier** $g_\psi$ then produces the final decision [2606.15187].

The benchmark compares AudioWMD against **WMD**, a single-query baseline that uses a **ConvNeXt-V2 backbone**, converts audio to spectrograms, produces a watermark confidence score, and is trained with an asymmetric loss [2606.15187]. The comparison functions as the paper’s main architectural ablation: WMD represents a conventional single-shot acoustic classifier, whereas AudioWMD adds multi-query stability modeling.

## 5. Empirical results and detector behavior under distribution shift

The principal reported result is that detector performance deteriorates sharply under OOD conditions, even when validation performance is relatively strong [2606.15187].

| Model | Validation AUROC / Accuracy | Test Set 1 AUROC / Accuracy | Test Set 2 AUROC / Accuracy |
|---|---|---|---|
| AudioWMD | 88.3 / 84.0 | 63.8 / 53.0 | 63.2 / 58.0 |
| WMD | 72.0 / 67.0 | 57.1 / 55.0 | 57.9 / 56.0 |

These OOD results are obtained under cross-lingual and cross-accent evaluation with **unseen** watermarking methods—**Patchwork, Echo, and WavMark**—outside the training set [2606.15187]. The paper’s interpretation is explicit: both methods degrade sharply on OOD data, but AudioWMD is consistently better in AUROC than WMD, which supports the claim that query-time stability modeling improves ranking robustness under linguistic and algorithmic shift.

The perturbation analysis shows that robustness is highly regime-dependent. Under **no-box** perturbations, both detectors degrade significantly and performance can approach chance; AudioWMD yields only modest average gains over WMD [2606.15187]. Under **white-box** perturbations, AudioWMD is substantially stronger: for **T1**, WMD records **48.63 / 45.0** in AUROC/F1 versus **77.15 / 53.0** for AudioWMD, and for **T2**, WMD records **41.18 / 36.0** versus **70.02 / 57.0** for AudioWMD [2606.15187].

The **black-box** results are more heterogeneous. For **T1**, overall TPR is **73.96** for WMD and **50.26** for AudioWMD; under **Square attack**, WMD records **40.62** and AudioWMD **77.34**; under **HSJA on spectrogram**, WMD records **96.09** and AudioWMD **3.91** [2606.15187]. For **T2**, overall TPR is **93.42** for WMD and **51.44** for AudioWMD; under **Square attack**, WMD records **84.15** and AudioWMD **73.17**; under **HSJA on spectrogram**, WMD records **100.00** and AudioWMD **2.50** [2606.15187]. The benchmark therefore does not present a single monotone robustness ranking; instead, it demonstrates that attack space, query structure, and feature domain interact strongly.

From these results, the benchmark draws four broad conclusions. First, **method diversity exposes weaknesses**: detectors trained on a subset of watermark schemes do not generalize uniformly to unseen schemes. Second, **distribution shift hurts stability**: language and accent shifts reduce confidence and separability. Third, **stability modeling helps but is not universal**: AudioWMD improves performance when perturbations induce local score instability that can be captured by query statistics, but it can fail when adversarial manipulations exploit a representation space misaligned with the query transforms. Fourth, **attack space matters**: waveform-level queries and spectrogram-level attacks do not stress detectors in the same way [2606.15187].

## 6. Position in the literature, adjacent systems, and unresolved issues

VoxWatermark sits within a broader literature on proactive watermarking for speech and voice generation. One of its included methods, **WavMark**, is a deep-learning audio watermarking framework for generated speech that encodes up to **32 bits per 1-second segment** and reports an **average BER of 0.48\%** across ten attacks in an utterance-level setting using 10–20 second hosts [2308.12770]. Adjacent work not explicitly defined as VoxWatermark but operating in the same problem space includes **True**, a temporal-aware robust watermarking method for speech and singing voice that reports an **average PESQ of 4.63** and robust performance up to **500 bps** [2504.14832], and **VoiceMark**, a zero-shot voice cloning-resistant watermarking method that achieves **0.964 ACC / 0.112 FAR** on CosyVoice, **0.979 / 0.070** on F5-TTS, and **0.957 / 0.141** on MaskGCT after zero-shot VC synthesis [2505.21568].

A distinct but closely related line of work concerns detector interaction rather than watermark embedding. "The Watermark Shortcut: How Provenance Marking Sabotages Audio Deepfake Detection" shows that when synthetic speech is watermarked and bona-fide speech is not, an audio deepfake detector can learn a spurious shortcut, effectively “watermark $\Rightarrow$ fake,” producing coupled failures in generalization, strip-to-evade, and mark-to-frame behavior [2606.23335]. In a controlled white-box experiment, watermarking bona-fide speech raises Equal Error Rate from **16%** to **75%**, and the paper proposes watermark augmentation on both classes during training as a remedy. A plausible implication is that detector benchmarks and provenance benchmarks should keep watermark detection, deepfake detection, and class-label correlations analytically distinct, because a detector may otherwise exploit watermark presence as a label proxy rather than learn the intended task.

The benchmark itself has explicit and implicit limitations. Only a subset of watermarking methods is used during training, with others reserved for OOD evaluation; AudioWMD uses a relatively simple **logistic-regression meta-head**; the scope is confined to speech/audio rather than multimodal provenance; black-box robustness is uneven; and white-box attacks on traditional schemes depend on differentiable surrogates such as differentiable QIM and differentiable Echo Hiding [2606.15187]. The paper also notes, at least implicitly, that real deployments may involve more compositionally layered distortions than any fixed perturbation suite can exhaustively model.

Within those limits, VoxWatermark establishes a standardized detector-centric evaluation substrate for multilingual, multi-method audio watermark detection under realistic perturbations. Its main contribution is not a new watermark embedding algorithm, but a benchmarked claim: watermark detection is fragile under distribution shift, and stability-aware detection offers a partial but non-universal path toward robustness at scale [2606.15187].

Source: https://www.emergentmind.com/topics/voxwatermark