Order-Agnostic Watermarking

• Order-agnostic watermarking is an information hiding technique that embeds detectable patterns regardless of token order in non-sequential, parallel, or permuted environments.
• It adapts classical strategies with methods like Pattern-mark, DMark, TokenMark, and AWARE to overcome challenges such as lack of stable prefixes and adversarial editing.
• Empirical evaluations show high true positive rates and minimal quality loss, ensuring practical ownership verification and traceability across diverse modalities.

Order-agnostic watermarking refers to information hiding techniques designed for environments in which the order of symbols (tokens, frames, blocks, etc.) is either not consistent, unknown, or actively modifiable—such as in parallel or diffusion-based generative models, or in signals subject to permutation or desynchronization attacks. Unlike classical watermarking, which typically assumes a fixed sequential decoding order, order-agnostic watermarking achieves robust detection and embedding regardless of token or segment order during generation or adversarial editing, enabling practical ownership verification and traceability in modern model architectures and media types.

1. Motivations and Core Challenges

Traditional watermarking schemes assume a left-to-right, autoregressive generation paradigm. Classical red/green-list methods, for example, seed a deterministic partition of the output vocabulary at each generation step based on the prefix, embedding a bias detectable by statistical tests on ordered output (Wu et al., 3 Oct 2025, Chen et al., 2024). In models or contexts where order is not fixed—diffusion LLMs, mask-based sequence infilling, permutation-invariant transformers, or audio/video subject to desynchronization—these approaches fail due to the absence of a stable context or unknown generation history at detection time.

Key structural challenges are:

• No stable prefix: Sequential dependence on prior outputs is invalid; arbitrary subsets of tokens may be present at generation.
• Unknown order at verification: The detection algorithm often observes only the unordered set of outputs; the original generation or embedding order is unrecoverable.
• Potential for overwriting or reordering: In models with parallel sampling or adversarially edited signals, token or segment order may be permuted, deleted, or truncated, with substantial risk of desynchronization.

2. Architectural Principles and Embedding Mechanisms

2.1. Pattern-Mark Framework (Markov Pattern Watermarking)

Pattern-mark (Chen et al., 2024) addresses order-agnostic LLMs (OALMs) such as non-autoregressive transformers, where sequential partitions are invalid. The key innovation is to pre-sample a key sequence $\mathbf{k} = (k_1, \dots, k_n)$ from a Markov chain with designed transition dynamics, ensuring that specific high-frequency patterns (e.g., alternating substrings) occur with atypically high probability.

• Embedding: For each position $i$ (independent of any unfinalized context), use $k_i$ to select a vocabulary partition $V_{k_i}$. At generation, tokens in $V_{k_i}$ are boosted, biasing the outputs towards detectable statistical patterns.
• Order-independence: The process never conditions on previous tokens; each position's embedding is independent of the output order or available context.

2.2. DMark and Order-Agnostic Watermarking in Diffusion LLMs

Diffusion-based LLMs (dLLMs) generate tokens in any order, sometimes overwriting previous guesses, and lack a sequential, causal context. DMark (Wu et al., 3 Oct 2025) introduces three embedding strategies:

• Predictive Watermarking: When context tokens are unavailable during generation, predict the most likely completion (via $\operatorname{argmax}$ over logits) and use this as proxy context in the partition hash.
• Bidirectional Watermarking: Construct separate green lists using available left or right neighbors (forward or backward context), since diffusion decoding can condition on both.
• Predictive-Bidirectional Watermarking: Combine both strategies, predicting missing left/right context, building both partitions using available or predicted tokens, and applying a double bias to tokens favored by either partition. Algorithm 1 in (Wu et al., 3 Oct 2025) formally details this embedding scheme.

2.3. Permutation-Equivariant Schemes in Transformers (TokenMark)

TokenMark (Xu et al., 2024) exploits the inherent permutation equivariance of standard transformer backbones. The watermark is triggered not by input order, but by permuting the hidden dimension with a secret permutation $P$, and decoding via a learned signature. As the core architecture is agnostic to token order, this watermark is robust to any permutation in hidden space.

• Embedding: Fine-tune model weights with both original and $P$-permuted inputs, simultaneously carving out watermark and normal functional modes. The trigger is hidden-dimension permutation rather than sequence order.

2.4. Order-Agnostic Detection in Adversarial Audio Watermarking (AWARE)

The AWARE framework (Pavlović et al., 20 Oct 2025) achieves order-agnostic watermarking in the audio domain through two architectural choices:

• Conv-1 (frame-wise) feature extraction: Each time segment is processed independently, preventing positional entanglement.
• Bitwise Readout Head (BRH): Global pooling over time-steps aggregates evidence for each watermark bit, rendering the detector invariant to time-segment order, cuts, or reordering.

No embedding neural network is required; watermarking is accomplished via adversarial optimization of a constrained STFT magnitude perturbation, guided by a frozen detector.

3. Statistical Detection and Provable Guarantees

Order-agnostic watermarking replaces explicit positional or sequential dependence in the detection algorithm with statistical hypothesis testing based on the unordered set of outputs:

• Pattern-based detection (Pattern-mark): Given the output, reconstruct the key sequence $k_i$ for each position by vocabulary lookup, regardless of token order. The frequency of target patterns (e.g., specific substrings) is compared to a null distribution of uniformly random keys, with $p$-value thresholds controlling false positive rates (Chen et al., 2024).
• Z-score testing (DMark): For watermark signals based on partition over-representation, a $z$-score calculated from token counts in bias partitions serves as the detection statistic, with null thresholds set to limit FPR (Wu et al., 3 Oct 2025).
• Watermark signature similarity (TokenMark): Watermark extraction checks the cosine similarity between the decoded signature and the secret, with thresholding for verification (Xu et al., 2024).
• Global pooling detection (AWARE): Frame or segment-level evidence is globally aggregated, so detection is robust to any reordering, deletion, or cropping of input segments (Pavlović et al., 20 Oct 2025).

Provable guarantees arise from the statistical independence and null behavior of the test statistics. For Pattern-mark, the false positive rate is analytically computable from the known distribution of pattern counts under the non-watermarked (null) hypothesis.

4. Empirical Performance and Quality–Detectability Trade-off

Evaluation across applications demonstrates that order-agnostic watermarking can achieve high true positive rates (TPR) while maintaining generation quality:

Framework	TPR@1% FPR (range)	Quality Impact	Robustness
DMark (dLLMs)	92–99.5%	PPL < 7 (for δ=2.0)	TPR > 83%@20% text edits
Pattern-mark (OALMs)	>95%@0.1% FPR	pLDDT drop < 0.5%	TPR >80%@20% token/phrase edits
TokenMark (BERT/GPT)	1.00 (100%)	Δaccuracy < 0.1%	100% WR after attacks
AWARE (audio)	BER ≤ 2%	PESQ/STOI unchanged	Robust to desync, cuts

For DMark in particular, increasing watermark strength (bias δ) increases detection rate but also perplexity: $\delta=0$ yields PPL $\approx$ 2.7, $\delta=2.0$ yields PPL $\approx$ 6.3; semantic coherence is preserved for $\delta \leq 2.0$ (Wu et al., 3 Oct 2025). Pattern-mark achieves a strong quality–detectability trade-off: at bias $\delta=1.25$, the protein design model's structure quality (pLDDT) drops by less than 0.5%, while TPR exceeds 95% (Chen et al., 2024). AWARE maintains high speech intelligibility and audio quality, BER$\,\lesssim\,$2% across most edits (Pavlović et al., 20 Oct 2025).

5. Robustness to Editing and Desynchronization Attacks

Order-agnostic watermarks are inherently more resilient to attacks that would desynchronize or destroy traditional, order-sensitive watermarks. Specific findings include:

• Textual manipulations: Predictive-bidirectional DMark retains $>$83% TPR under 20% token insertions/deletions/swaps, and over 51% TPR under GPT-based paraphrase attacks—surpassing classical methods by a wide margin (Wu et al., 3 Oct 2025).
• Token and paraphrase edits (Pattern-mark): TPR stays above 80% at 0.1% FPR under heavy token-flip and paraphrase attacks, where baseline methods degrade to random guessing (Chen et al., 2024).
• Model-level attacks (TokenMark): Robustness is maintained under fine-tuning, pruning, quantization, model extraction (EaaS), and known “overwriting” attacks; watermark retrieval remains at 100% extraction rate (Xu et al., 2024).
• Temporal and spectral attacks (AWARE): Detection is correct after frame reordering, insertion/deletion, cropping, or heavy codec and filter attacks (e.g., MP3, low/high-pass, vocoder re-synthesis). BER remains $<$2% for most conditions, with only moderate increase (3–6%) for severe cuts (Pavlović et al., 20 Oct 2025).

6. Algorithmic Complexity and Limitations

• Computational complexity: DMark and Pattern-mark both incur $O(|\mathcal{V}|)$ per-token overhead for partition and bias computation, matching AR watermarking schemes (Wu et al., 3 Oct 2025, Chen et al., 2024). Pattern-mark's detection involves dynamic programming with $O(n^2 \ell^m)$ for pattern length $m$ and alphabet size $\ell$; this is practical for moderate $m$ and $\ell$.
• Tuning and scalability: Pattern-mark requires selection of the Markov chain, pattern set, and pattern window $m$, balancing detection power and output distortion. For large vocabularies or complex key alphabets, precomputing partition mappings and null distributions can be a bottleneck (Chen et al., 2024).
• Potential limitations: AWARE requires a clean mapping from time-frequency perturbations to robust global-permutation invariance; scaling to finer-grained payloads or ultra-short signals may suffer. Pattern-mark's optimality may depend on Markov chain selection and is tied to Markov pattern frequencies (Pavlović et al., 20 Oct 2025, Chen et al., 2024).

7. Broad Applicability and Future Directions

Order-agnostic watermarking has been demonstrated across diverse modalities and architectures:

• Diffusion LLMs: DMark establishes the first reliable watermarking protocol for dLLMs with robust text and paraphrase resilience (Wu et al., 3 Oct 2025).
• Protein and translation sequence generation: Pattern-mark is empirically validated for order-agnostic protein design (ProteinMPNN) and non-autoregressive translation (CMLM) (Chen et al., 2024).
• Generic transformers: TokenMark's permutation-equivalent embedding provides a modality-agnostic watermark for both vision transformers and NLP models (Xu et al., 2024).
• Audio watermarking: AWARE demonstrates practical resilience to desynchronization and edits in real-world audio pipelines (Pavlović et al., 20 Oct 2025).

Potential directions include extending pattern-based and order-agnostic frameworks to more complex Markov models, real-time streaming detection, hybrid schemes for hierarchical or mixed generative orders, and adversarial/semantic watermarking strategies robust to learned paraphrasing or attack-aware remapping. These developments position order-agnostic watermarking as a foundational technique for ownership verification and content provenance in the age of non-sequential, parallel, and edit-resilient generative modeling.