MajorMark: Watermarking Method for LLMs

• MajorMark is a watermarking technique that encodes multi-bit binary messages into generated text using preferred token sets determined by the majority bit.
• It modifies token logits based on shard-wise occurrence counts to balance output quality and decoding accuracy, outperforming previous frequency-based methods.
• The MajorMark+ extension partitions the message into blocks, employing deterministic decoding to enhance text utility and reduce computational costs.

MajorMark is a multi-bit watermarking method for LLMs that embeds a binary message $m \in \{0,1\}^b$ into generated text by selecting preferred token sets according to the message’s majority bit, rather than by fixing a small preferred set and decoding through direct token-frequency counting. The method is designed to improve the standard trade-off between text quality and decoding accuracy in LLM watermarking. Its extension, MajorMark$^+$, partitions the message into multiple blocks and deterministically decodes each block, with the stated aim of further improving both watermarked-text quality and message recovery accuracy (Xu et al., 5 Aug 2025).

1. Problem setting and design objective

MajorMark is situated in the literature on multi-bit watermarking for LLMs, where the objective is to embed a recoverable binary message into generated text for origin verification and misuse tracing. In this setting, prior methods face a quality–decoding accuracy trade-off governed by the green list ratio $\gamma = |G|/|V|$, where $G$ is the preferred token set and $V$ is the vocabulary. When decoders rely on counting how many generated tokens fall into $G$, small $\gamma$ strengthens the decoding signal but distorts the output distribution, whereas large $\gamma$ preserves generation quality but weakens statistical recoverability. The paper identifies this dependence on token-frequency-based decoding as the core limitation of prior multi-bit schemes such as MPAC and RSBH, and notes that some methods, including CTWL, CycleShift, and DepthW, additionally require brute-force enumeration across $2^b$ messages as $b$ grows (Xu et al., 5 Aug 2025).

Within this formulation, MajorMark redefines the role of the green list. At the $^+$0-th generation step, the encoder boosts the logits of tokens in $^+$1 by a bias $^+$2, increasing their sampling probability after softmax. The method’s central claim is that reliable decoding need not depend on direct green-list frequency counting. Instead, it can depend on shard-wise occurrence structure induced by the message itself. This permits a large preferred-token set and, consequently, less distortion of the base model’s distribution.

2. Majority bit-aware encoding

The defining concept of MajorMark is the majority bit. For a binary message $^+$3, let $^+$4 and $^+$5 denote the number of occurrences of bit $^+$6 and bit $^+$7, respectively. The majority bit $^+$8 is defined as $^+$9 if $\gamma = |G|/|V|$0, and $\gamma = |G|/|V|$1 otherwise. In the tie case $\gamma = |G|/|V|$2, the method sets $\gamma = |G|/|V|$3 by default. Because the majority bit appears at least $\gamma = |G|/|V|$4 times, it provides a stable signal for preferred-token construction (Xu et al., 5 Aug 2025).

At each generation step $\gamma = |G|/|V|$5, MajorMark computes a pseudo-random seed

$\gamma = |G|/|V|$6

where $\gamma = |G|/|V|$7 is a secret key and $\gamma = |G|/|V|$8 are the last two generated tokens. The paper explicitly contrasts this with the common practice of using $\gamma = |G|/|V|$9, arguing that inclusion of $G$0 balances seed diversity and inclusion of $G$1 enables recovery of the correct majority-bit hypothesis during decoding.

Using $G$2, the encoder permutes the vocabulary $G$3 and evenly partitions $G$4 into $G$5 disjoint shards $G$6. The green list is then formed as

$G$7

Thus, preferred tokens are those belonging to shards whose message bits equal the majority bit. The next-token logits $G$8 are modified only by a bias on $G$9:

$V$0

after which the token is sampled by

$V$1

The paper emphasizes that this is the only modification of the token distribution: MajorMark does not impose additional top-$V$2, top-$V$3, or temperature changes beyond the model’s default. Its principal hyperparameters are $V$4 for message length, $V$5 for watermark strength, $V$6 for generated length, and $V$7 for keyed hashing; unlike prior methods, $V$8 is treated as an emergent property of the majority-bit construction rather than as a separately tuned hyperparameter.

3. Decoding mechanism and the MajorMark$V$9 extension

MajorMark’s decoder operates on shard-wise token occurrence counts rather than on counts of tokens falling into a single preferred set. For each generated token $G$0 with $G$1, the decoder reconstructs the shard structure under both majority-bit hypotheses $G$2 using

$G$3

This yields occurrence vectors $G$4, where each component counts how often generated tokens fall into the corresponding shard over the entire text. The decoder computes a skewness proxy by taking the standard deviation

$G$5

and sets

$G$6

The correct majority-bit hypothesis is therefore the one producing the more skewed shard-count distribution (Xu et al., 5 Aug 2025).

Once $G$7 is determined, the decoder applies KMeans with $G$8 to $G$9, using default scikit-learn parameters. The cluster with the higher average count is identified as the set of shards boosted during encoding. If $\gamma$0 denotes the corresponding subset of shard indices, then each bit is assigned by

$\gamma$1

and

$\gamma$2

This procedure replaces direct green-list counting with clustering over shard-wise count geometry.

MajorMark$\gamma$3 modifies the encoding by partitioning $\gamma$4 into $\gamma$5 equal-sized blocks $\gamma$6. At step $\gamma$7, it chooses a block index

$\gamma$8

computes the block’s majority bit $\gamma$9 and its occurrence count $\gamma$0, and uses

$\gamma$1

to permute the vocabulary and partition it into $\gamma$2 shards. The green list is the union of shards whose block bits equal $\gamma$3, and logits are boosted by the same additive bias $\gamma$4.

The MajorMark$\gamma$5 decoder does not use clustering. Instead, for each block it exhaustively enumerates $\gamma$6 and all feasible $\gamma$7, reconstructs the block-level occurrence counts, computes the standard deviation, and selects the configuration $\gamma$8 with maximal skew. It then assigns $\gamma$9 to the top-$2^b$0 shards by count and $2^b$1 to the remaining shards. The paper states that this deterministic procedure requires $2^b$2 full passes over the generated sequence, which remains far below the $2^b$3 enumeration cost of brute-force schemes (Xu et al., 5 Aug 2025).

4. Theoretical properties

For MajorMark, the paper states a theorem on preferred-token-set size: for any message $2^b$4, the green list satisfies

$2^b$5

equivalently $2^b$6. Under the assumption that message bits are i.i.d. Bernoulli$2^b$7, the expected green list ratio is

$2^b$8

The paper remarks that $2^b$9 for finite $b$0, and that $b$1 as $b$2. It also notes that extreme messages $b$3 or $b$4 create $b$5, which makes the watermark ineffective, and can therefore be excluded in practice (Xu et al., 5 Aug 2025).

For MajorMark$b$6, the analogous guarantee is

$b$7

for any block, with expected value

$b$8

Because $b$9 for $^+$00, the expected green list ratio increases relative to MajorMark. The paper interprets this as a text-utility advantage. Extreme all-zero or all-one blocks are again excluded, and the infeasible code fraction is described as negligible; the paper gives the example that at $^+$01 it is approximately $^+$02.

The statistical derivation uses De Moivre–Laplace’s normal approximation and the mean absolute deviation of a normal random variable. This suggests that MajorMark’s theoretical contribution is not merely a new encoder but a redefinition of the watermarking regime: the preferred set can be large because recoverability is tied to shard-level imbalance rather than direct green-list occupancy.

5. Experimental results and robustness

The main experimental setup uses LLaMA-2-7B for generation and LLaMA-2-13B for perplexity evaluation, with additional experiments on Qwen2.5-7B and Gemma-2B; perplexity for those models is computed with Qwen2.5-32B and Gemma-7B, respectively. The main prompt source is C4 news, with additional evaluation on OpenGen and Essays. The paper reports experiments with 20 users, 2 prompts per user, and 250 generated tokens per prompt, giving $^+$03 tokens per user for decoding. Message lengths are $^+$04 and watermark biases are $^+$05. The primary metrics are bit accuracy (BA), perplexity (PPL), and Top-5 hit rate (Xu et al., 5 Aug 2025).

At $^+$06, all methods achieve BA $^+$07 or approximately $^+$08, but MajorMark and MajorMark$^+$09 record the best quality, with average PPL $^+$10 and $^+$11, respectively, and Top-5 hit rate approximately $^+$12–$^+$13. At $^+$14 and $^+$15, MajorMark reports BA $^+$16, PPL $^+$17, and Top-5 $^+$18, while MajorMark$^+$19 reports BA $^+$20, PPL $^+$21, and Top-5 $^+$22. Under the same setting, RSBH reports BA $^+$23 and PPL $^+$24, and MPAC reports BA $^+$25 and PPL $^+$26. At $^+$27, averaged across $^+$28, MajorMark reports BA $^+$29, PPL $^+$30, and Top-5 $^+$31, while MajorMark$^+$32 reports BA $^+$33, PPL $^+$34, and Top-5 $^+$35; RSBH reports BA $^+$36 and PPL $^+$37, and MPAC reports BA $^+$38 and PPL $^+$39.

The robustness study evaluates Copy-Paste, defined as random interleaving of $^+$40 non-watermarked human text while maintaining total length, and Paraphrase, implemented by prompting LLaMA-2-13B with “Paraphrase this: {text}”. At $^+$41, average BA across $^+$42 is reported as $^+$43 for MajorMark and $^+$44 for MajorMark$^+$45. These values correspond to improvements over MPAC of $^+$46 and $^+$47, and over RSBH of $^+$48 and $^+$49.

The ablation studies compare KMeans against Agglomerative Clustering and GMM for MajorMark decoding, with BA stable at strong $^+$50 and KMeans slightly better at low $^+$51. For MajorMark$^+$52, increasing the number of blocks $^+$53 improves PPL in accordance with the theoretical increase in $^+$54; the paper identifies $^+$55 as a balance between BA and PPL. On OpenGen and Essays, MajorMark$^+$56 again achieves the highest BA and lowest PPL across $^+$57 and $^+$58. One reported example is Essays with $^+$59, where MajorMark$^+$60 attains BA $^+$61 and PPL $^+$62, compared with MPAC at BA $^+$63, PPL $^+$64, and RSBH at BA $^+$65, PPL $^+$66.

6. Computational profile, security assumptions, and limitations

MajorMark’s encoding overhead is described as comparable in magnitude to other logit-based watermarking methods. Per generation step it computes the seed, permutes the vocabulary, partitions it into shards, boosts the logits of all tokens in $^+$67, and samples from softmax. In practice, the implementation uses torch.randperm for permutation and simple slicing for partitioning. Because $^+$68, the logit update touches at least half the vocabulary. The MajorMark decoder requires two passes over the generated sequence to evaluate $^+$69, followed by one KMeans operation over $^+$70 shard counts. MajorMark$^+$71 requires $^+$72 passes over the sequence, but the paper still characterizes this as computationally efficient and substantially cheaper than $^+$73 enumeration (Xu et al., 5 Aug 2025).

The security model assumes a secret key $^+$74 known to the decoder. According to the paper, security relies on the secrecy of $^+$75 and the hash construction, because an adversary without $^+$76 cannot reconstruct shard mappings or infer $^+$77 or $^+$78 reliably. The attack surfaces discussed are watermark removal by aggressive paraphrasing or editing designed to disrupt shard-wise count skew, and adversarial detection or stripping of the watermark without access to model logits or hash inputs. False positives on human-written text are not explicitly quantified.

The paper also states several limitations. Because $^+$79 is emergent from the majority-bit design, it is not directly tunable; future work is proposed on decoupling $^+$80 from the encoding while preserving decoding independence from green-list frequency counts. MajorMark$^+$81 adds decoding passes even though it remains efficient relative to brute-force methods. The co-design of $^+$82 and $^+$83 is left as future work. Ethical considerations are framed around provenance and misuse tracing, together with the need for transparency, privacy awareness, and fair-use sensitivity. A plausible implication is that MajorMark is best understood not as a generic watermarking template, but as a specific architectural answer to the claim that multi-bit watermarking must sacrifice output quality in order to remain decodable.