---
title: 'MajorMark: Watermarking Method for LLMs'
url: https://www.emergentmind.com/topics/majormark
type: topic
---

# MajorMark: Watermarking Method for LLMs

MajorMark is a multi-bit watermarking method for large language models that embeds a binary message $m \in \{0,1\}^b$ into generated text by selecting preferred token sets according to the message’s majority bit, rather than by fixing a small preferred set and decoding through direct token-frequency counting. The method is designed to improve the standard trade-off between text quality and decoding accuracy in LLM watermarking. Its extension, MajorMark$^+$, partitions the message into multiple blocks and deterministically decodes each block, with the stated aim of further improving both watermarked-text quality and message recovery accuracy [2508.03829].

## 1. Problem setting and design objective

MajorMark is situated in the literature on multi-bit watermarking for LLMs, where the objective is to embed a recoverable binary message into generated text for origin verification and misuse tracing. In this setting, prior methods face a quality–decoding accuracy trade-off governed by the green list ratio $\gamma = |G|/|V|$, where $G$ is the preferred token set and $V$ is the vocabulary. When decoders rely on counting how many generated tokens fall into $G$, small $\gamma$ strengthens the decoding signal but distorts the output distribution, whereas large $\gamma$ preserves generation quality but weakens statistical recoverability. The paper identifies this dependence on token-frequency-based decoding as the core limitation of prior multi-bit schemes such as MPAC and RSBH, and notes that some methods, including CTWL, CycleShift, and DepthW, additionally require brute-force enumeration across $2^b$ messages as $b$ grows [2508.03829].

Within this formulation, MajorMark redefines the role of the green list. At the $t$-th generation step, the encoder boosts the logits of tokens in $G$ by a bias $\delta$, increasing their sampling probability after softmax. The method’s central claim is that reliable decoding need not depend on direct green-list frequency counting. Instead, it can depend on shard-wise occurrence structure induced by the message itself. This permits a large preferred-token set and, consequently, less distortion of the base model’s distribution.

## 2. Majority bit-aware encoding

The defining concept of MajorMark is the majority bit. For a binary message $m \in \{0,1\}^{b \ge 2}$, let $h_0$ and $h_1$ denote the number of occurrences of bit $0$ and bit $1$, respectively. The majority bit $\lambda \in \{0,1\}$ is defined as $\lambda = 1$ if $h_1 \ge h_0$, and $\lambda = 0$ otherwise. In the tie case $h_0 = h_1$, the method sets $\lambda = 1$ by default. Because the majority bit appears at least $\lceil b/2 \rceil$ times, it provides a stable signal for preferred-token construction [2508.03829].

At each generation step $t$, MajorMark computes a pseudo-random seed
$$
s = \operatorname{Hash}(k, x_{t-1}, x_{t-2}, \lambda),
$$
where $k$ is a secret key and $x_{t-1}, x_{t-2}$ are the last two generated tokens. The paper explicitly contrasts this with the common practice of using $\operatorname{Hash}(k, x_{t-1})$, arguing that inclusion of $x_{t-2}$ balances seed diversity and inclusion of $\lambda$ enables recovery of the correct majority-bit hypothesis during decoding.

Using $s$, the encoder permutes the vocabulary $V \to V'$ and evenly partitions $V'$ into $b$ disjoint shards $[V_1, V_2, \dots, V_b]$. The green list is then formed as
$$
G = \bigcup_{i : m_i = \lambda} V_i.
$$
Thus, preferred tokens are those belonging to shards whose message bits equal the majority bit. The next-token logits $\ell^t = f(\cdot)$ are modified only by a bias on $G$:
$$
\ell^t_j \leftarrow \ell^t_j + \delta \quad \text{if } j \in G,
$$
after which the token is sampled by
$$
x_t \sim \operatorname{Softmax}(\ell^t).
$$

The paper emphasizes that this is the only modification of the token distribution: MajorMark does not impose additional top-$k$, top-$p$, or temperature changes beyond the model’s default. Its principal hyperparameters are $b$ for message length, $\delta$ for watermark strength, $T$ for generated length, and $k$ for keyed hashing; unlike prior methods, $\gamma$ is treated as an emergent property of the majority-bit construction rather than as a separately tuned hyperparameter.

## 3. Decoding mechanism and the MajorMark$^+$ extension

MajorMark’s decoder operates on shard-wise token occurrence counts rather than on counts of tokens falling into a single preferred set. For each generated token $x_t$ with $t \ge 3$, the decoder reconstructs the shard structure under both majority-bit hypotheses $\lambda' \in \{0,1\}$ using
$$
\operatorname{Hash}(k, x_{t-1}, x_{t-2}, \lambda').
$$
This yields occurrence vectors $\operatorname{occ}[\lambda'] \in \mathbb{R}^b$, where each component counts how often generated tokens fall into the corresponding shard over the entire text. The decoder computes a skewness proxy by taking the standard deviation
$$
\sigma_{\lambda'} = \operatorname{std}(\operatorname{occ}[\lambda']),
$$
and sets
$$
\lambda \leftarrow \arg\max(\sigma_0, \sigma_1).
$$
The correct majority-bit hypothesis is therefore the one producing the more skewed shard-count distribution [2508.03829].

Once $\lambda$ is determined, the decoder applies KMeans with $K=2$ to $\operatorname{occ}[\lambda]$, using default scikit-learn parameters. The cluster with the higher average count is identified as the set of shards boosted during encoding. If $C$ denotes the corresponding subset of shard indices, then each bit is assigned by
$$
m_i' = \lambda \quad \text{if } i \in C,
$$
and
$$
m_i' = 1-\lambda \quad \text{otherwise}.
$$
This procedure replaces direct green-list counting with clustering over shard-wise count geometry.

MajorMark$^+$ modifies the encoding by partitioning $m$ into $r$ equal-sized blocks $\{m_1,\dots,m_r\}$. At step $t$, it chooses a block index
$$
p \leftarrow (x_{t-1} + x_{t-2}) \bmod r,
$$
computes the block’s majority bit $\lambda_p$ and its occurrence count $h_{\lambda_p}$, and uses
$$
\operatorname{Hash}(k, x_{t-1}, x_{t-2}, \lambda_p, h_{\lambda_p})
$$
to permute the vocabulary and partition it into $b/r$ shards. The green list is the union of shards whose block bits equal $\lambda_p$, and logits are boosted by the same additive bias $\delta$.

The MajorMark$^+$ decoder does not use clustering. Instead, for each block it exhaustively enumerates $\lambda' \in \{0,1\}$ and all feasible $h' \in \mathcal{H}_{\lambda'}$, reconstructs the block-level occurrence counts, computes the standard deviation, and selects the configuration $(\lambda_p, h_{\lambda_p})$ with maximal skew. It then assigns $\lambda_p$ to the top-$h_{\lambda_p}$ shards by count and $1-\lambda_p$ to the remaining shards. The paper states that this deterministic procedure requires $(b-r)$ full passes over the generated sequence, which remains far below the $2^b$ enumeration cost of brute-force schemes [2508.03829].

## 4. Theoretical properties

For MajorMark, the paper states a theorem on preferred-token-set size: for any message $m \in \{0,1\}^{b \ge 2}$, the green list satisfies
$$
|G| \ge 0.5|V|,
$$
equivalently $\gamma \ge 0.5$. Under the assumption that message bits are i.i.d. Bernoulli$(0.5)$, the expected green list ratio is
$$
E_m[\gamma] = 0.5 + \frac{1}{\sqrt{2\pi b}}.
$$
The paper remarks that $E_m[\gamma] > 0.5$ for finite $b$, and that $E_m[\gamma] \to 0.5$ as $b \to \infty$. It also notes that extreme messages $m = 0^b$ or $1^b$ create $\gamma = 1.0$, which makes the watermark ineffective, and can therefore be excluded in practice [2508.03829].

For MajorMark$^+$, the analogous guarantee is
$$
\gamma \ge 0.5
$$
for any block, with expected value
$$
E_m[\gamma] = 0.5 + \frac{1}{\sqrt{2\pi (b/r)}}.
$$
Because $b/r < b$ for $r>1$, the expected green list ratio increases relative to MajorMark. The paper interprets this as a text-utility advantage. Extreme all-zero or all-one blocks are again excluded, and the infeasible code fraction is described as negligible; the paper gives the example that at $b=32, r=2$ it is approximately $0.006\%$.

The statistical derivation uses De Moivre–Laplace’s normal approximation and the mean absolute deviation of a normal random variable. *This suggests* that MajorMark’s theoretical contribution is not merely a new encoder but a redefinition of the watermarking regime: the preferred set can be large because recoverability is tied to shard-level imbalance rather than direct green-list occupancy.

## 5. Experimental results and robustness

The main experimental setup uses LLaMA-2-7B for generation and LLaMA-2-13B for perplexity evaluation, with additional experiments on Qwen2.5-7B and Gemma-2B; perplexity for those models is computed with Qwen2.5-32B and Gemma-7B, respectively. The main prompt source is C4 news, with additional evaluation on OpenGen and Essays. The paper reports experiments with 20 users, 2 prompts per user, and 250 generated tokens per prompt, giving $T=500$ tokens per user for decoding. Message lengths are $b \in \{8,32,64\}$ and watermark biases are $\delta \in \{2,4,6\}$. The primary metrics are bit accuracy (BA), perplexity (PPL), and Top-5 hit rate [2508.03829].

At $b=8$, all methods achieve BA $=100\%$ or approximately $100\%$, but MajorMark and MajorMark$^+$ record the best quality, with average PPL $=6.06$ and $5.72$, respectively, and Top-5 hit rate approximately $90$–$91\%$. At $b=32$ and $\delta=4$, MajorMark reports BA $=99.06$, PPL $=6.32$, and Top-5 $=89.28$, while MajorMark$^+$ reports BA $=100.00$, PPL $=6.05$, and Top-5 $=90.17$. Under the same setting, RSBH reports BA $=97.66$ and PPL $=6.43$, and MPAC reports BA $=98.75$ and PPL $=9.37$. At $b=64$, averaged across $\delta$, MajorMark reports BA $=92.03$, PPL $=6.57$, and Top-5 $=89.36$, while MajorMark$^+$ reports BA $=94.82$, PPL $=6.29$, and Top-5 $=89.85$; RSBH reports BA $=89.74$ and PPL $=6.85$, and MPAC reports BA $=90.52$ and PPL $=9.86$.

The robustness study evaluates Copy-Paste, defined as random interleaving of $10\%$ non-watermarked human text while maintaining total length, and Paraphrase, implemented by prompting LLaMA-2-13B with “Paraphrase this: \{text\}”. At $b=32$, average BA across $\delta \in \{2,4,6\}$ is reported as $93.44\%$ for MajorMark and $94.48\%$ for MajorMark$^+$. These values correspond to improvements over MPAC of $+1.00\%$ and $+2.04\%$, and over RSBH of $+3.96\%$ and $+5.00\%$.

The ablation studies compare KMeans against Agglomerative Clustering and GMM for MajorMark decoding, with BA stable at strong $\delta$ and KMeans slightly better at low $\delta$. For MajorMark$^+$, increasing the number of blocks $r$ improves PPL in accordance with the theoretical increase in $E_m[\gamma]$; the paper identifies $r=2$ as a balance between BA and PPL. On OpenGen and Essays, MajorMark$^+$ again achieves the highest BA and lowest PPL across $b \in \{32,64\}$ and $\delta \in \{2,4,6\}$. One reported example is Essays with $b=64$, where MajorMark$^+$ attains BA $=95.21\%$ and PPL $=6.03$, compared with MPAC at BA $=89.22\%$, PPL $=9.23$, and RSBH at BA $=89.69\%$, PPL $=6.22$.

## 6. Computational profile, security assumptions, and limitations

MajorMark’s encoding overhead is described as comparable in magnitude to other logit-based watermarking methods. Per generation step it computes the seed, permutes the vocabulary, partitions it into shards, boosts the logits of all tokens in $G$, and samples from softmax. In practice, the implementation uses `torch.randperm` for permutation and simple slicing for partitioning. Because $|G| \ge 0.5|V|$, the logit update touches at least half the vocabulary. The MajorMark decoder requires two passes over the generated sequence to evaluate $\lambda' \in \{0,1\}$, followed by one KMeans operation over $b$ shard counts. MajorMark$^+$ requires $(b-r)$ passes over the sequence, but the paper still characterizes this as computationally efficient and substantially cheaper than $2^b$ enumeration [2508.03829].

The security model assumes a secret key $k$ known to the decoder. According to the paper, security relies on the secrecy of $k$ and the hash construction, because an adversary without $k$ cannot reconstruct shard mappings or infer $\lambda$ or $h$ reliably. The attack surfaces discussed are watermark removal by aggressive paraphrasing or editing designed to disrupt shard-wise count skew, and adversarial detection or stripping of the watermark without access to model logits or hash inputs. False positives on human-written text are not explicitly quantified.

The paper also states several limitations. Because $\gamma$ is emergent from the majority-bit design, it is not directly tunable; future work is proposed on decoupling $\gamma$ from the encoding while preserving decoding independence from green-list frequency counts. MajorMark$^+$ adds decoding passes even though it remains efficient relative to brute-force methods. The co-design of $\gamma$ and $\delta$ is left as future work. Ethical considerations are framed around provenance and misuse tracing, together with the need for transparency, privacy awareness, and fair-use sensitivity. *A plausible implication is* that MajorMark is best understood not as a generic watermarking template, but as a specific architectural answer to the claim that multi-bit watermarking must sacrifice output quality in order to remain decodable.

Source: https://www.emergentmind.com/topics/majormark