---
title: 'MOLM: Watermarking with LoRA Adapters'
url: https://www.emergentmind.com/topics/mixture-of-lora-markers-molm
type: topic
---

# MOLM: Watermarking with LoRA Adapters

Searching arXiv for recent papers on Mixture of LoRA Markers and closely related methods.
{"query":"Mixture of LoRA Markers MOLM arXiv 2025 LoRA experts routing watermarking", "max_results": 10}
{"query":"2510.00293 MOLM Mixture of LoRA Markers", "max_results": 5}
Mixture of LoRA Markers (MOLM) is a watermarking framework for frozen generative models in which a binary key is encoded as a key-dependent perturbation of model parameters, realized by routing among lightweight Low-Rank Adaptation (LoRA) adapters rather than by retraining a distinct model for each key [2510.00293]. In its main instantiation, MOLM activates one LoRA adapter per routed block according to chunks of the key, preserves the original sampling procedure of the generator, and pairs the marked generator with a learned extractor that decodes the key from generated images. The framework targets imperceptibility, fidelity, verifiability, scalability, and robustness while avoiding per-key retraining [2510.00293].

## 1. Definition, motivation, and terminology

MOLM is motivated by the fact that modern image generators such as Stable Diffusion and FLUX can generate photorealistic images at scale, which raises concerns about detection, attribution, and misuse [2510.00293]. The watermarking objective is not merely to detect that an image is synthetic, but to attribute it to a specific key while maintaining image quality and robustness under realistic distortions, compression, regeneration, averaging attacks, and adversarial attacks on the extractor [2510.00293].

The framework is positioned against three shortcomings of prior watermarking schemes. First, existing methods can be fragile under realistic distortions; the paper notes that regeneration through diffusion can erase watermarks and that averaging attacks can remove or forge content-agnostic patterns. Second, adaptive removal remains a concern under black-box and white-box attacks. Third, many methods require per-key retraining or fine-tuning, making large-scale deployment and key rotation costly [2510.00293].

The term is specific to watermarking. In adjacent PEFT literature, the more common terminology is “Mixture of LoRA Experts” (MoLE), which refers to routing among LoRA adapters for downstream-task adaptation rather than watermarking [2404.13628]. That distinction is substantive: MOLM uses routing to realize key-conditioned parameter perturbations for provenance and attribution, whereas MoLE methods use routing to improve task adaptation, specialization, or efficiency.

## 2. Parameter-perturbation framework and verification theory

MOLM formalizes watermarking as key-dependent perturbation of a frozen generator. Let \(G_\theta(z,c)\) denote a generative model with parameters \(\theta\), latent or noise input \(z\), and optional conditioning \(c\). For a binary key \(k \in \{0,1\}^m\), the marked generator is

\[
\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).
\]

This formulation separates the base model from the watermark: the backbone remains frozen, and the watermark is expressed entirely through \(\Delta \theta(k)\) [2510.00293].

Verification is defined through an extractor \(E_\phi(y)\) that maps an image \(y\) to key logits \(u \in \mathbb{R}^m\), followed by bit decoding \(\tilde{k} = \mathrm{round}(\sigma(u))\). The key-match statistic is the number of matching bits,
\[
D(k,\tilde{k}),
\]
and under the null hypothesis \(H_0\) of no watermark, \(D \sim \mathrm{Binomial}(m, 0.5)\). This yields the false-positive rate
\[
\mathrm{FPR}(\tau) = P(D > \tau \mid H_0) = I_{1/2}(\tau + 1, m - \tau),
\]
where \(I_{1/2}\) is the regularized incomplete beta function [2510.00293]. The framework therefore supports thresholded attribution at a target operating point such as TPR@1%FPR. For \(m = 28\) bits and FPR \(=1\%\), the threshold \(\tau\) corresponds to about 20 matching bits [2510.00293].

This statistical verification layer is central to MOLM’s design. It makes attribution explicit, rather than reducing watermarking to binary detection alone. A plausible implication is that MOLM is intended for operational settings in which many distinct keys must coexist and be auditable under a common statistical criterion.

## 3. Marker mixture architecture and key routing

MOLM instantiates \(\Delta \theta(k)\) with routed LoRA adapters. For a weight \(W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}\), LoRA uses
\[
W = W_0 + s\, U V^\top,
\]
with \(U \in \mathbb{R}^{d_{\text{out}} \times r}\), \(V \in \mathbb{R}^{d_{\text{in}} \times r}\), and \(r \ll \min(d_{\text{out}}, d_{\text{in}})\) [2510.00293]. MOLM inserts multiple LoRA adapters per routed block and selects among them with hard gating:
\[
W_b(k) = W_{b,0} + s_b \sum_{i=1}^{M} \alpha_{b,i}(k)\, U_{b,i} V_{b,i}^\top,
\]
where \(\alpha_{b,i}(k) \in \{0,1\}\) and exactly one adapter is active per routed block in the main implementation [2510.00293].

The routing mechanism is deterministic and key-driven. If \(L\) blocks are routed and each block has \(P\) adapters, the \(m\)-bit key is partitioned into \(L\) chunks \(k_b\) of \(\log_2 P\) bits. Each chunk is converted from binary to a block-local adapter index \(s_b \in \{1,\dots,P\}\), and the corresponding adapter is activated. New keys are therefore realized by changing the routing mask rather than retraining the generator or the adapters [2510.00293].

The main implementation routes ResNet blocks in the VAE decoder of Stable Diffusion v1.5. The default configuration uses \(L = 14\) residual routing layers and \(P = 4\) adapters per layer, yielding
\[
m = L \cdot \log_2(P) = 28 \text{ bits}.
\]
UNet routing is also possible—adding adapters to 22 ResNet, 16 cross-attention, and 16 self-attention layers, for total capacity up to 108 bits—but this degrades fidelity and is not used in the main setup [2510.00293].

| Component | Main setup | Alternative noted |
|---|---|---|
| Routed blocks | 14 VAE decoder residual layers | UNet-wide routing up to 108 bits |
| Adapters per routed block | \(P=4\) | \(P=8\) for 42 bits |
| Default key length | 28 bits | 56 bits with independent bits per convolution |

This architecture is structurally distinct from MoLE systems in language or speech models, where routers choose experts based on token semantics or task statistics [2504.00661]. Here, the routing signal is the watermark key itself.

## 4. Optimization, extraction, and operational workflow

MOLM jointly optimizes the LoRA parameters \(\Psi\) and the extractor parameters \(\phi\) under an objective that balances imperceptibility and robust key recovery:
\[
\min_{\Psi,\phi} \;\mathbb{E}_{k,z,c,T\sim\Pi}
\Big[
\lambda\, \mathcal{L}_{\text{imp}}
\big(
G_{\theta+\Delta\theta(k)}(z,c), G_{\theta}(z,c)
\big)
+ \mathrm{BCE}\big( E_{\phi}( T( G_{\theta+\Delta\theta(k)}(z,c) ) ), k \big)
\Big].
\]
The distortion distribution \(\Pi\) includes crops, rotations, resizes, color jitter, JPEG, and regeneration, and imperceptibility is measured with LPIPS-style perceptual features comparing clean and marked outputs [2510.00293].

The extractor is CNN-based and outputs \(m\) logits. Training uses bitwise binary cross-entropy on distorted images, which hardens decoding against removal attempts and benign corruption [2510.00293]. This design shifts robustness partly from the generator-side perturbation to the decoder-side invariance of the extractor.

The experimental setup uses Stable Diffusion v1.5 at \(512 \times 512\) and FLUX at \(1024 \times 1024\), trained on 10k image–text pairs from MS-COCO 2014. Sampling uses PNDM, 50 denoising steps, and CFG \(= 7.5\), with evaluation on MS-COCO prompts and LAION-Aesthetics captions [2510.00293]. Only LoRA matrices are trainable; the backbone is frozen. Inference time is described as essentially unchanged, and training the adapters and extractor takes approximately one day on a single A100, with no per-key cost [2510.00293].

Operationally, encoding consists of partitioning the key, activating the corresponding adapters blockwise, and sampling with the frozen generator plus routed LoRA updates. Decoding consists of extractor inference, bit rounding, and thresholding via the binomial statistic \(D(k,\tilde{k})\). For multi-user attribution, the decoded bits can be matched against a stored key set and assigned to the top match above threshold [2510.00293].

## 5. Empirical performance, robustness, and ablations

The main empirical result is that MOLM preserves image quality while supporting accurate key recovery under both benign distortions and adversarial pressure [2510.00293]. On Stable Diffusion v1.5, the FID degradation is small, at \(\le 1.5\) relative to vanilla SD, and the paper reports FID \(\approx 27.7\). SSIM and PSNR remain close to baseline, and qualitative examples show negligible artifacts with preserved content [2510.00293].

On the 28-bit Stable Diffusion v1.5 setup, bit accuracy is 0.98 on clean images, 0.91 under crop, 0.84 under rotation, 0.90 under resize, 0.95 under brightness change, and 0.89 under JPEG compression [2510.00293]. Under the detection-oriented metric TPR@1%FPR, the same setup achieves 1.00 on clean samples, 0.98 under crop, 0.98 under rotation, 0.96 under resize, 0.99 under brightness change, and 0.99 under JPEG compression [2510.00293]. Under distribution shift to LAION-Aesthetics, TPR@1%FPR is approximately 0.99 on clean data and at least 0.94 across distortions. On FLUX, TPR@1%FPR is at least 0.92 across distortions, with clean performance at 0.98 [2510.00293].

Robustness extends to more aggressive attacks. Under BMSHJ2018 compression, augmentation training improves bit accuracy from \(0.50/0.77/0.96\) to \(0.61/0.95/0.99\) at quality levels \(q = 1/4/8\). Under Cheng2020 compression, it improves from \(0.70/0.82/0.93\) to \(0.94/0.95/0.97\) at \(q = 1/3/6\). Under diffusion regeneration with 30/60/100 steps, augmentation-trained MOLM attains \(0.85/0.85/0.82\), compared with \(0.72/0.69/0.62\) without augmentation. Under adversarial PGD with MSE \(\epsilon = 10^{-3}/10^{-2}/10^{-1}\), augmentation-trained accuracy is \(1.00/0.99/0.96\), whereas the non-augmented model drops to \(0.93/0.81/0.60\) [2510.00293].

Averaging attacks are treated separately. Forgery through averaging remains at chance, approximately 0.5, even in same-message settings up to \(k = 5000\) images. For removal through averaging, MOLM maintains at least 0.96 bit accuracy up to \(k = 5000\), while WOUAF drops below 0.90 [2510.00293].

Ablations show the expected capacity–fidelity trade-off. With adapter rank \(r=64\), the model attains FID 27.7 and bit accuracy 0.98; with \(r=32\), FID 28.2 and bit accuracy 0.96; with \(r=16\), FID 29.5 and bit accuracy 0.91; with \(r=8\), FID 34.8 and bit accuracy 0.75 [2510.00293]. Increasing capacity to 42 bits by setting \(P=8\) yields bit accuracy 0.90 and FID 27.3, while a 56-bit configuration yields bit accuracy 0.89 and FID 27.9. UNet routing at 108 bits increases capacity but degrades fidelity, with visible artifacts and FID increases of 4–5 [2510.00293].

## 6. Threat model, limitations, and relation to adjacent LoRA-mixture research

The stated threat model includes distortions, compression, regeneration, averaging, and adversarial perturbations. It assumes black-box access to the generator API and no access to model or extractor parameters for the main attacker, while also evaluating a white-box adversary against the extractor via PGD under perceptual constraints [2510.00293]. The paper attributes MOLM’s robustness to three factors: the watermark is distributed across many blocks in the generator’s internal feature flow; it is not tied to deterministic sampling or inversion; and key updates are cheap because only routing masks change, reducing the attack window for per-key removal strategies [2510.00293].

Several limitations are explicit. Extremely strong compression or regeneration can increase BER without augmentation training. Capacity and fidelity trade off against one another as more layers are routed or more bits are encoded. Certain content types and heavy geometric changes, including severe rotations and crops, can lower bit accuracy modestly. The paper also notes ethical considerations: watermarking aids provenance and attribution, but should avoid content bias, respect privacy, and be accompanied by transparent verification and key-management policies [2510.00293].

Within the broader LoRA-mixture landscape, MOLM is best understood as a watermarking specialization rather than a PEFT routing method. Expert-routing systems such as MoLE and DynMoLE route LoRA experts to improve downstream-task adaptation, convergence, or expert utilization in language models [2404.13628][2504.00661]. MOLM instead routes LoRA adapters according to a binary key, couples them to an extractor and a hypothesis test, and uses the resulting marked generator for attribution. This suggests a broader methodological pattern—routing over low-rank modules—but the objective, threat model, and verification semantics are different.

MOLM therefore occupies a distinct position at the intersection of generative watermarking, parameter-efficient adaptation, and statistical attribution. Its defining contribution is not merely to embed a watermark in weights, but to do so in a routing-based form that decouples key updates from retraining while maintaining imperceptibility, fidelity, verifiability, and robustness [2510.00293].

Source: https://www.emergentmind.com/topics/mixture-of-lora-markers-molm