MOLM: Watermarking with LoRA Adapters

• The paper introduces a watermarking framework that encodes a binary key as key-dependent perturbations via routed LoRA adapters, eliminating the need for per-key retraining.
• MOLM is evaluated on models like Stable Diffusion, demonstrating high fidelity with minimal FID degradation and robust key recovery under distortions such as cropping, rotation, and compression.
• The framework decouples watermarking from retraining by routing key chunks across model layers, enabling scalable, efficient, and statistically verifiable attribution in generative models.

Searching arXiv for papers on Mixture of LoRA Markers and closely related methods. {"query":"Mixture of LoRA Markers MOLM arXiv 2025 LoRA experts routing watermarking", "max_results": 10} {"query":"(Fares et al., 30 Sep 2025) MOLM Mixture of LoRA Markers", "max_results": 5} Mixture of LoRA Markers (MOLM) is a watermarking framework for frozen generative models in which a binary key is encoded as a key-dependent perturbation of model parameters, realized by routing among lightweight Low-Rank Adaptation (LoRA) adapters rather than by retraining a distinct model for each key (Fares et al., 30 Sep 2025). In its main instantiation, MOLM activates one LoRA adapter per routed block according to chunks of the key, preserves the original sampling procedure of the generator, and pairs the marked generator with a learned extractor that decodes the key from generated images. The framework targets imperceptibility, fidelity, verifiability, scalability, and robustness while avoiding per-key retraining (Fares et al., 30 Sep 2025).

1. Definition, motivation, and terminology

MOLM is motivated by the fact that modern image generators such as Stable Diffusion and FLUX can generate photorealistic images at scale, which raises concerns about detection, attribution, and misuse (Fares et al., 30 Sep 2025). The watermarking objective is not merely to detect that an image is synthetic, but to attribute it to a specific key while maintaining image quality and robustness under realistic distortions, compression, regeneration, averaging attacks, and adversarial attacks on the extractor (Fares et al., 30 Sep 2025).

The framework is positioned against three shortcomings of prior watermarking schemes. First, existing methods can be fragile under realistic distortions; the paper notes that regeneration through diffusion can erase watermarks and that averaging attacks can remove or forge content-agnostic patterns. Second, adaptive removal remains a concern under black-box and white-box attacks. Third, many methods require per-key retraining or fine-tuning, making large-scale deployment and key rotation costly (Fares et al., 30 Sep 2025).

The term is specific to watermarking. In adjacent PEFT literature, the more common terminology is “Mixture of LoRA Experts” (MoLE), which refers to routing among LoRA adapters for downstream-task adaptation rather than watermarking (Wu et al., 2024). That distinction is substantive: MOLM uses routing to realize key-conditioned parameter perturbations for provenance and attribution, whereas MoLE methods use routing to improve task adaptation, specialization, or efficiency.

2. Parameter-perturbation framework and verification theory

MOLM formalizes watermarking as key-dependent perturbation of a frozen generator. Let $G_\theta(z,c)$ denote a generative model with parameters $\theta$, latent or noise input $z$, and optional conditioning $c$. For a binary key $k \in \{0,1\}^m$, the marked generator is

$$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$

This formulation separates the base model from the watermark: the backbone remains frozen, and the watermark is expressed entirely through $\Delta \theta(k)$ (Fares et al., 30 Sep 2025).

Verification is defined through an extractor $E_\phi(y)$ that maps an image $y$ to key logits $u \in \mathbb{R}^m$, followed by bit decoding $\theta$0. The key-match statistic is the number of matching bits,

$\theta$1

and under the null hypothesis $\theta$2 of no watermark, $\theta$3. This yields the false-positive rate

$\theta$4

where $\theta$5 is the regularized incomplete beta function (Fares et al., 30 Sep 2025). The framework therefore supports thresholded attribution at a target operating point such as TPR@1%FPR. For $\theta$6 bits and FPR $\theta$7, the threshold $\theta$8 corresponds to about 20 matching bits (Fares et al., 30 Sep 2025).

This statistical verification layer is central to MOLM’s design. It makes attribution explicit, rather than reducing watermarking to binary detection alone. A plausible implication is that MOLM is intended for operational settings in which many distinct keys must coexist and be auditable under a common statistical criterion.

3. Marker mixture architecture and key routing

MOLM instantiates $\theta$9 with routed LoRA adapters. For a weight $z$0, LoRA uses

$z$1

with $z$2, $z$3, and $z$4 (Fares et al., 30 Sep 2025). MOLM inserts multiple LoRA adapters per routed block and selects among them with hard gating: $z$5 where $z$6 and exactly one adapter is active per routed block in the main implementation (Fares et al., 30 Sep 2025).

The routing mechanism is deterministic and key-driven. If $z$7 blocks are routed and each block has $z$8 adapters, the $z$9-bit key is partitioned into $c$0 chunks $c$1 of $c$2 bits. Each chunk is converted from binary to a block-local adapter index $c$3, and the corresponding adapter is activated. New keys are therefore realized by changing the routing mask rather than retraining the generator or the adapters (Fares et al., 30 Sep 2025).

The main implementation routes ResNet blocks in the VAE decoder of Stable Diffusion v1.5. The default configuration uses $c$4 residual routing layers and $c$5 adapters per layer, yielding

$c$6

UNet routing is also possible—adding adapters to 22 ResNet, 16 cross-attention, and 16 self-attention layers, for total capacity up to 108 bits—but this degrades fidelity and is not used in the main setup (Fares et al., 30 Sep 2025).

Component	Main setup	Alternative noted
Routed blocks	14 VAE decoder residual layers	UNet-wide routing up to 108 bits
Adapters per routed block	$c$7	$c$8 for 42 bits
Default key length	28 bits	56 bits with independent bits per convolution

This architecture is structurally distinct from MoLE systems in language or speech models, where routers choose experts based on token semantics or task statistics (Li et al., 1 Apr 2025). Here, the routing signal is the watermark key itself.

4. Optimization, extraction, and operational workflow

MOLM jointly optimizes the LoRA parameters $c$9 and the extractor parameters $k \in \{0,1\}^m$0 under an objective that balances imperceptibility and robust key recovery: $k \in \{0,1\}^m$1 The distortion distribution $k \in \{0,1\}^m$2 includes crops, rotations, resizes, color jitter, JPEG, and regeneration, and imperceptibility is measured with LPIPS-style perceptual features comparing clean and marked outputs (Fares et al., 30 Sep 2025).

The extractor is CNN-based and outputs $k \in \{0,1\}^m$3 logits. Training uses bitwise binary cross-entropy on distorted images, which hardens decoding against removal attempts and benign corruption (Fares et al., 30 Sep 2025). This design shifts robustness partly from the generator-side perturbation to the decoder-side invariance of the extractor.

The experimental setup uses Stable Diffusion v1.5 at $k \in \{0,1\}^m$4 and FLUX at $k \in \{0,1\}^m$5, trained on 10k image–text pairs from MS-COCO 2014. Sampling uses PNDM, 50 denoising steps, and CFG $k \in \{0,1\}^m$6, with evaluation on MS-COCO prompts and LAION-Aesthetics captions (Fares et al., 30 Sep 2025). Only LoRA matrices are trainable; the backbone is frozen. Inference time is described as essentially unchanged, and training the adapters and extractor takes approximately one day on a single A100, with no per-key cost (Fares et al., 30 Sep 2025).

Operationally, encoding consists of partitioning the key, activating the corresponding adapters blockwise, and sampling with the frozen generator plus routed LoRA updates. Decoding consists of extractor inference, bit rounding, and thresholding via the binomial statistic $k \in \{0,1\}^m$7. For multi-user attribution, the decoded bits can be matched against a stored key set and assigned to the top match above threshold (Fares et al., 30 Sep 2025).

5. Empirical performance, robustness, and ablations

The main empirical result is that MOLM preserves image quality while supporting accurate key recovery under both benign distortions and adversarial pressure (Fares et al., 30 Sep 2025). On Stable Diffusion v1.5, the FID degradation is small, at $k \in \{0,1\}^m$8 relative to vanilla SD, and the paper reports FID $k \in \{0,1\}^m$9. SSIM and PSNR remain close to baseline, and qualitative examples show negligible artifacts with preserved content (Fares et al., 30 Sep 2025).

On the 28-bit Stable Diffusion v1.5 setup, bit accuracy is 0.98 on clean images, 0.91 under crop, 0.84 under rotation, 0.90 under resize, 0.95 under brightness change, and 0.89 under JPEG compression (Fares et al., 30 Sep 2025). Under the detection-oriented metric TPR@1%FPR, the same setup achieves 1.00 on clean samples, 0.98 under crop, 0.98 under rotation, 0.96 under resize, 0.99 under brightness change, and 0.99 under JPEG compression (Fares et al., 30 Sep 2025). Under distribution shift to LAION-Aesthetics, TPR@1%FPR is approximately 0.99 on clean data and at least 0.94 across distortions. On FLUX, TPR@1%FPR is at least 0.92 across distortions, with clean performance at 0.98 (Fares et al., 30 Sep 2025).

Robustness extends to more aggressive attacks. Under BMSHJ2018 compression, augmentation training improves bit accuracy from $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$0 to $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$1 at quality levels $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$2. Under Cheng2020 compression, it improves from $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$3 to $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$4 at $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$5. Under diffusion regeneration with 30/60/100 steps, augmentation-trained MOLM attains $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$6, compared with $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$7 without augmentation. Under adversarial PGD with MSE $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$8, augmentation-trained accuracy is $$\tilde{G}_{\theta,k}(z,c) := G_{\theta + \Delta\theta(k)}(z,c).$$9, whereas the non-augmented model drops to $\Delta \theta(k)$0 (Fares et al., 30 Sep 2025).

Averaging attacks are treated separately. Forgery through averaging remains at chance, approximately 0.5, even in same-message settings up to $\Delta \theta(k)$1 images. For removal through averaging, MOLM maintains at least 0.96 bit accuracy up to $\Delta \theta(k)$2, while WOUAF drops below 0.90 (Fares et al., 30 Sep 2025).

Ablations show the expected capacity–fidelity trade-off. With adapter rank $\Delta \theta(k)$3, the model attains FID 27.7 and bit accuracy 0.98; with $\Delta \theta(k)$4, FID 28.2 and bit accuracy 0.96; with $\Delta \theta(k)$5, FID 29.5 and bit accuracy 0.91; with $\Delta \theta(k)$6, FID 34.8 and bit accuracy 0.75 (Fares et al., 30 Sep 2025). Increasing capacity to 42 bits by setting $\Delta \theta(k)$7 yields bit accuracy 0.90 and FID 27.3, while a 56-bit configuration yields bit accuracy 0.89 and FID 27.9. UNet routing at 108 bits increases capacity but degrades fidelity, with visible artifacts and FID increases of 4–5 (Fares et al., 30 Sep 2025).

6. Threat model, limitations, and relation to adjacent LoRA-mixture research

The stated threat model includes distortions, compression, regeneration, averaging, and adversarial perturbations. It assumes black-box access to the generator API and no access to model or extractor parameters for the main attacker, while also evaluating a white-box adversary against the extractor via PGD under perceptual constraints (Fares et al., 30 Sep 2025). The paper attributes MOLM’s robustness to three factors: the watermark is distributed across many blocks in the generator’s internal feature flow; it is not tied to deterministic sampling or inversion; and key updates are cheap because only routing masks change, reducing the attack window for per-key removal strategies (Fares et al., 30 Sep 2025).

Several limitations are explicit. Extremely strong compression or regeneration can increase BER without augmentation training. Capacity and fidelity trade off against one another as more layers are routed or more bits are encoded. Certain content types and heavy geometric changes, including severe rotations and crops, can lower bit accuracy modestly. The paper also notes ethical considerations: watermarking aids provenance and attribution, but should avoid content bias, respect privacy, and be accompanied by transparent verification and key-management policies (Fares et al., 30 Sep 2025).

Within the broader LoRA-mixture landscape, MOLM is best understood as a watermarking specialization rather than a PEFT routing method. Expert-routing systems such as MoLE and DynMoLE route LoRA experts to improve downstream-task adaptation, convergence, or expert utilization in LLMs (Wu et al., 2024, Li et al., 1 Apr 2025). MOLM instead routes LoRA adapters according to a binary key, couples them to an extractor and a hypothesis test, and uses the resulting marked generator for attribution. This suggests a broader methodological pattern—routing over low-rank modules—but the objective, threat model, and verification semantics are different.

MOLM therefore occupies a distinct position at the intersection of generative watermarking, parameter-efficient adaptation, and statistical attribution. Its defining contribution is not merely to embed a watermark in weights, but to do so in a routing-based form that decouples key updates from retraining while maintaining imperceptibility, fidelity, verifiability, and robustness (Fares et al., 30 Sep 2025).