Robust Steganography from LLMs

• The paper establishes formal frameworks and threat models for encoding messages, ensuring correctness, invisibility, and resilience to local and semantic adversarial edits.
• It introduces distribution-preserving methods like arithmetic coding, ShiMer, and SyncPool that achieve near-optimal embedding rates while maintaining statistical indistinguishability from natural text.
• The work highlights robust semantic steganography and adaptive algorithms that effectively counter perturbations such as paraphrasing and token-level edits, enhancing secure communication.

Robust steganography from LLMs constitutes a convergence of cryptography, information theory, and modern natural language processing. The field focuses on concealing arbitrary messages within syntactically and semantically plausible text generated by LLMs, maintaining both imperceptibility to adversaries and robustness to realistic channel perturbations. Recent research has led to formal taxonomies of robustness, principled recipes for maximal throughput under statistical constraints, provably secure protocols, and practical systems combining deep learning, cryptography, and symbolic reasoning.

1. Formal Frameworks and Threat Models

Robust LLM steganography is established within the framework of symmetric-key or public-key channels, integrating requirements for correctness, invisibility (indistinguishability from natural text), and robustness to channel noise or tampering (Perry et al., 11 Apr 2025). The standard primitive is a triple:

• $\text{KeyGen}(1^\lambda) \to k$, producing a secret key,
• $\text{Enc}(k, m, h) \to \text{st} \in T^*$, mapping secret message $m$ and context/history $h$ to a stegotext,
• $$\text{Dec}(k, h, \text{st}) \to m \in M \cup \{\bot\}$$, decoding or failing.

Robustness is formalized in two principal senses:

• Weak/Local Robustness: Guarantees recovery under a family $\mathcal{F}_{k, \epsilon}$ of locally consistent functions; for example, adversaries making small local edits (n-gram swaps, token insertions/deletions), parameterized via $k$ (window size) and $\epsilon$ (fraction preserved).
• Strong/Semantic Robustness: Ensures recoverability under paraphrase or rewording attacks. Here, adversarial output $\text{st}'$ must be semantically close to $\text{st}$ with respect to a metric (e.g., embedding-space cosine distance $\leq \delta$) (Perry et al., 11 Apr 2025, Bai et al., 2024).

Invisibility/security mandates that a computationally bounded adversary with access to either the genuine channel or the steganographic encoder cannot distinguish the two with non-negligible advantage. In the information-theoretic regime (for distribution-preserving schemes), the induced output distribution matches the LLM's own, yielding $$D_{\mathrm{KL}}(P_\mathrm{stego}\|P_\mathrm{cover}) = 0$$ (Ziegler et al., 2019, Qi et al., 2024, Norelli et al., 22 Oct 2025).

2. Distribution-Preserving and Information-Theoretic Protocols

2.1 Arithmetic Coding Approaches

Principle: Steganography as reversed arithmetic coding on LLM token distributions. The secret bitstream is mapped to a real number in $[0,1)$; at each token step, the cover distribution $q$ is used to partition $[0,1)$, and the next token is determined so as to narrow the interval corresponding to the secret (Ziegler et al., 2019, Huang et al., 2024).

Optimized Distribution Selection: Embedding is cast as maximizing entropy of a new token distribution $p$ subject to a statistical constraint (e.g., per-step KL or TV divergence $D_{KL}(p\|q) \leq \delta$):

$$\max_{p} -\sum_x p_x \log p_x \quad \text{s.t.} \quad \sum_x p_x \log \frac{p_x}{q_x} \leq \delta$$

The closed-form solution is exponential flattening:

$$p^*_x = \frac{q_x^{\alpha}}{\sum_y q_y^{\alpha}}, \quad \alpha \in [0,1], \quad \text{found so that } D_{KL}(p^*\|q) = \delta$$

Embedding proceeds via an arithmetic decoder using $p^*$ at each step (Huang et al., 2024).

Capacity and Imperceptibility: Theoretically optimal: for a $k$-ary vocabulary, average embedding rate is the entropy $H(q)$. With careful $\delta$ selection (e.g., adaptive to local entropy), and practical vocabulary truncation, methods achieve near-optimal embedding rate at stringently bounded detectability (see Table below).

Method	KL ≈ 0.02	KL ≈ 0.04	KL ≈ 0.06
Truncation Baseline	12 B	15 B	17 B
OD-Stega	15 B	18 B	20 B

—Table: Bytes per 25 tokens under different constraints (Huang et al., 2024).

Practicalities: Tokenization mismatch is addressed via nonce-prefixed messages and verification. Embedding adapts dynamically to entropy, and robust decoding is provably guaranteed when the candidate pool is filtered for ambiguous tokens (see (Yan et al., 28 Aug 2025, Qi et al., 2024)).

2.2 Shifting-Merging (ShiMer)

ShiMer applies a keyed pseudorandom shift of the probability interval per token, sampling via "rolling and rotating," and adds a reordering algorithm to minimize interval splitting. The mechanism preserves exact output distributions while allowing nearly full entropy utilization per token ($>98\%$ in practice) and extremely low detectability by statistical means (Bai et al., 1 Jan 2025). It is cryptographically secure provided the PRG is secure.

Model	Method	Utilization	Capacity (bit/s)	PPL
Llama2	ShiMer (R)	98%	44.3	5.98
	METEOR	59.8%	26.8	6.01

—Table: ShiMer capacity and efficiency (Bai et al., 1 Jan 2025).

2.3 Distribution-preserving Disambiguation

The SyncPool wrapper guarantees zero extraction errors due to subword segmentation ambiguity, using ambiguity pools and synchronized pseudorandom sampling without altering the underlying per-token distribution (Qi et al., 2024). Empirical error rates drop from 2–5% to 0% while maintaining KL=0 and modest throughput reduction.

3. Robustness Against Adversarial Perturbations

3.1 Watermarking and Weak Robustness

Watermarking-based encoding diffuses message bits throughout a text via small statistical biases in token selection, tested via z-scores after local or global edits. Under local n-gram shuffles, recovery rates near 100% are obtained; however, even minor paraphrasing or synonym replacement rapidly degrades performance, demonstrating the limits of watermark-based weak robustness (Perry et al., 11 Apr 2025).

3.2 Embedding-based and Strong Robustness

Robust semantic steganography (Bai et al., 2024, Perry et al., 11 Apr 2025) maps messages into semantic classes, buckets, or sentence types (e.g., via ontology-entity trees or LSH over embedding spaces). Extraction is resilient against paraphrasing and token-level modifications provided the semantics are preserved.

• Semantic Steganography with Ontology Trees: Maps secret bits through arithmetic coding on an ontology-entity tree, achieving $D_{KL}=0$ security and decoding success rates $>80\%$ under strong paraphrasing or channel noise (Bai et al., 2024).
• Embedding-based Robustness: Generates candidate text until a hash of its embedding matches the desired message bits. Under paraphrasing (semantic distance $\leq \delta$), chunkwise recovery remains $>95\%$ while classical variable-rate (METEOR-like) methods completely fail (Perry et al., 11 Apr 2025).

3.3 Diffusion LLM (DLM)-based Robust Steganography

STEAD leverages discrete DLMs to identify robust sets of positions for parallel bit embedding and error correction (repetition codes, neighborhood search). Unlike ARM-based schemes, the error propagation is strictly localized to the denoised block, yielding formal robustness guarantees under insertion, deletion, and substitution up to a provable bound (Qi et al., 21 Jan 2026). Even under mix of $10$ insertions/deletions in $512$ tokens, correct extraction remains $>80\%$.

4. Semantic, Black-box, and Adaptive Steganography

• LLM-Stega (Wu et al., 2024): Black-box LLM methods circumvent lack of logit access by using forced keyword sets and encrypted mapping. Extraction accuracy approaches $100\%$ after at most two reject-sampling passes, fluency is high ($\mathrm{PPL}\approx166$), and steganalysis remains near random guessing.
• Sentence-to-Image (S²LM) (Wu et al., 7 Nov 2025): Embeds semantically rich sentence- or paragraph-level messages in images via an LLM-driven latent embedding and neural MLP mappings, achieving remarkable capacity and PSNR/SSIM with robust recovery under JPEG and noise.
• Adaptive Cost Functions (Wu et al., 10 Dec 2025): LLM-guided program synthesis coupled with evolutionary retraining over deep steganalyzers yields new interpretable image or text cost functions for embedding, outperforming traditional expert-designed metrics in anti-steganalysis.

5. Fine-tuning-Based and Geometry-Driven Embedding

Fine-tuned LLMs can "carry" secret messages, emitting them only under trigger prompts. However, straightforward fine-tuning is vulnerable to black-box extraction via Unconditional Token Forcing (UTF). The Unconditional Token Forcing Confusion (UTFC) defense modifies the training objective to render UTF ineffective with minimal main-task performance loss (Hoscilowicz et al., 2024). Geometry-based embedding (random-hyperplane bucketing or PCA-based secrets) offers better payload and generalization, but leaves detectable linear traces in internal representations even when outputs are unrecoverable—a target for mechanistic interpretability-based detection (Westphal et al., 30 Jan 2026).

Model/Data	A+P	E+E (Δ)
Llama-8B LoRA (T)	17±2%	26±4% (+58%)
Ministral-8B LoRA(T)	24±2%	38±6% (+62%)

—Table: Geometry-based vs. TrojanStego-style exact-match recovery (Westphal et al., 30 Jan 2026).

6. Limitations, Practicalities, and Future Directions

• Edit Robustness: Most high-per-bit schemes are not robust to edits without layered error correction (via ECC/repetition, as in STEAD, S²LM, robust semantic methods).
• Tokenization Ambiguity: Stepwise verification (Yan et al., 28 Aug 2025) and disambiguating wrappers (Qi et al., 2024) are now standard for 100% reliable extraction.
• Tradeoffs: Maximum-capacity (e.g., (Norelli et al., 22 Oct 2025)) sacrifices all robustness; high-robustness methods reduce throughput by chunking or semantic bucketing.
• Semantic Robustness: Open questions include information-theoretic lower bounds under adversarial paraphrasing, optimal semantic hashings, and joint watermarking+embedding schemes.

Ongoing work explores hybrid approaches, stronger error-correcting codes for semantic channels, and mechanistic defenses based on interpretability tools (Westphal et al., 30 Jan 2026). As LLMs become more powerful and widely available, robust steganographic schemes will play a central role in both privacy-preserving communication and model forensics.