Cryptographic Steganography via LLM-CAT

• The paper introduces the LLM-CAT paradigm, achieving provable secure steganography with near-entropy embedding rates using adaptive dynamic sampling.
• It leverages conditional arithmetic coding and pseudorandom shifts to ensure stegotexts remain statistically indistinguishable from natural LLM outputs.
• Robust encoding strategies, including Markov decision policies and constrained optimization, provide resilience against up to 50% adversarial paraphrasing or token edits.

Cryptographic steganography via the LLM-CAT (“LLM Conditional Arithmetic-coding Text”) paradigm refers to a family of provably secure methods for embedding secret information within natural language text generated by LLMs. These schemes leverage the existence of powerful autoregressive generative models as samplable oracles and exploit either their conditional next-token distributions or the internal parameters of a fine-tuned model, yielding covert channels whose stego-output is computationally or information-theoretically indistinguishable from natural LLM text. Key technical innovations include black-box adaptive dynamic sampling, information-theoretic channel optimization, Markov decision theoretic policy control, and robust encoding under adversarial paraphrasing or local edit attacks. LLM-CAT unifies prior arithmetic coding, rejection sampling, and model perturbation approaches, enabling both stegotext generation and robust message extraction under a broad spectrum of threat models (Pang, 17 Apr 2025, Huang et al., 3 Feb 2025, Perry et al., 11 Apr 2025, Bai et al., 1 Jan 2025).

1. Formal Foundations and Security Models

Stegosystem formalisms in LLM-CAT instantiate a triple $\mathsf{II} = ({\rm KeyGen}, {\rm Enc}, {\rm Dec})$ over samplable covertext channels $\mathcal{C}$, where the distribution over the next token given history—$P_{\rm model}(w|H)$—is accessible via black-box sampling or white-box probability vectors. The sender and receiver share a cryptographic secret key $K$ (for pseudo-random generator synchronization, keyed hashing, or arithmetic code initialization). Security is defined in the chosen-hiddentext indistinguishability sense: the advantage of any PPT adversary in distinguishing $(Enc_K(m;H))$ from direct $Samp_\mathcal{C}$ sampling is negligible in the security parameter.

Extensions consider robust variants. Weak robustness (resistance to local k-gram edits) and strong robustness (resistance to semantic-preserving paraphrasing) are formalized via adversarial games. Strong undetectability pertains to the statistical distance (or f-divergence) between the induced distribution over stegotexts and the genuine model. Modern LLM-CAT approaches rigorously quantify total variation or KL divergence budgets, maximizing embedding bits subject to explicit statistical indistinguishability constraints (Pang, 17 Apr 2025, Perry et al., 11 Apr 2025).

2. Dynamic Sampling and Arithmetic Coding Embedding

LLM-CAT systems employ arithmetic coding and dynamic next-token distribution manipulation:

• Adaptive Dynamic Sampling: At each token position, both encoder and decoder sample $n$ i.i.d. tokens from the model using deterministic randomness (PRG seeded by $K$), mapping secret message bits to the index of the chosen sample. A collision set tracks message candidates consistent with token choices, reducing until message recovery is unique. Asymptotically, for large $n$, embedding capacity approaches the channel entropy $H(\mathcal{C})$ (Pang, 17 Apr 2025).
• Pseudorandom Shift & Conditional Arithmetic Coding: LLM output sampling $u \sim [0,1)$ (for next-token selection via cumulative mass) is replaced by $u = r + \bar m \mod 1$—a cryptographically pseudorandom shift of the binary-expanded secret. This preserves the output law, yielding zero-KL-gap security. Reordering techniques further reduce capacity loss from interval partitioning phenomena (Bai et al., 1 Jan 2025).
• Constrained Distribution Optimization: Embedding rate is maximized by solving constrained optimization at each time step:

$$\max_{q \in \Delta_N} H(q)\quad \text{subject to}\quad D_{\rm KL}(q\|p)\leq \varepsilon,$$

yielding a closed-form, soft-truncation Gibbs distribution for the replacement sampling law (Huang et al., 2024).

These methodologies either treat the LLM as a black-box sampler—asking for sampled outputs only—or as a white-box probability vector provider, depending on access granularity.

3. Practical Embedding, Extraction, and Robustness

LLM-CAT embedding schemes adopt several algorithmic strategies:

• Embedding bitstreams via binary expansion and mapping bits to sample indices or arithmetic code intervals.
• Extraction proceeds via running the same deterministic sampling/decoding process, with the received stegotext allowing recovery of message bits whenever the collision set resolves uniquely.
• Black-box techniques require only public sampling APIs and avoid internal LLM distribution access, broadening deployability to API-served models.
• Robust constructions (e.g., locally-sensitive hash, LSH, keyed encoding on embeddings) can recover messages even after targeted paraphrasing or local token edits, with empirical robustness up to 50% reordering or paraphrase rates under PCA-aligned LSH (Perry et al., 11 Apr 2025).

Defense against extraction attacks, such as Unconditional Token Forcing (UTF)—a method exploiting anomalous output patterns under forced model continuation—relies on fine-tuning with confusion datasets (UTFC), protecting the embedded channel by promoting high-entropy, benign outputs under adversarial probing (Hoscilowicz et al., 2024).

4. Channel Capacity, Embedding Efficiency, and Evaluation

Embedding rate is determined by the entropy $H(P)$ of the cover channel, subject to divergence or robustness constraints:

• Adaptive dynamic sampling and arithmetic coding approaches asymptotically attain channel entropy; for realistic $n\approx 30$, achieved rates reach 90–98% of $H$.
• Empirical studies show black-box adaptive sampling (LLM-CAT) matches or surpasses white-box approaches (e.g., METEOR, DISCOP) in both efficiency and stego-quality, with per-token overheads as low as 1–3 ms and perplexity/detectability essentially unchanged compared to natural sampling (Pang, 17 Apr 2025, Bai et al., 1 Jan 2025).
• Table: Comparative Per-Token Results (QA, |V| ≈ 50k, H ≈ 4.8)

Method	Embed (bits)	Util. (%)	PPL	F1	Time (ms)
METEOR	0.57	12	4.86	0.85	3.9
DISCOP	0.42	8.8	4.84	0.84	1.3
LLM-CAT (ours)	0.90	19	4.91	0.84	1.6

This demonstrates LLM-CAT achieves higher embedding rate and utilization than prior baselines with comparable computational cost (Pang, 17 Apr 2025).

5. Security Analysis: Indistinguishability and Adversarial Resistance

Security proofs for LLM-CAT are typically game-based, reducing the stego-output distribution to true cover sampling under suitable cryptographic assumptions (secure PRG, i.i.d. sampling):

• In adaptive sampling and pseudorandom-shift schemes, per-token output is information-theoretically indistinguishable from natural sampling due to the uniformity of the mapped secret bits (arising from the PRG).
• Capacity-optimized schemes using total variation/KL constraints provably bound eavesdropper advantage to the divergence budget, and CMDP-based constructions provide a mathematically exact policy for maximizing rate under this budget (Huang et al., 3 Feb 2025).
• Robust formulations establish upper bounds on message loss under both local edit and semantic-preserving paraphrase attacks, controlled by the collision tolerance of the hash or LSH under the specified metric (Perry et al., 11 Apr 2025).

Empirical attack evaluation confirms that robust embedding (LSH) surpasses watermarking for semantic resilience; watermark-based schemes rapidly degrade above 5% paraphrastic change, while LSH maintains >90% recovery up to 50% perturbation.

6. Limitations, Extensions, and Open Problems

While LLM-CAT achieves state-of-the-art in cryptographic steganography, several technical constraints and directions for future research remain:

• All methods are subject to the entropy bottleneck: deterministic or highly predictable LLM outputs offer limited embedding.
• Tokenizer mismatches (e.g., in subword splits) create decoding ambiguity; practical variants employ brute-force trial blocks or error-correcting codes to counteract failures (Huang et al., 2024).
• CMDP-based abstractions require state aggregation or context clustering in real LLMs; robust protocol design for high-dimensional output spaces is an open technical challenge (Huang et al., 3 Feb 2025).
• Extreme compression or paraphrastic adversaries outside the robustness metric can still stymie extraction; this motivates richer semantic encoding and hybrid error-correcting methods (Perry et al., 11 Apr 2025).
• Open questions persist regarding provable guarantees under deep learning model transformations, scalability of robust chunk alignment, and minimization of latency-bandwidth tradeoffs.

7. Comparative Methods and Practical Trade-Offs

LLM-CAT represents a spectrum of methods, each with distinct trade-offs in access requirements, embedding efficiency, security assurance, and robustness. The table below summarizes salient properties:

Method	Model Access	Capacity	Robustness	Security Guarantee	Reference
Adaptive Dynamic Sample	Black-box	near-entropy	No	Comp. Indistinguishability	(Pang, 17 Apr 2025)
CMDP/TV-Constrained	Policy (black/white)	controlled	No	TV-constrained	(Huang et al., 3 Feb 2025)
OD-Stega (KL-opt.)	White-box	controlled	No	KL-constrained	(Huang et al., 2024)
ShiMer (shift-merge)	Black-box	near-entropy	No	Zero-KL-gap	(Bai et al., 1 Jan 2025)
Watermarking/LSH	White-box	domain-bound	Yes	Weak/strong robustness	(Perry et al., 11 Apr 2025)
Fine-tuned LLM/UTFC	Model params	high	Triggered, weak	Only w.r.t. trigger/UTF	(Hoscilowicz et al., 2024)

A plausible implication is that for applications requiring only message hiding with minimal model assumptions and no need for robustness, black-box arithmetic coding or dynamic sampling achieves optimal trade-off between security and throughput. For operational environments with likelihood of paraphrasing or lossy transformation, embedding-robust (LSH-hash) designs provide stronger guarantees at reduced channel capacity.

---

References:

• "Provable Secure Steganography Based on Adaptive Dynamic Sampling" (Pang, 17 Apr 2025)
• "Relatively-Secure LLM-Based Steganography via Constrained Markov Decision Processes" (Huang et al., 3 Feb 2025)
• "OD-Stega: LLM-Based Near-Imperceptible Steganography via Optimized Distributions" (Huang et al., 2024)
• "Shifting-Merging: Secure, High-Capacity and Efficient Steganography via LLMs" (Bai et al., 1 Jan 2025)
• "Robust Steganography from LLMs" (Perry et al., 11 Apr 2025)
• "LLMs as Carriers of Hidden Messages" (Hoscilowicz et al., 2024)