Anchored Sliding Window: Toward Robust and Imperceptible Linguistic Steganography

Abstract: Linguistic steganography based on LLMs typically assumes that steganographic texts are transmitted without alteration, making them fragile to even minor modifications. While previous work mitigates this fragility by limiting the context window, it significantly compromises text quality. In this paper, we propose the anchored sliding window (ASW) framework to improve imperceptibility and robustness. In addition to the latest tokens, the prompt and a bridge context are anchored within the context window, encouraging the model to compensate for the excluded tokens. We formulate the optimization of the bridge context as a variant of prompt distillation, which we further extend using self-distillation strategies. Experiments show that our ASW significantly and consistently outperforms the baseline method in text quality, imperceptibility, and robustness across diverse settings. The code is available at github.com/ryehr/ASW_steganography.

• The paper presents the Anchored Sliding Window approach, which decomposes text context into prompt, bridge, and token segments to improve resistance to adversarial modifications.
• It leverages hard and soft bridge contexts optimized via self-distillation to minimize KL divergence and preserve natural text quality.
• Empirical results across various LLMs demonstrate significant gains in imperceptibility, fluency, and robustness compared to previous steganographic methods.

Anchored Sliding Window: Robust and Imperceptible Linguistic Steganography

Motivation and Problem Scope

Autoregressive LLM-based linguistic steganography provides a mechanism for covert communication by embedding information within generated text. A longstanding practical limitation is fragility: existing methods (including classic full-context and recent robustness-enhanced frameworks like WinStega) fail under active attacks, such as a single-token modification, deletion, or insertion. This is due to the strict left-to-right dependency inherent to text generation—the entire token history typically influences the next-token distribution. This results in high brittleness as adversarial edits propagate forward, corrupting message extraction.

Figure 1: Steganographic extraction is fragile to active attack; even a single token alteration in the stegotext disrupts all subsequent autoregressive inferences.

Classic robustness-centric mitigations (e.g., constraining the context window to recent tokens as in WinStega) guarantee that only the latest $w$ tokens influence generation, thereby localizing the effect of minor tampering. However, such truncation systematically sacrifices text quality and statistical imperceptibility, given the loss of full prompt and semantic history required for high-quality LLM outputs.

Anchored Sliding Window Framework

To address the imperceptibility–robustness trade-off, the paper introduces the Anchored Sliding Window (ASW) framework. The ASW context window decomposes into three segments: (1) the original shared prompt, (2) a bridge context (either hard or soft), and (3) the latest $w$ tokens. The bridge context is expressly anchored after the prompt and serves to fill the gap left by discarded tokens, remaining independent of the excluded segment's content and position. This architectural separation is essential to avoid robustness degradation due to bridge–history entanglement.

Figure 2: Conventional robustness-unaware context window, using the entire previous history.

Figure 3: In ASW, the bridge context replaces excluded tokens and connects the prompt to the latest tokens, as shown with a hard bridge context.

ASW thus generalizes previous truncation techniques by preserving prompt information while interpolating the missing context, yielding substantial gains in both robustness and imperceptibility.

Hard and Soft Bridge Contexts: Parameterization and Optimization

The hard bridge context substitutes for truncated tokens using discrete hand-crafted text segments (e.g., “[CONTEXT TRUNCATED]”). Empirical analysis demonstrates that the choice of bridge—especially context-aware phrasing over random or empty tokens—reduces the KL divergence between distributions conditioned on full and truncated contexts, boosting imperceptibility without compromising robustness.

To further improve adaptability, the bridge context is parameterized as soft tokens at the embedding level. Training is formulated as a self-distillation objective, where gradients update only the soft bridge vectors to minimize the forward (or reverse) KL divergence between logits output from full context and ASW-restricted context. This is a frozen-model prompt optimization process; only the bridge embeddings are tunable.

Figure 4: Overview of the distillation framework; only the soft bridge context parameters are tunable, with the distillation objective reducing the gap between logits from the full context (teacher) and the ASW window (student).

Empirical Evaluation and Results

Across Qwen2.5 and Llama-3 model families, ASW consistently outperforms WinStega on multiple axes:

• Imperceptibility: Steganalysis accuracy is reduced by up to 21.99% (0.955 → 0.745), indicating closer alignment with natural text statistics.
• Text Quality: Improvements in per-token perplexity (ΔPPL↓ by 99.36%), BLEU (0.082 → 0.276, +236%), ROUGE-L (+104%), and BERTScore (+18%) with soft bridge and Self-Distillation.
• Human Evaluation: ASW stegotext achieves a fluency score of 2.56 vs. 1.88 for WinStega (scale 1–5), closing the gap with full context baselines.
• Robustness: For adversarial experiments with $m=1,2,3$ token modifications, the proportion of unaffected inference positions is always higher for ASW than for WinStega, especially as $w$ increases. Theoretical analysis shows the expected number of impacted positions grows monotonically with $w$, but ASW consistently mitigates this deterioration more effectively than prior approaches.

  Figure 5: Metric averages under varying $w$ for WinStega and ASW, with multiple bridge context strategies.

The data conclusively supports that the soft bridge, particularly when distilled with forward KL, achieves a favorable trade-off: minimal divergence from full-context distribution (improving imperceptibility and fluency) and resistance to active token-level attacks.

Analysis Under Alternate Architectures and Datasets

The superiority of ASW is robust across different backbone LLMs, including large-scale Qwen2.5 and Llama-3, and different prompt or QA datasets (InstructionWild, Dolly-15k, Super-NaturalInstructions). The benefits are amplified with larger models, as substantial prompt retention and context interpolation become more effective due to deeper and broader model representations.

Figure 6: Relative gains of ASW over WinStega across LLM scales; improvements intensify with larger model capacity.

Enhanced Training Strategies and Efficiency

Further experiments explore LoRA-based adaptation, where LoRA modules, trained in addition to the soft bridge context, provide marginal additional improvements at the cost of vastly increased trainable parameters (28k versus >1M). The incremental benefit does not outweigh the efficiency and simplicity of the soft bridge distillation alone for real-world deployment scenarios.

Figure 7: LoRA-based distillation pipeline; both the soft bridge and LoRA modules are tunable, with the same self-distillation objective.

Practical Considerations: Tokenization and Hardware Robustness

The paper also addresses:

• Tokenization Inconsistency: Problems where Alice's detokenized output cannot always be exactly retokenized by Bob, leading to desynchronization and extraction failure. ASW compensates for this via its modular context window, though architectural and tokenizer harmonization remains vital.
• Hardware Indeterminism: Even with identical context, actual LLM outputs may vary across devices or configurations, posing additional practical challenges for robust steganographic systems.

Theoretical Perspective: Forward vs Reverse KL

The choice of forward KL for self-distillation is not arbitrary. Forward KL is mass-covering, preserving the full candidate space for embedding, thereby maximizing embedding capacity without collapsing to narrow modes (as reverse KL typically does). This property is critical for maximizing steganographic payload while maintaining natural text distributional properties.

Conclusion

The Anchored Sliding Window paradigm advances the state of the art in LM-based linguistic steganography by systematically addressing the robustness–imperceptibility dichotomy. By decomposing context into prompt, bridge, and latest tokens, and by optimizing bridge content through self-distillation, ASW achieves higher text quality, better anti-detection performance, and enhanced resistance to adversarial perturbation than previous sliding or truncation frameworks. The design operates efficiently, with lightweight trainable parameters, and generalizes well across LLM architectures and domains.

The ASW strategy offers a template for future work in both linguistic steganography and related robust-inference constrained text-generation problems, especially as models and downstream adversarial environments continue to evolve. Further effort is warranted for tackling challenges around tokenizer inconsistency, hardware-level nondeterminism, and adaptive adversarial attacks in practical, decentralized, or cross-platform deployments.

---

Reference: "Anchored Sliding Window: Toward Robust and Imperceptible Linguistic Steganography" (2604.09066)