Doc-to-LoRA: Persistent Context Adaptation

• The paper demonstrates that D2L internalizes long or structured prompts into low-rank LoRA adapters using a meta-learned hypernetwork in a single forward pass.
• D2L achieves rapid adaptation and modular skill management by reducing attention computation, memory usage, and repeated prompting overhead.
• D2L's framework outperforms conventional context distillation methods with significantly lower latency and memory cost, enabling efficient document-level QA and procedural skill distillation.

Doc-to-LoRA (D2L) is a framework for persistent internalization of long or structured prompts (documents, instructions, or skill definitions) into low-rank parameter adaptations (LoRA adapters) for LLMs. By leveraging meta-learned hypernetworks, D2L generates adapters that enable a base LLM to condition its behavior on an unseen context without repeatedly consuming the original input tokens, thereby reducing attention computation and memory while facilitating rapid adaptation and modular skill management (Charakorn et al., 13 Feb 2026). Recent applications include procedural skill distillation (Zhang et al., 15 Jun 2026) and context internalization for document-level question answering (Charakorn et al., 13 Feb 2026).

1. Conceptual Motivation

D2L addresses the inference and efficiency bottlenecks arising from the quadratic complexity of Transformer attention with growing context length. In standard in-context learning (ICL), the prompt $x = [x_{\mathrm{context}}, x_{\mathrm{query}}]$ is concatenated and input to the model, with output distribution $y \sim p_\theta(y \mid x)$. This approach demands $O(N^2)$ compute and memory ($N=|x|$), with substantial GPU burden from both the attention operations and the KV-cache. Furthermore, repeatedly supplying lengthy or complex prompts such as SKILL.md files (for agent skills) or multi-page documents incurs unsustainable latency and cost.

While context distillation (CD) methods allow internalizing a context $c$ into model parameters by fine-tuning on synthetic datasets $\mathcal{D}_c = \{(x_i, y_i)\}$ with teacher-student objectives, per-context CD is impractical due to the computational cost of per-prompt fine-tuning (tens to hundreds of seconds and tens of GB RAM per context) (Charakorn et al., 13 Feb 2026). D2L eliminates this bottleneck through a meta-learned hypernetwork that produces LoRA updates in a single forward pass.

2. D2L Architecture and Meta-Training

D2L introduces a lightweight hypernetwork $H_\phi$ that, given a context $c$, produces a set of low-rank updates $\{\Delta W_l\}_{l=1}^L$ for the base model weights $\theta$, thereby forming the adapted model $y \sim p_\theta(y \mid x)$0 (Charakorn et al., 13 Feb 2026). For each layer $y \sim p_\theta(y \mid x)$1, the LoRA update has canonical form: $y \sim p_\theta(y \mid x)$2

D2L proceeds by (i) passing the context $y \sim p_\theta(y \mid x)$3 through the frozen base LLM and capturing per-layer activations $y \sim p_\theta(y \mid x)$4, (ii) using a Perceiver-style cross-attention network within $y \sim p_\theta(y \mid x)$5 to summarize $y \sim p_\theta(y \mid x)$6 into $y \sim p_\theta(y \mid x)$7 latent vectors, and (iii) projecting these to $y \sim p_\theta(y \mid x)$8, $y \sim p_\theta(y \mid x)$9 via small output heads. For long contexts, chunking and rank concatenation are used to maintain scalability: $O(N^2)$0

Meta-training uses a global context-distillation loss over a corpus of pairs $O(N^2)$1: $O(N^2)$2 During inference, a new context $O(N^2)$3 is mapped to a LoRA adapter via a single pass through $O(N^2)$4, and subsequent queries $O(N^2)$5 are answered by the base LLM with $O(N^2)$6, omitting $O(N^2)$7 from the prompt.

3. Applications and Specializations

Skill-to-LoRA (S2L)

S2L specializes D2L for procedural agent skills distributed as markdown documents (SKILL.md) (Zhang et al., 15 Jun 2026). The transformation pipeline replaces repeated context injection with skill-specific LoRA adapters, capturing the behavioral change induced by each skills document:

• Synthetic task prompts $O(N^2)$8 and demonstrations $O(N^2)$9 are generated with LLMs conditioned on $N=|x|$0 (SKILL.md for skill $N=|x|$1).
• Adapters $N=|x|$2 are trained offline to minimize autocausal LM loss on $N=|x|$3 pairs using only assistant tokens.
• At runtime, skill activation loads $N=|x|$4 onto the frozen base model, and $N=|x|$5 is omitted from the prompt.

S2L demonstrates that such adapters efficiently encode procedural behaviors: with Qwen3.6-27B, S2L outperforms Full-Skill-Text and No-Skill baselines on SWE-Skills-Bench across 21 skills, achieving a 2.9–5.2 pp pass rate gain and −4.89% per-step token cost compared to Full-Skill-Text (Zhang et al., 15 Jun 2026).

Doc-to-LoRA for In-Context Learning

On document-level question answering and needle-in-a-haystack (NIAH) tasks, D2L adapts a model to internalize long documents as adapters. With gemma-2-2b-it, D2L maintains near-perfect zero-shot accuracy out to 40K tokens—over 5× the model’s native context window—without significant accuracy drop. Memory savings are substantial: for 128K-token contexts, ICL requires >12GB KV-cache while D2L adapters occupy <50MB. In real-world QA (SQuAD, ROPES, DROP), D2L achieves $N=|x|$6 relative ROUGE-L to the ICL upper bound with <2GB update memory and $N=|x|$7s update latency (Charakorn et al., 13 Feb 2026).

4. Empirical Performance and Efficiency

D2L empirically yields dramatic reductions in both update latency and inference memory compared to both repeated ICL and per-prompt context distillation:

• Latency: D2L adapter generation takes 0.09–0.55 s (batched vs. iterative), while conventional CD requires 40–465 s; generation latency drops 20–50% with no context reprocessing.
• GPU Memory Usage: Peak update memory for D2L is 2–11GB (depending on context length and batching strategy), compared to 40–80GB for CD with generated queries.
• Inference Token Cost: S2L removes the need for repetitive context injection, reducing overhead by 4.89% over vanilla prompting (and by 6.6% versus Full-Skill-Text for skills).

The approach effectively supports rapid, persistent knowledge or skill updates, with the caveat that hypernetwork meta-training is itself resource intensive (e.g., $N=|x|$85 days on 8 H200 GPUs for adapter generation with gemma-2-2b) (Charakorn et al., 13 Feb 2026).

5. Architectural Insights and Limitations

D2L’s hypernetwork parameterization ensures persistent internalization by mapping arbitrary context $N=|x|$9 to parameter deltas, preserving information across queries. The use of LoRA adapters—lightweight, low-rank additions—enables modular routing and fast, context-specific adaptation for diverse tasks and domains. For longer documents, chunked projection and latent concatenation scale efficiently with context length.

However, the method introduces several limitations:

• Knowledge Overwrite: Applying context-derived adapters may overwrite existing internal knowledge if the context is unrelated or conflicting.
• Base Model Tiedness: Hypernetwork $c$0 must be meta-trained for each base LLM architecture; universal cross-model adapters are not supported.
• Interpretability: Unlike textual prompts or instruction-following, the LoRA adapters generated are not human-interpretable, posing debugging and auditing challenges.
• Resource Cost: Meta-training of $c$1 is computationally expensive.

A plausible implication is that hybrid strategies (adapter-aware logging, partially interpretable hybrids) may be needed in deployments requiring transparency or human-in-the-loop control (Zhang et al., 15 Jun 2026).

6. Comparative Position and Future Directions

D2L significantly advances the context distillation paradigm by enabling real-time (<1s) adapter synthesis for unseen contexts, surpassing previous prompt compression and meta-learning hypernetwork (e.g., T2L) formulations in both speed and memory footprint. Unlike D2-LoRA (Fujisawa et al., 16 Feb 2026), which addresses incremental expressivity and mergeability of LoRA parameterizations, D2L focuses on “context-to-adapter” mapping via a hypernetwork for instant knowledge injection.

Open research avenues include:

• Improving meta-training sample efficiency or universal pretraining to amortize $c$2 cost across LLM families.
• Fast online refinement by combining D2L-adapter initialization with per-context distillation.
• Extensions to continual learning, machine unlearning, and cross-modal adaptation (e.g., VLM-to-LLM internalization).
• Handling knowledge interference and composing adapters for multiple simultaneous contexts or skills (Charakorn et al., 13 Feb 2026, Zhang et al., 15 Jun 2026).

7. Summary Table: Comparative Performance

Method	Update Latency	Peak Update Memory	Context Token Limit	Relative QA Perf (2WikiMHQA)
ICL	N/A	N/A	~8,000	1.00 (upper bound)
CD (oracle)	40–100 s	7.8 GB	any	0.901
CD (gen)	465 s	59.9 GB	any	0.745
D2L (batched)	0.21 s	11.5 GB	32,000+	0.857
D2L (iter.)	0.55 s	3.8 GB	32,000+	0.844

D2L achieves near-oracle QA performance with $c$3200$c$4 lower latency and 4–20$c$5 lower peak memory than CD, and can internalize arbitrarily long contexts as persistent, parameter-efficient LoRA adapters.

---

D2L defines a paradigm for document-conditioned, persistent adaptation of LLMs—enabling scalable, memory-efficient, and modular context internalization for advanced in-context learning, skill representation, and procedural behavior distillation (Charakorn et al., 13 Feb 2026, Zhang et al., 15 Jun 2026).