ContextForge: Efficient Context Recycling

• ContextForge is a scalable architecture for LLM context management that uses a fixed-budget recyclable workspace and a five-layer memory hierarchy.
• It employs deterministic retrieval and rule-based prompt assembly to maintain accuracy while significantly reducing token usage and latency.
• Evaluated on healthcare analytics, ContextForge achieves up to 13× fewer tokens and 8× faster responses compared to traditional methods.

ContextForge is a system and architecture for running LLMs over very long conversations and very large knowledge bases without blowing up context length, latency, or cost. It treats the LLM’s context window as a recyclable workspace and wraps the model with a multi-layer memory hierarchy and deterministic retrieval pipeline. Its central objective is to maintain accuracy and consistency over long horizons while keeping active context size bounded rather than replaying an ever-growing conversation buffer, and it is evaluated on structured healthcare analytics tasks using GPT‑5.4 via the Azure OpenAI Chat Completions API (Thomas, 1 May 2026).

1. Problem formulation and systems perspective

ContextForge starts from a basic property of current LLM deployments: LLMs are stateless, so each API call has no built‑in memory of prior calls. All memory must be re-sent in the prompt, up to the model’s fixed context window, and the paper explicitly frames the resulting failure mode as a systems problem rather than as a purely modeling problem (Thomas, 1 May 2026).

The naive pattern is to send “system prompt + full conversation so far + retrieved documents + user message” at every turn. As the conversation grows, tokens per request grow roughly linearly with turns, latency grows similarly, context eventually must be truncated or summarized, and data-heavy systems repeatedly restate schemas, metrics, and prior answers that have not changed. The paper’s design premise is that the context window should be managed like OS memory—load what is needed, use it, then release it—rather than treated as a single ever-growing buffer.

This reframing leads to the notion of context recycling: the active prompt is a fixed-budget workspace assembled per query from persistent memory layers and then cleared after generation. A plausible implication is that long-horizon degradation is not only a model-capability issue but also a prompt-lifecycle issue: if the working set can be reconstructed cheaply and deterministically, then larger context windows or retraining are not the only route to long-horizon competence.

2. Five-layer memory hierarchy

ContextForge implements a five-layer memory hierarchy organized from closest to the model to farthest. The core claim is that only Layer 1 tokens change between queries; everything else is either static or external to the LLM context (Thomas, 1 May 2026).

Layer	Name	Role
Layer −1	LoRA Weights (optional)	Domain knowledge encoded as LoRA adapters
Layer 0	Residual States	Stable KV-cache prefix across the session
Layer 1	Branch Cache (Active Context)	Working set for the current query
Layer 2	Memory Index	In-memory SQLite FTS5 BM25 index
Layer 3	Knowledge Store	SQLite database on disk (WAL mode)

Layer −1: LoRA Weights (optional) is for self‑hosted models only. Domain knowledge is encoded as LoRA adapters in the model weights, giving zero tokens and no prompt cost.

Layer 0: Residual States stores a stable KV-cache prefix, such as the system prompt, top-level tree summaries, and core metrics. It is encoded once and reused as a fixed prefix, with the paper reporting ~48% memory savings compared to re-encoding the system prompt every time.

Layer 1: Branch Cache (Active Context) is the current working set: the specific branch of the knowledge tree relevant to the current turn. The paper describes it as 2–10K tokens, and it is the only layer that changes between turns.

Layer 2: Memory Index is an in-memory SQLite FTS5 BM25 index over summaries of knowledge tree nodes. It performs routing rather than storage and incurs no tokens.

Layer 3: Knowledge Store is a SQLite database on disk in WAL mode. It stores tree nodes, documents, schemas, session histories, and cached results, is described as effectively unbounded in size, and supports retrieval in <5 ms.

This layered design makes the context window an execution substrate rather than a storage substrate. Long-term knowledge remains in the database and knowledge tree, session state is compacted, and the prompt is reconstructed from durable state instead of being replayed verbatim.

3. Deterministic retrieval and structured query generation

The retrieval path in ContextForge is explicitly deterministic. The system does not use an LLM in the retrieval loop; instead, a user message and existing session state are turned into a structured retrieval query through rule-based keyword extraction and BM25 lookup over indexed summaries (Thomas, 1 May 2026).

The per-turn pipeline is described as:

1. User message arrives.
2. Keyword extraction using simple stop-word removal / pattern matching.
3. Index lookup (Layer 2) with BM25 in FTS5 to find the best matching knowledge tree branch.
4. Cache check (Layer 1) and, if needed, branch load from Layer 3 into the branch cache.
5. Context assembly from Layer 0, Layer 1, compacted conversation history, and the current user message.
6. LoRA activation (Layer −1) if applicable.
7. LLM generation.
8. Context recycling by releasing the branch tokens after generation.
9. Return response.

The retrieval query is not JSON and not a learned formal grammar; it is the deterministic transformation from text into an FTS5-compatible query. The indexed objects are knowledge tree nodes with a summary field used for routing and a content field containing the full payload, such as document chunks, table schemas, metric definitions, SQL examples, and prior QA.

The paper contrasts this with vector-based RAG. In ContextForge, search is exact lexical via BM25 rather than approximate embeddings, the organization is a tree rather than flat chunks, access control is per branch, and lookup is described as $\mathcal{O}(1)$ inverted-index lookup in practice. A standard BM25 similarity, included for context in the paper’s walkthrough, is:

$$\text{BM25}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t, d) \cdot (k_1+1)}{f(t, d) + k_1 \cdot \left(1 - b + b \cdot \frac{|d|}{\text{avgdl}}\right)}$$

although the implementation detail emphasized in ContextForge is specifically SQLite FTS5 BM25 rather than a custom scoring derivation.

A notable consequence of this design is that routing uncertainty is removed from the memory-management path. The LLM is reserved for answering and summarization; memory selection is handled by rules, indexes, and caches.

4. Context recycling and controlled synthesis

The defining mechanism of ContextForge is context recycling: the context window is treated as a fixed-budget workspace filled with what is needed for the current turn, used once, and then cleared. The paper does not define an explicit state update of the form $M_{t+1} = f(M_t, x_t, y_t)$, but its operational state consists of a fixed prefix, a chosen branch, a compacted history summary, and the latest user message (Thomas, 1 May 2026).

The paper reports the following bounded active-token regime:

Component	Token budget
Permanent context (L0)	~500 tokens
Loaded branch (L1)	2–10K tokens
Compacted history	~0–1K tokens
Total active	~3–12K tokens

The important claim is that this total active context is independent of the number of turns and the size of the knowledge store. History growth is controlled by a compaction engine that triggers summarization when the buffer exceeds ~3K tokens. The resulting summaries preserve key facts, decisions, and action items while removing filler, with reported 4–8× compression.

“Controlled synthesis” refers to deterministic prompt assembly. The LLM does not decide which branch to load or what historical content to preserve. Instead, middleware constructs a prompt from four components: system and permanent context, active branch content, compacted session history, and current user message. When the history buffer exceeds the threshold, the system calls the LLM with a summarization prompt and replaces the prior detailed history with the summary.

This means that ContextForge uses the LLM only for content generation and summarization, while the policy for what to keep, what to retrieve, and what to discard is rule-based. A plausible implication is that the system is designed to reduce failure modes caused by retrieval drift or uncontrolled conversational accretion, because prompt composition is constrained by an external memory policy.

5. Evaluation, scaling behavior, and empirical results

ContextForge is evaluated on structured healthcare analytics built on CMS Medicare Provider Utilization and Payment Data, described as 276M fact rows, 5 tables, deployed in Microsoft Fabric. Both ContextForge and the baseline use the same model and endpoint: GPT‑5.4 via Azure OpenAI Chat Completions API. The baseline is an Azure AI Foundry agent using the Fabric Data Agent tool, representing the standard single-thread conversational pattern (Thomas, 1 May 2026).

Two benchmarks are reported.

In the 12-turn benchmark, progressive complexity ranges from simple lookups to multi-step analytics, and each turn is scored 0–10 for correctness, completeness, and consistency with verified data.

In the 15-turn benchmark (two cycles), the emphasis is on multi-turn reasoning, back-references to earlier turns, domain shifts, and structured healthcare queries, with 300 max score over 30 turns overall across two cycles.

Benchmark	ContextForge	Baseline
12-turn score	85 / 120	84 / 120
12-turn tokens	31K	132K
12-turn latency	9.7 s	45.7 s
15-turn score	225 / 300	194 / 300
15-turn tokens	25,666	345,112
15-turn latency	7.6 s	60.5 s

On the 12-turn benchmark, the two systems are essentially similar in accuracy—85 vs. 84 out of 120—but ContextForge uses 31K vs. 132K tokens and 9.7 s vs. 45.7 s average response time, corresponding to 4.2× fewer tokens and 4.7× faster responses.

On the 15-turn benchmark, ContextForge scores 225 vs. 194 out of 300 (75.0% vs. 64.7%), uses 25,666 vs. 345,112 tokens (13.4× fewer), and has 7.6 s vs. 60.5 s latency (8.0× faster). Turn success is 28 / 30 for ContextForge and 27 / 30 for the baseline. The cycle breakdown is 113/150 and 112/150 for ContextForge, versus 96/150 and 98/150 for the baseline.

The paper highlights the scaling pattern directly: in the baseline, tokens per turn climb to ~22K by turn 15; in ContextForge they stay around ~850 per turn. It also reports that Azure content safety triggered on 2/30 turns for both systems, identified as false positives on drug-prescription-related turns, and that there was one additional Fabric infrastructure error in baseline; none in ContextForge.

A further systems result concerns knowledge-base size. For 10M–100B tokens knowledge base, the reported response-time components are Index lookup: <1 ms, Branch load: 0 (cached) or 2–4 s (cold), and Generation: ~0.7–2 s. This is presented as largely independent of database size because of the in-memory FTS index, branch-wise loading, and bounded tree depth.

6. Positioning, strengths, limitations, and future directions

ContextForge sits at the intersection of conversation summarization, retrieval-augmented generation, long-context models, memory-augmented models, cache-augmented generation, and hierarchical agent memory systems, but it targets a distinct layer of the stack: the context-window lifecycle itself (Thomas, 1 May 2026).

Relative to RAG, it is complementary rather than substitutive: RAG retrieves documents per query, while ContextForge adds hierarchical organization and deterministic context assembly across turns. Relative to conversation summarization, summarization is only one component; the more important distinction is that long-term knowledge is not carried in context at all. Relative to long-context models, ContextForge is independent of window size and addresses what should be fed into whatever window exists. Relative to memory-augmented models, its memory management is deterministic and index-driven rather than delegated to the LLM. Relative to cache-augmented generation, the Layer 1 branch cache plays a similar role but within a broader hierarchy that includes proactive loading and predictive prefetch. The paper summarizes its novelty as treating the context window as a recyclable working set, implementing a five-layer hierarchy, and obtaining substantial efficiency gains without changing the model.

The system’s principal strengths are bounded active context, significant efficiency gains, model-agnostic deployment, deterministic retrieval, and compatibility with existing RAG-style components and tools. The paper states that it works with any OpenAI-compatible LLM API, including GPT models and local servers, and does not require model architecture changes or retraining. It also reports several memory optimizations: LoRA injection: 0 tokens, Residual states: 48% savings via system prompt KV reuse, TQ3 quantization: 6× less KV memory (3-bit), Context compaction: 4–8× token reduction, and Prefix cache: near-100% savings on repeated system prompts.

Its limitations are also explicit. The benchmark scope is limited to one domain (Medicare data analytics) and a small number of runs. There are no broad cross-domain benchmarks such as legal or code. Claims about 100B-token trees are described as analytically motivated rather than validated at that extreme scale in production. The optional LoRA path was evaluated on internal medical QA sets and needs external validation on public benchmarks. The paper also notes that nightly caching and precomputation show promising cache hit rates (20%→90% over weeks) but are based on limited deployments, and that other domains may require different tree designs and indexing schemes.

The paper’s stated future directions are independent validation of training-free LoRA on public benchmarks, large-scale deployment with many concurrent users, hybrid retrieval: BM25 + learned sparse or dense representations, and extending the hierarchy to multimodal content such as images, code, and mixed structured/unstructured data. This suggests that the core abstraction is not tied to healthcare analytics. A plausible implication is that ContextForge is best understood as middleware for bounded-context inference, with the knowledge tree, cache hierarchy, and deterministic retrieval policy as its transferable primitives.