Pichay: LLM Context Demand Paging

• Pichay is a demand-paging system that recasts LLM context windows as an L1 cache, optimizing memory with eviction and fault-driven pinning.
• It uses a transparent HTTP proxy to interpose on API calls, reducing structural waste by managing stale and redundant content.
• The system demonstrates significant token savings and improved session efficiency, addressing challenges like thrashing and working-set misestimation.

Searching arXiv for the specified paper to ground the article in the current record. Pichay is a demand-paging system for LLM context windows that recasts the context window not as memory in itself, but as the top, scarce level of a larger memory hierarchy—effectively an L1 cache. Introduced in "The Missing Memory Hierarchy: Demand Paging for LLM Context Windows" (Mason, 9 Mar 2026), it is implemented as a transparent proxy between an agentic client and an inference API, where it can evict stale content, leave compact retrieval handles in place of removed material, detect page faults when the model later re-requests evicted content, and adapt eviction behavior through fault-driven pinning. The paper’s central claim is that context exhaustion, attention dilution, long-session cost growth, and loss of state across sessions are not merely prompt-design problems but memory-management problems, and that concepts from virtual memory—demand paging, working sets, fault-driven replacement, pinning, thrashing, and multi-level hierarchies—map structurally to LLM session management (Mason, 9 Mar 2026).

1. Problem formulation and theoretical framing

Pichay begins from the observation that contemporary agent systems reconstruct the full prompt on every API call: system prompt, tool schemas, full message history, and stale tool outputs. Because nothing is evicted and nothing is demand-loaded, long sessions resemble pre-virtual-memory overlay systems that repeatedly cram all state into a fixed address space. The paper argues that this pathology is structural rather than stylistic: agent frameworks repeatedly resend static and stale content on every call, and transformers repeatedly pay for those tokens in attention (Mason, 9 Mar 2026).

The paper quantifies this as "structural waste." Across 857 production sessions, 54,170 API calls, and 4.45 billion effective input tokens, 21.8% of all input tokens were structural waste. The waste taxonomy combines unused tool definitions at 11.0%, duplicated content or skill deduplication at 2.2%, and stale tool results or static resend categories, including 8.7% static re-send in the corpus projection. In the broader corpus analysis, 79.4% of conversation bytes were tool results, and Read alone accounted for 75% of all tool output bytes. The average API call processed 82,061 effective input tokens and generated only 88 output tokens, yielding a 933:1 input/output ratio. These measurements are presented as evidence that the dominant cost is not only large prompts once, but persistent residency of cold material over many turns (Mason, 9 Mar 2026).

To formalize repeated reprocessing, the paper defines amplification for tool results as

$$A = \frac{\sum_{r \in R} \text{size}(r) \times \text{turns\_survived}(r)} {\sum_{r \in R} \text{size}(r)}$$

where $R$ is the set of tool results. For main sessions, median amplification was $84.4\times$, with P75 $217.9\times$ and P90 $570.8\times$. In effect, a byte that remains resident is not paid for once, but repeatedly. The paper treats this residency-versus-fault tradeoff as the direct analogue of the problem virtual memory was designed to solve.

A common misconception addressed explicitly is that the remedy is simply to "make prompts smaller." The paper rejects that framing. It characterizes Pichay not as prompt engineering and not as another prompt compressor, but as a proxy-level memory manager for agentic LLM sessions. This suggests that the salient abstraction is a managed resident set rather than a monolithic prompt.

2. Proxy architecture and page abstraction

Pichay is deployed as a transparent HTTP proxy between the client and the inference provider. The client continues to build the full request in the usual way, but the proxy receives the JSON request at the Messages API layer, inspects the message array, tool definitions, and system prompt, optionally rewrites them, forwards the modified request to the model API, and returns the provider’s response unchanged (Mason, 9 Mar 2026). This placement gives the system "MMU"-like behavior in the paper’s analogy: it can interpose on every request without changing the client, changing the model, or requiring provider-side API modifications.

The client remains the backing store because it retains the full conversation history and original tool outputs. The proxy manages only what is resident in the current request. That distinction is operationally important: Pichay does not become the authoritative store of conversation state, and its checkpointing concerns metadata rather than the full corpus.

The unit of management is not a fixed-size block such as a 4 KB hardware page. Instead, a "page" is an addressable tool-produced object, especially outputs such as file reads. The paper sharply distinguishes paging from garbage collection. Ephemeral outputs such as Bash results, search outputs, and directory listings are garbage collectable: once removed, they are not meaningfully re-requestable. Addressable content such as Read results, plan documents, or specifications are pageable because they have stable identity, typically a tool name plus arguments such as a file path. Only pageable evictions can generate page faults; including non-pageable garbage in the denominator would artificially depress the fault rate.

When content is evicted, the original block is replaced by a compact tombstone-like retrieval handle. The paper gives the example

$$\texttt{[Paged out: Read /path/to/file.py (12,450 bytes, 287 lines). Re-read if needed.]}$$

The handle is about 200 bytes regardless of original size. It functions simultaneously as identifier, recovery hint, and semantic anchor. The paper stresses that these handles are not merely summaries; they behave like late-binding page-table entries that preserve enough metadata for the model to understand what disappeared and how to restore it.

3. Eviction policy, page faults, and working-set identification

The baseline eviction policy is intentionally simple: FIFO by user-turn age. Tool results older than $\tau$ user turns and larger than $s_{\min}$ bytes become eviction candidates, with all experiments using $\tau = 4$ and $s_{\min} = 500$. Error results are never evicted (Mason, 9 Mar 2026). The paper presents this baseline not as a fully optimized replacement strategy but as a deliberately simple policy against which the utility of paging and fault observability can be assessed.

Page faults are detected when the model later issues a tool call matching an evicted entry, defined by the same tool name and arguments. For Read operations, this means the same path. Internally, the proxy indexes evicted entries by a key derived from tool name and input arguments. If the model reissues such a call, the proxy records a page fault: the model needed content that had previously been resident, was evicted, and had to be fetched again. One of the paper’s methodological points is that faults are observable, measurable events, unlike many prompt-pruning approaches where degradation is inferred only indirectly.

The main policy innovation is fault-driven pinning, described as an adaptation of working-set identification. The algorithm is procedural. On eviction of addressable content, the system records the file path and content hash. On page fault, it records the evicted content’s hash in a fault history table. On the next eviction attempt for the same path, if the current content hash matches the fault-history entry, the system pins the page and skips eviction permanently for the session. If the pinned file is read again with different content, it is unpinned because the earlier version was stale and the new version begins a fresh fault cycle.

This is explicitly Denning-like in motivation. A page that faults after eviction is treated as probable working-set material, and the system learns from that mistake. The authors summarize the deployed rule as "one fault per file, permanently pinned for the session," while also noting that this is likely too strong. They propose future "pin decay," under which pin strength would halve every $R$0 turns since last access. A plausible implication is that Pichay’s current working-set estimator is conservative after a single demonstrated miss but does not yet model working-set drift in long sessions.

4. Memory hierarchy and model-cooperative mechanisms

The paper proposes a four-level memory hierarchy plus persistent storage. Only the first three levels are deployed in some form, and only the first two are evaluated. The structure is as follows (Mason, 9 Mar 2026):

Level	Role	Status in paper
L1	Active generation window	Deployed and evaluated
L2	Working set via demand paging and pinning	Deployed and evaluated
L3	Session history managed by compaction	Deployed, not evaluated at scale
L4	Persistent cross-session memory	Mostly design
Storage	Full archived corpus, retrievable but not resident by default	Conceptual backing layer

L1 is the current context window attended to on an API call. L2 is the working set: pages identified as actively needed but managed through demand paging and pinning rather than left in L1 forever. In deployed terms, L2 is the set of faulted-and-pinned content the proxy attempts to preserve against further eviction. L3 is session history managed through compaction rather than exact residency; older dialogue, plans, and completed interactions are compressed into lossy but declared summaries. L4 is persistent cross-session memory, including authored compressions, semantic indices, activity records, and graph- or retrieval-addressable memory that survives session death. Beneath all levels lies persistent storage, consisting of the full corpus—transcripts, files, tool outputs, and documents—archived and retrievable but not resident by default.

Pichay extends the hardware analogy by allowing cooperation from the "processor," here the LLM itself. Two mechanisms are used. The first is phantom tools, injected by the proxy but never seen by the actual application framework because the proxy intercepts them. The paper defines memory_release(paths), allowing the model to declare that it no longer needs some files, and memory_fault(paths), allowing direct restoration of evicted content from the proxy’s eviction cache without a real filesystem read. The second mechanism is cleanup tags embedded in model output and parsed by the proxy before forwarding: drop: block:ID, summarize: block:ID "text", anchor: block:ID, and collapse: turns N-M "text".

The collapse operation underlies L3 compaction by replacing a run of turns with a single synthetic summary block. The summary is intentionally lossy, and only metadata—not original content—is checkpointed by the proxy. State persists across restarts through atomic metadata checkpointing, while the client’s message array remains the true backing store.

Intervention is governed by graduated pressure zones based on token count. Under 60K tokens is "Normal," where the system only observes. Between 60K and 100K is "Advisory," where the proxy informs the model about current fill, the largest resident blocks, and available cleanup operations. Between 100K and 120K is "Involuntary," where automatic FIFO eviction begins. At 120K and above is "Aggressive," where thresholds are relaxed for emergency survival. The paper’s point is that memory management is not binary; pressure can rise gradually and can induce either cooperation or forced reclamation.

5. Empirical results and operating characteristics

The empirical contribution is organized around corpus analysis, offline replay, live task comparison, and deployment case studies (Mason, 9 Mar 2026). The production corpus covered about four months, 15 software projects, 857 sessions, 54,170 API calls, and 4.45 billion effective input tokens. Session types included 59 main sessions, 567 subagent sessions, 154 compact sessions, and 21 prompt-suggestion sessions.

A smaller proxy-captured decomposition used 5 sessions, 99 API calls, and 24.4 MB of request data. In that sample, total addressable waste was 60.5% of request bytes, decomposed into 26.5% dead tool output, 20.2% unused tool definition stubs, 11.0% static re-send, and 2.9% skill triplication. Using a measured conversion constant of 4.15 bytes per effective input token from 139 proxy-captured API calls, the corpus-scale projection estimated 970.4 million addressable wasted tokens, or 21.8% of all input tokens. Average savings were 17,913 tokens per API call, equivalent to 85 billion fewer token-token attention pairs across the corpus.

Offline replay was used to test eviction safety without issuing real API calls. Across 29 proxy-captured sessions, the replay simulated 1,393,000 evictions, detected 354 page faults, and reported a fault rate of 0.0254%, with 8.49 GB of content evicted. The paper interprets this as evidence that a simple age-based policy is sound for stale pageable content in the studied workload, while also noting that the result depends on the workload and on the distinction between pageable and non-pageable outputs.

The live treatment comparison used baseline, trimmed, and compact+trim conditions on a standardized task. Effective input tokens fell from 114,222 in baseline to 88,421 with trimming and 71,816 with compact+trim, corresponding to reductions of 22.6% and 37.1%. Cache-read tokens fell from 79,712 to 38,639 and then 32,228. The task completed correctly in all three conditions.

The paper also reports a quality check on 18 sessions in which paired baseline and tombstoned contexts were continued using the same next user message, then evaluated by three LLM judges on correctness, completeness, and coherence. Treatment mean compression was 48%. Judges preferred treatment 37% of the time, baseline 28% of the time, and recorded 35% ties. Completeness was identical at 3.59, and correctness and coherence differed by at most 0.15 on a 5-point scale. Detection of which output had reduced context was 57%, not significantly above chance, with $R$1. These results suggest that substantial context reduction can be achieved without broad quality collapse, but not without failures.

6. Failure modes, limitations, and broader significance

The paper’s case studies make the failure modes concrete (Mason, 9 Mar 2026). In "Session A," a steady-state coding session with compact mode enabled, context free space increased from 7% before compaction to 43% after compaction. There were 15 total evictions: 11 garbage-collected ephemeral results and 4 pageable Read evictions. One of those four caused a fault, giving a 25% fault rate on Read-only evictions. The faulted object was a plan file read early in the session and needed throughout, which the paper interprets as a classic working-set miss: FIFO tracks age rather than reuse.

In "Session B," a 681-turn sustained, multi-agent coordination session, the system evicted almost continuously. There were 680 total evictions, of which 74 were garbage collection and 606 were pageable Read evictions, alongside 659 page faults. The reported fault rate against total evictions was 97% $R$2, and the session exhibited severe thrashing. Compression peaked at 5,038 KB down to 339 KB, although the paper notes that this figure is partly artifactual because re-reads caused by prior evictions inflated the "before" size. The session ended because of API rate limiting rather than context exhaustion, demonstrating that faults consume not only compute but also billable round trips and rate budget.

This is where the working-set analogy becomes operational rather than metaphorical. The paper states that the session’s working set exceeded available resident memory, so the system spent most of its effort paging. Repeated cycles of evicting and faulting a few files, especially during planning phases that touched a broader simultaneous set of repositories and files than the age threshold could retain, are presented as the LLM analogue of classical VM thrashing.

The paper also presents a cost model and argues that LLM paging inverts a classical virtual-memory assumption. In traditional VM, keeping a page resident is effectively free while faults are expensive, so the optimization target is minimizing page faults. Here, every token kept in context costs money and attention on every turn, while faulting it back in may cost only a single reread. The paper writes the objective as

$R$3

with explanatory text stating that

$R$4

is the cumulative cost of keeping page $R$5 resident for $R$6 turns, and

$R$7

is the one-time cost of restoring it. The paper then gives the break-even condition

$R$8

which simplifies to evict whenever the page will not be referenced for more than one turn. The paper notes that the LaTeX is visibly malformed in places, but states that the intended meaning is clear. It further argues that the true fault cost is not actually linear in page size because a fault induces another full inference pass whose cost is roughly quadratic in current context size $R$9. This suggests that eviction should become more conservative under extreme pressure even though aggressive eviction is often rational at moderate fill levels.

Several implementation limitations are identified. Pichay assumes the client retains full conversation history, so the proxy can rely on the client-side transcript as backing store. It also assumes stable tool identities for pageable objects, especially file paths for Read calls. Because the current implementation uses a single PageStore for all connections, subagent sessions can contaminate one another’s eviction state; the paper identifies per-connection isolation as future work. Prompt-cache invalidation is another practical concern: structural mutations such as collapse can disrupt provider-side prefix caching. A reported live case saw one collapse drop cache hit rate from 100% to 25% for a turn, forcing recomputation of about 105K tokens before cache stability returned. The paper therefore recommends batching structural mutations rather than issuing many small ones.

The broader significance attributed to Pichay is that once the context window is reinterpreted as L1 rather than total memory, a range of current LLM pathologies—context exhaustion, attention dilution, long-session quadratic cost, and loss of state across restarts—become hierarchy-management problems. Pichay demonstrates that L1 eviction and L2 fault-driven working-set management can be implemented transparently against existing inference APIs. L3 session-local compaction exists in deployed form but has not yet been evaluated at scale, and L4 persistent cross-session memory remains the unresolved frontier. Within the paper’s own framing, Pichay is therefore best understood as a memory-management substrate for agentic LLM sessions: it manages a resident set, measures faults directly, uses those faults to infer the working set, and treats demand paging not as metaphor but as systems architecture.