Early-Exit Guided Prefetching

• The paper introduces a method that uses an auxiliary early-exit head to predict future token dependencies, enabling asynchronous staging of SSD-backed phrase memory.
• It combines offline train-free semantic memory construction with a hierarchical GPU–DRAM–SSD storage to reduce synchronous memory lookup delays during decoding.
• Predictive prefetching leverages an intermediate hidden state to overlap compute and memory transfers, recovering throughput and reducing decode latency by nearly 4%.

Early-Exit Guided Predictive Prefetching denotes a latency-hiding inference strategy in which an intermediate Transformer state is used to predict likely future data dependencies early enough that memory movement can overlap with the remainder of the forward pass. In the formulation introduced within TF-Engram, the mechanism uses an auxiliary early-exit prefetch head at layer $L-r$ to estimate likely next tokens, maps those predictions to phrase-memory candidates, and asynchronously stages the corresponding entries through a GPU–DRAM–SSD hierarchy before the final lookup occurs during autoregressive decoding (Ma et al., 8 Jul 2026). The idea is closely related to the predictive execution logic of ADEPT, which makes token-level early exit practical by predicting a final-like hidden state and generating Key-Value caches for skipped layers without sequentially executing them, thereby turning early exit into a predict-and-fill procedure rather than a simple truncation rule (Yoo et al., 7 Jan 2026).

1. Definition and operating context

Within TF-Engram, Early-Exit Guided Predictive Prefetching is the component that makes an SSD-backed phrase memory usable at every autoregressive decoding step without placing external-memory latency directly on the critical path. The underlying system reframes Engram-style memory from a small GPU-resident, hash-compressed side table into a train-free, SSD-backed phrase memory system. In a vanilla design, each token generation step would have to consult phrase memory synchronously, and if the needed entry lived in host DRAM or NVMe SSD, the decoder would stall on random I/O and CPU–GPU transfer. Because autoregressive decoding is inherently sequential, every such delay directly reduces tokens per second (Ma et al., 8 Jul 2026).

TF-Engram therefore couples three elements: offline train-free semantic memory construction, a hierarchical memory store, and predictive prefetching driven by an intermediate early-exit signal. The objective is not to change the final language-model decision rule. Rather, it is to issue memory transfers while the remaining Transformer layers are still computing, so that likely-needed phrase entries have already been promoted into a faster tier when lookup time arrives.

This operational setting distinguishes the method from conventional early exit. In TF-Engram, the early-exit signal is used only for scheduling. The final output token still comes from the original LM head after all $L$ layers complete. This directly addresses a common misconception that the auxiliary head substitutes for the backbone decoder; it does not. The early prediction is advisory for memory movement, not authoritative for generation.

2. Predictive prefetch mechanism

The prefetch algorithm attaches an auxiliary head at an intermediate layer near the end of the decoder, specifically at layer $L-r$. Given the intermediate hidden state $h_t^{L-r}$ at decoding step $t$, this head produces an approximate next-token distribution:

$$p_{\mathrm{early}}(x_{t+1}\mid x_{\le t}) = \mathrm{softmax}\!\left(W_{\mathrm{pref}} h_t^{L-r}\right).$$

From that distribution, the system selects the top-$K$ likely next tokens:

$$\mathcal{T}_t^{\mathrm{early}} = \mathrm{TopK}\!\left(p_{\mathrm{early}}(x_{t+1}\mid x_{\le t})\right).$$

These token candidates are then expanded through the token-to-phrase index into candidate phrase-memory targets:

$$\mathcal{Q}_t = \left\{ \phi_i \mid \tau_i \in \mathcal{M}(\mathcal{T}_t^{\mathrm{early}}) \right\},$$

where $\tau_i$ is the tokenized form of phrase $L$0, and $L$1 maps predicted token paths to phrase entries through the prefix index (Ma et al., 8 Jul 2026).

Scheduling is priority-based. Each candidate phrase is assigned a score

$L$2

where $L$3 is the predicted probability of the token path, $L$4 estimates the phrase’s usefulness, and $L$5 measures the benefit of prefetching from the current storage tier. Entries still on SSD receive higher benefit than entries already in GPU memory. The system then asynchronously issues fetches for the highest-priority phrases, subject to an I/O budget and queue capacity, while low-priority candidates are dropped.

The timing of the request is central. Prefetch is issued immediately after layer $L$6 finishes, while layers $L$7 are still executing. The available overlap window is approximately

$L$8

This means that SSD-to-DRAM or DRAM-to-GPU promotion can proceed in parallel with useful Transformer computation rather than serially after it. A plausible implication is that the choice of $L$9 is fundamentally a balance between prediction fidelity and overlap opportunity: earlier prediction increases time to hide latency, while later prediction improves the quality of the predicted token distribution.

3. Hierarchical memory and lookup integration

TF-Engram stores large phrase tables across a storage hierarchy

$L-r$0

with hot entries in GPU memory, warm entries in host DRAM, and the full table on NVMe SSD. Offline construction mines phrases from FineWeb-Edu, Wikipedia, C4 English, arXiv, and PubMed; tokenizes each phrase with the target LLM tokenizer; encodes it with a frozen semantic encoder, Qwen3-Embedding in the implementation; and stores a phrase-specific key-value entry. Rather than hashing many phrases into ambiguous slots, the system maintains explicit logical entries and addresses them through a two-level index: token prefix $L-r$1 phrase id $L-r$2 memory address (Ma et al., 8 Jul 2026).

At runtime, the backbone still computes the current hidden state, and the memory module still performs phrase matching and gated residual injection. The lookup path checks whether an entry is already in GPU cache, then checks DRAM, and finally relies on SSD. The Engram-style injection remains

$L-r$3

and if multiple phrases match, their vectors can be aggregated as

$L-r$4

Predictive prefetching does not modify this lookup logic; it changes only where the likely-needed vectors reside when the lookup is executed.

The handling of incorrect predictions is also explicitly bounded. A prefetch may miss because the early token guess is wrong, because the predicted phrase is not used, or because the entry is already hot in GPU memory. In those cases, the cost is wasted I/O and possible cache pollution, but correctness is unchanged because the authoritative copy remains available in DRAM or SSD. The top-$L-r$5 filter, priority score, and queue limits are the main controls against runaway wasted fetches. This suggests that the mechanism is conservative in semantic effect: errors influence efficiency, not the final decoding semantics.

4. Empirical behavior in TF-Engram

On Qwen3-0.6B, TF-Engram improves the average downstream score from 57.6 to 59.4, outperforming both the frozen backbone and a parameter-matched LoRA baseline. System evaluation further reports that large TF-Engram tables can be built with moderate offline cost, that SSD-backed storage substantially reduces GPU memory demand, and that predictive prefetching recovers much of the throughput loss caused by external memory access (Ma et al., 8 Jul 2026).

The runtime measurements isolate the effect of prefetching. For throughput, an SSD-backed 100M-entry table without prefetching reduces throughput from 481.2 tokens/s to 439.2 tokens/s. With predictive prefetching, throughput recovers to 460.3 tokens/s with Top-64 and 462.4 tokens/s with Top-128. For latency, the baseline Qwen3-0.6B runs at 2.08 ms/token. A GPU-resident 10M-entry table increases latency slightly to 2.16 ms/token, whereas the SSD-backed 100M-entry table without prefetching reaches 2.28 ms/token, a 9.57% overhead. Prediction-guided prefetching reduces latency to 2.24 ms/token with Top-16, 2.17 ms/token with Top-64, and 2.16 ms/token with Top-128.

The paper summarizes this as reducing the decode overhead from 9.57% to around 4%, bringing performance close to the GPU-resident configuration while supporting a 10× larger table. In sensitivity analysis, the best balance was achieved at an early-exit layer of 8 with Top-64 prefetching, which gave the highest average benchmark score and 2.177 ms/token decode latency.

The storage trade-off is substantial. A 100M-entry table occupies 215.86 GB with 1024-dimensional FP16 vectors. Capacity pressure is therefore shifted away from GPU memory toward host and SSD storage. The construction pipeline is also nontrivial, though the reported build time for 100M entries is about two hours, with n-gram extraction dominating offline cost.

5. Relation to ADEPT and predictive execution in Transformer inference

ADEPT addresses a different bottleneck, but its core mechanism is structurally analogous. The paper argues that prior early-exit methods save compute only partially because of a Transformer-specific constraint: if a token exits early, the model still needs Key-Value representations for skipped layers so that later attention computations remain consistent. This “fundamental problem of sequential dependency of KV cache in attention computation” has historically restricted early exit to the prompt level in prefill or to the first generated token in generation. ADEPT’s contribution is to decouple early-exit decisions from this sequential KV-generation constraint, enabling token-level exits in both prefill and generation (Yoo et al., 7 Jan 2026).

The exit rule is based on state convergence. For token $L-r$6 at layer $L-r$7, ADEPT defines the cumulative history vector

$L-r$8

and allows exit when

$L-r$9

with $h_t^{L-r}$0 calibrated to a computational budget $h_t^{L-r}$1. During prefill, each token can independently exit at a different layer, and the active batch shrinks dynamically as exited tokens are removed from subsequent decoder computation.

What makes ADEPT especially relevant to predictive prefetching is the Hidden State Mapper. The mapper takes the current hidden state, the previous hidden state, and the inertia/history state; concatenates them; applies RMS normalization; applies a $h_t^{L-r}$2 convolution; passes the result through an MLP with skip connections; and outputs a mapped final hidden state $h_t^{L-r}$3. For each skipped layer $h_t^{L-r}$4, ADEPT then generates KV caches directly from that mapped state:

$h_t^{L-r}$5

equivalently,

$h_t^{L-r}$6

The operational analogy can be summarized as follows.

System	Predicted object	Operational effect
ADEPT	final-like hidden state and skipped-layer KV caches	eliminates the need to compute queries, attention, and feedforward blocks for tokens that have exited early
TF-Engram	approximate next-token distribution and candidate phrase entries	starts asynchronous fetches so likely-needed memory is staged into GPU cache or DRAM before lookup

In both cases, an intermediate representation is used to anticipate future dependencies and materialize the expensive-to-wait-for state ahead of time. In ADEPT, the anticipated object is internal Transformer state; in TF-Engram, it is external phrase memory. This suggests a broader category of inference methods in which early signals serve as scheduling or reconstruction oracles rather than as simple stopping criteria.

6. Trade-offs, misconceptions, and significance

Both TF-Engram and ADEPT expose the standard predictive-systems trade-off between earlier action and more accurate prediction. In TF-Engram, more aggressive prefetching and earlier exits increase overlap opportunity but may reduce prediction quality and waste I/O; later exits improve prediction but leave less time to hide SSD latency. In ADEPT, lower thresholds optimize smaller reductions, higher thresholds generalize better to larger reductions, and more aggressive thresholds increase reduction but can worsen perplexity or accuracy. ADEPT also notes that threshold selection remains heuristic and that future work should adapt thresholds dynamically to input characteristics (Ma et al., 8 Jul 2026, Yoo et al., 7 Jan 2026).

Several misconceptions are directly addressed by the reported designs. Early-exit guidance in TF-Engram does not alter the final LM output; it affects only memory scheduling. Incorrect prefetches do not affect correctness because the system falls back to ordinary lookup through the memory hierarchy. Likewise, ADEPT is not merely a prompt-level or first-token-only early-exit method; its explicit contribution is token-level adaptive depth selection in both prefill and generation, made practical by decoupled skipped-layer KV generation.

The broader significance lies in how both methods redefine what early exit can do in Transformer inference. TF-Engram uses an intermediate layer as a scheduling oracle for asynchronous movement of phrase-specific memory through a GPU–DRAM–SSD hierarchy, thereby making large, precise, train-free phrase memory practical during autoregressive decoding. ADEPT uses an intermediate signal to predict final-like hidden state and reconstruct skipped-layer KV caches, thereby making token-level early exit feasible without sequentially unfolding every skipped layer. The empirical consequences reported for these two systems are concrete: TF-Engram recovers throughput and lowers latency overhead for SSD-backed memory, while ADEPT improves efficiency by up to 25% in language generation tasks, achieves a 4× speed-up in downstream classification tasks, and reports up to a 45% improvement in performance. Together, these results indicate that early-exit guidance can function not only as a mechanism for conditional computation, but also as a way to anticipate and provision future inference-state requirements before they become blocking dependencies.