StorInfer: Accelerating LLM Inference

• StorInfer is a storage-assisted system that precomputes query-response pairs for semantic caching, reducing large language model inference latency.
• It employs advanced techniques like adaptive query masking and adaptive sampling to build a compact, diverse database achieving up to 17.3% latency reduction.
• At runtime, StorInfer uses embedding-based semantic matching to retrieve precomputed responses, balancing storage cost with performance gains.

StorInfer is a precompute-and-retrieve acceleration system for LLM inference that shifts substantial work from the online serving path to an offline preparation phase. Rather than generating every response from scratch at runtime, it builds a large storage of representative query–response pairs offline, embeds and indexes the queries, and at inference time attempts to find a stored query that is semantically close enough to the incoming query; when such a match exists, the system returns the precomputed response instead of invoking the full LLM generation path (Park et al., 30 Sep 2025). The design is framed as storage-assisted inference and as a practical form of semantic caching, but with a stronger emphasis on systematic precomputation, storage layout, and query selection. In the reported evaluation, StorInfer generated 150K unique precomputed pairs, occupied up to 830 MB of storage space, and achieved up to 17.3% latency reduction with no loss in response quality (Park et al., 30 Sep 2025).

1. Conceptual basis and operating principle

StorInfer is organized around a storage-for-latency trade-off. Offline, the system spends computation and storage to generate many query–response pairs in advance; online, it performs fast semantic matching between a user query and the stored query set; if the match is sufficiently good, it returns the cached response directly and avoids a full model forward pass. The paper’s contribution is not merely the idea of storing answers, but the problem of choosing stored queries so that the resulting precomputed database is both compact and useful (Park et al., 30 Sep 2025).

The core abstraction is therefore not a conventional exact-match cache, but a semantic cache lookup. The online workload is partially transformed from autoregressive decoding into approximate semantic retrieval. This design is explicitly motivated by deployment regimes in which latency is critical and query distributions are predictable enough that semantically similar requests recur.

The paper summarizes the governing trade-off as

$$\text{lower latency} \;\Longleftrightarrow\; \text{more stored coverage} \;\Longleftrightarrow\; \text{higher storage cost}.$$

This formulation makes clear that StorInfer is neither a universal replacement for generation nor a purely retrieval-based QA system. Its role is to absorb a meaningful fraction of inference traffic when future requests fall within the semantic coverage of a sufficiently diverse stored query set.

2. System architecture and end-to-end workflow

StorInfer’s architecture is described as a four-stage pipeline. First, an offline query-generation module creates candidate prompts or questions from source text or task data. Second, each generated query is run through the target LLM once offline, producing a response that is stored together with the query. Third, the queries are embedded into vectors and indexed so that the system can quickly search for semantic neighbors at runtime. Fourth, when an online query arrives, the system embeds it, searches the index for the most semantically similar precomputed query, compares that similarity to a threshold, and either returns the stored response or falls back to normal LLM inference (Park et al., 30 Sep 2025).

The paper also presents the workflow in pair notation. In the offline phase, the system stores

$(q_i, r_i),$

where $q_i$ is the query and $r_i$ is the response. At runtime, given a user query $q$, it computes its embedding $e(q)$, retrieves the nearest stored query embedding $e(q_i)$, measures semantic similarity, and returns $r_i$ only if the similarity satisfies the retrieval criterion. This formalization emphasizes that latency savings depend on semantic reuse rather than literal repetition.

The storage and indexing layer is central to this design. The abstract specifies a disk-backed vector database for fast, similarity-based retrieval, while the detailed description emphasizes scalable ANN-style retrieval conceptually aligned with systems such as DiskANN. This suggests a system-level orientation in which storage layout and retrieval structure are treated as first-class components of inference acceleration rather than as auxiliary infrastructure.

3. Offline query generation: adaptive masking and adaptive sampling

StorInfer’s distinctive offline phase is built around two adaptive query-generation techniques: adaptive query masking and adaptive sampling. Together they are intended to produce a diverse, deduplicated set of precomputed query pairs that cover the query space more effectively than naive random generation (Park et al., 30 Sep 2025).

Adaptive query masking operates on the premise that many real queries share underlying structure. The system varies which parts of a source passage or prompt are exposed, thereby creating semantically related but distinct queries. The described algorithmic view is to parse the source text into tokens, spans, or semantic chunks; identify candidate masking regions; score those masks by their expected usefulness for producing diverse but answerable queries; generate masked variants; prompt the LLM to convert each masked variant into a query; and then keep those queries that are diverse and useful for storage. The heuristic objective is to balance semantic preservation against variation: masking too little yields near-duplicates, whereas masking too much risks incoherent or unusable prompts.

Adaptive sampling addresses redundancy from a different direction. Instead of masking fixed portions of the input, it samples candidate queries according to usefulness and diversity. The workflow described in the paper is to generate a candidate pool, embed each candidate, measure similarity to already selected queries, retain candidates that maximize novelty and coverage, and optionally refine sampling probabilities on the basis of observed redundancy. This is characterized as a coverage-maximization strategy: each new stored query should reduce overlap and extend the span of likely future inputs.

Taken together, these methods define StorInfer’s offline precomputation policy. The paper’s broader claim is that the utility of a semantic cache is governed not only by the retrieval mechanism but by how the stored query set is constructed. In this sense, StorInfer treats cache population as an optimization problem over semantic coverage rather than as passive logging of previously seen traffic.

4. Semantic matching, retrieval criterion, and storage design

At runtime, StorInfer formulates retrieval as an embedding-based nearest-neighbor problem. A typical scoring form is cosine similarity,

$$\mathrm{sim}(q, q_i) = \frac{e(q) \cdot e(q_i)}{\|e(q)\| \, \|e(q_i)\|},$$

and the system retrieves

$i^* = \arg\max_i \mathrm{sim}(q, q_i).$

It returns the corresponding response $(q_i, r_i),$0 only if

$(q_i, r_i),$1

where $(q_i, r_i),$2 is a similarity threshold (Park et al., 30 Sep 2025).

This threshold controls the precision–latency trade-off. A higher $(q_i, r_i),$3 is safer but yields fewer cache hits; a lower $(q_i, r_i),$4 increases hit rate but also increases the risk of semantic mismatch. The paper therefore treats semantic matching not as a binary retrieval primitive but as a calibrated decision rule governing when stored computation may safely substitute for live generation.

The storage design is correspondingly motivated by scale. Embeddings are stored and indexed for fast search rather than linearly scanning all precomputed queries at runtime. The resulting system behaves as a semantic cache with explicit support for large stored coverage. The reported database size—about 150K query–response pairs occupying roughly 830 MB—illustrates the intended operating region: large enough to deliver useful semantic coverage, yet still framed as practical from a systems perspective.

A plausible implication is that StorInfer occupies an intermediate design point between classical exact-response caching and retrieval-augmented generation. It does not merely retrieve supporting documents for a fresh synthesis; when the similarity criterion is satisfied, it directly reuses a precomputed response.

5. Evaluation, empirical results, and workload assumptions

The evaluation is described as focusing on offline generation quality and coverage, storage footprint, runtime retrieval effectiveness, and end-to-end latency reduction. The abstract states that the experiments were conducted across multiple QA datasets and emphasizes practicality and scalability, especially in scenarios with predictable query distributions (Park et al., 30 Sep 2025).

The detailed summary identifies the headline results as about 150K precomputed pairs, about 830 MB of storage footprint, and up to 17.3% latency reduction. The same summary states that this reduction was obtained with no loss in response quality. These results define the paper’s empirical claim: storage-assisted inference can produce noticeable latency gains with a moderate storage budget when semantic reuse is sufficiently common.

The task framing references common NLP reading-comprehension and generation datasets in the bibliography, including SQuAD, NarrativeQA, and TriviaQA. The citation context also suggests comparison with systems in the semantic caching, prompt caching, and retrieval-augmented acceleration family, including GPTCache, MeanCache, Prompt Cache, TurboRAG, and CacheBlend. Because the provided excerpt does not include detailed numeric quality tables, the article’s strongest directly stated quantitative conclusion remains the latency–storage trade-off summarized above.

The workload assumptions are explicit. StorInfer is most effective when query distributions exhibit repetition or semantic clustering, such as FAQ-style questions, customer support, educational QA, structured information lookup, repeated enterprise workflows, or prompt patterns around similar documents or topics. It is correspondingly less effective when queries are highly novel, outputs require exact reasoning over fresh context, the similarity threshold must be set very conservatively, or the domain changes rapidly enough that offline precomputation becomes stale.

6. Limitations, deployment considerations, and terminological boundaries

The paper’s design makes several limitations clear. First, StorInfer is coverage-dependent: if a runtime query has no good stored neighbor, the system falls back to full inference. Second, the latency benefit is purchased with storage overhead, since a sizable precomputed database must be maintained. Third, precomputed responses can become outdated if the knowledge base or task distribution changes. Fourth, embedding-based retrieval introduces semantic mismatch risk, since a nearby query in embedding space is not necessarily truly equivalent. Fifth, efficient ANN search is necessary to keep retrieval overhead low enough for net latency benefit (Park et al., 30 Sep 2025).

The deployment discussion is particularly attentive to edge and on-device settings. The potential advantage is reduced invocation of expensive LLM generation, especially for narrow domains with stable workloads and repeated semantics. The principal constraint is that the reported 830 MB footprint is nontrivial for strict on-device memory budgets, and building as well as maintaining the index may itself be expensive on resource-limited hardware. The paper therefore presents edge deployment as most plausible for domain-specific, high-repetition applications rather than open-ended general chat.

A further point of clarification concerns nomenclature. The term “StorInfer” can be confused with broader storage-centric inference architectures, especially work that uses storage-class memory as an extension of the memory hierarchy for inference. One relevant example is “Supporting Massive DLRM Inference Through Software Defined Memory” (Ardestani et al., 2021), which addresses tiered memory for DLRM inference through fast memory, slow memory, software-managed caching, fine-grained I/O, and related placement policies. That work is about software-defined memory for embedding-heavy recommendation serving, not the named StorInfer system for precomputed query storage. The distinction matters because the LLM system and the DLRM system both treat storage as a primary performance enabler, but they do so through different mechanisms: semantic reuse in the former, and memory-hierarchy extension in the latter.