Latency-Aware Cluster Pre-Computation

• Latency-aware cluster-level pre-computation is a coordinated suite of techniques that groups queries and data to proactively prefetch resources, reducing cache miss penalties and I/O delays.
• It employs methods like hierarchical clustering, graph neural encoding, and spectral clustering to efficiently identify correlated workload clusters for targeted precomputation and load balancing.
• Quantitative evaluations show up to a 33% reduction in query latency and significant improvements in cache hit ratios, ensuring robust performance under bursty and non-uniform loads.

Latency-aware cluster-level pre-computation refers to a suite of coordinated techniques that organize, anticipate, and prefetch data or computation at the level of clusters or groups—of queries, data objects, computation tasks, or users—to minimize end-to-end latency, especially under conditions of non-uniform workload or I/O bottlenecks. It is a recurring principle in disk-based vector search, distributed ML inference, retrieval-augmented generation, and fog/edge computing, where locality, correlated access, and cache-aware orchestration dominate the latency profile.

1. Foundational Models and Objectives

Latency-aware cluster-level pre-computation frameworks formalize latency as the principal cost to optimize, typically by focusing on the dominant term in the system latency decomposition: cache miss penalty (in storage), remote fetch (in distributed compute), or I/O access (in fog/edge). For disk-based vector databases, over 90% of end-to-end query latency is attributable to SSD reads for cluster files absent from in-memory cache. The per-query latency $L(q)$ is

$$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$

where $C(q)$ denotes clusters accessed by query $q$ and $\tau_{\mathsf{miss}} \gg \tau_{\mathsf{hit}}$ (Jeong et al., 23 Sep 2025).

In edge/fog scenarios, total latency combines local computation time, channel upload delay, remote compute, and caching: $$D_a(t) = \sum_{e \in E} x_{ea}(t) D_{ea}^f(t) + (1-\sum_e x_{ea}(t)) D_{ua}^l(t)$$ subject to reliability constraints such as $\Pr\{D_{ea}^f(t)\ge D_{\mathrm{th}}\}\le \epsilon$ (Elbamby et al., 2017).

The overarching objective is to minimize the sum of cache miss penalties, cache-miss-induced fetches, or overall end-to-end delay, subject to reliability or resource constraints.

2. Cluster/Group Identification and Representation

A prerequisite for cluster-level pre-computation is accurate, efficient grouping of queries, data objects, or compute tasks exhibiting correlated or redundant access patterns.

• Vector Search (CALL): For a query batch $Q=\{q_1,\dots,q_n\}$, each query’s IVF cluster set $C(q)$ is condensed to a $K$-bit vector $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$0. Pairwise similarities between queries are calculated using bitwise Jaccard index:

$$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$1

Hierarchical agglomerative clustering (single-linkage, threshold $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$2) groups queries into sets $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$3 with high cluster overlap (Jeong et al., 23 Sep 2025).

• ML Inference (Correlation Grouping): Application developers specify at pipeline startup a mapping $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$4 from a request $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$5 to one or more correlation keys. All objects and compute tasks sharing a key are handled as a group for placement and scheduling (Garrett et al., 2023).
• Graph-based RAG (SubGCache): Retrieved subgraphs $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$6 are encoded with graph neural networks into vectors $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$7. Pairwise Euclidean distances define agglomerative clusters $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$8 of structurally similar subgraphs (2505.10951).
• Fog Computing: Users/devices are assigned to clusters based on a convex combination of geographical and popularity-based similarity matrices $$L(q) = |C(q) \cap \mathsf{Cache}|\,\tau_{\mathsf{hit}} + |C(q) \setminus \mathsf{Cache}|\,\tau_{\mathsf{miss}}$$9 using spectral clustering, aligning with spatial locality and mutual task popularity (Elbamby et al., 2017).

3. Prefetching and Pre-computation Mechanisms

Once clusters or groups are defined, frameworks launch explicit or implicit prefetching and/or pre-computation at group boundaries to minimize observed latency.

• CALL: At each group boundary, metadata $C(q)$0 specifying the next group’s first query $C(q)$1 and its required clusters $C(q)$2 is attached. On group completion, asynchronous I/O fetches $C(q)$3 for the next group, ensuring boundary queries find their clusters in cache, eliminating boundary-spike cache misses. Pseudocode orchestrates prefetch and query execution to rely on precomputed cache content (Jeong et al., 23 Sep 2025).
• SubGCache: For each subgraph cluster $C(q)$4, the union subgraph $C(q)$5 is constructed. The key-value (KV) cache for $C(q)$6 is prefetched and reused by all queries within $C(q)$7, reducing redundant computation. The cluster’s cached tensors are used as prefix for answer generation, with only task-specific suffix computation executed per query (2505.10951).
• Correlation Grouping in ML Inference: Group-level collocation and replication ensure tasks and their input data reside in the same node or shard. Prefetch occurs implicitly as triggers and puts propagate correlated data to the destination ahead of computation (Garrett et al., 2023).
• Fog Computing: Popular task results for each user cluster are proactively computed and cached at their favored cloudlets. Upon request, if the result is cached, the cloudlet serves it directly, avoiding recomputation and transmission (Elbamby et al., 2017).

4. Latency-Aware Scheduling and Load Balancing

Latency-aware cluster-level scheduling mitigates possible straggler effects or bottlenecks due to imbalanced cluster/data sizes.

• Cluster Loader Balancing (CALL): When clusters of variable size must be loaded, missing clusters $C(q)$8 are partitioned among $C(q)$9 threads to minimize the maximum cumulative SSD read per thread. A greedy "largest first" (first-fit decreasing) heuristic sorts clusters by size and assigns each to the loader thread with the smallest total so far, typically reducing tail distribution of load latency by up to 10% (Jeong et al., 23 Sep 2025).
• ML Inference Shard Load Balancing: Nodes within a storage or compute shard are chosen based on least current load, and prefetching is limited to network-free fetches within the same shard, raising average node utilization and minimizing idle periods (Garrett et al., 2023).
• Fog Task-Cloudlet Matching: Decentralized stable matching (deferred acceptance) aligns user offloads to cloudlets, balancing both latency and reliability, and preventing cache or processing overload (Elbamby et al., 2017).

5. Quantitative Evaluation and System Impact

Comprehensive benchmarks across settings reveal that latency-aware cluster-level pre-computation consistently reduces both average and tail latencies, raises cache hit rates, and—where applicable—increases node utilization.

Paper/System	Cache Hit Ratio	99th-percentile Latency	Speedup
CALL (Jeong et al., 23 Sep 2025)	60-92% (FIFO/LRU/CALL)	1.8s→1.2s (33% lower, FEVER)	N/A
SubGCache (2505.10951)	N/A	N/A	Up to 6.68× TTFT
Correlation Grouping (Garrett et al., 2023)	N/A	Median E2E: 600–900ms→200–300ms (local); Azure: 4–8s→200–300ms	>10× tail reduction
Fog Computing (Elbamby et al., 2017)	N/A	Up to 91% average delay reduction	Proactive vs Reactive

Grouping and prefetch overheads remain modest: query clustering and prefetch logic contribute <1.5s overhead even at 2,500 QPS (CALL); clustering for subgraph caching remains under 6% of total latency (SubGCache). Cascade’s implementation of correlation grouping eliminated cache miss storming in distributed ML inference at scale (Garrett et al., 2023).

6. Limitations and Extension Opportunities

Current frameworks reveal residual challenges and opportunities:

• Batch Granularity: Fixed batch-level grouping introduces latency under high arrival rates; adaptive batch sizing could mitigate this (Jeong et al., 23 Sep 2025).
• Prefetch Horizon: Prefetches based only on the first query of each group may be insufficient if cluster sets drift; multi-query or lookahead prefetching generalizations are possible (Jeong et al., 23 Sep 2025).
• Load Imbalance: Hotspots can emerge among clusters/shards; runtime splitting and adaptive resharding are recommended (Garrett et al., 2023).
• Static Latency Modeling: Empirical static parameters (e.g., $q$0 in $q$1) may underrepresent SSD dynamics; incorporating queue depths and parallelism in future models would refine scheduling (Jeong et al., 23 Sep 2025).
• Manual Correlation Specification: Developer-supplied key mapping $q$2 entails some manual effort; more expressive or automated correlation discovery could reduce burden (Garrett et al., 2023).
• Multi-Tenancy and Fairness: Resource contention in shared environments may require budget- or SLA-weighted group allocation (Garrett et al., 2023).
• Scaling to Distributed/Partitioned Systems: Methods are directly portable to partition-level or cross-rack scoping with network-aware cost terms (Jeong et al., 23 Sep 2025).

A plausible implication is that latency-aware cluster-level pre-computation will remain integral to high-throughput, latency-sensitive database, ML, and edge/fog applications, with future work likely to emphasize finer-grained adaptivity, distributed coordination, and more expressive correlation discovery.

7. Synthesis and Cross-domain Applicability

Despite implementation variations, all frameworks analyzed couple three core elements:

• Context-aware grouping (exploiting access/redundancy structure for maximal reuse),
• Group-level/cluster-level prefetching or pre-computation (proactive resource warming to avoid cold-start or boundary misses),
• Load-balanced scheduling (parallelized, bottleneck-aware orchestration to avoid stragglers).

The approach generalizes beyond disk-based vector retrieval to distributed ML pipelines, graph-based RAG, and edge/fog computing, as evidenced across (Jeong et al., 23 Sep 2025, 2505.10951, Garrett et al., 2023), and (Elbamby et al., 2017). Across all domains, the primary system-level benefit is dramatic reduction in both median and tail latencies, robust cache efficiency, and improved utilization under bursty load.