Quest: Query-Aware Sparsity

• Quest: Query-Aware Sparsity is a technique that dynamically activates only the most relevant model partitions based on the current query vector, optimizing memory and compute resources.
• It employs methods like page-based, token-level, and channel-level sparsity to achieve up to 7x faster inference and over 30% memory savings in Transformer and multimodal tasks.
• The approach supports efficient long-context and multimodal processing while maintaining near-baseline accuracy, benefiting real-time and resource-constrained applications.

Query-aware sparsity refers to a class of algorithms and systems that selectively activate or load only the most relevant portions of large model state—including key-value (KV) caches or memory blocks—based on the characteristics of the current query vector at each decoding or inference step. Motivated by the computational and memory bottlenecks in Transformer-based models, especially during long-context inference or with multimodal inputs, query-aware sparsity exploits the empirical sparsity of attention: only a subset of past tokens or input regions are typically critical for generating each output token. This paradigm enables significant reductions in memory bandwidth and latency by maintaining or retrieving only the most salient components, as dynamically estimated in a query-driven fashion.

1. Core Principles of Query-Aware Sparsity

At the heart of query-aware sparsity is the dynamic estimation of relevance for tokens, KV cache pages, or features, conditioned on the current query vector produced by the model at step $t$. The foundational design adopted by multiple recent frameworks—such as Quest, TinyServe, AsyncSpade, SparseVILA, and SparK—is to partition model state (e.g., KV cache, visual token cache, or feature channels) and score these partitions with respect to the active query or a projection thereof.

For page-based approaches, per-page metadata such as coordinate-wise minimum and maximum statistics over keys are precomputed and efficiently stored. For channel-based or token-level sparsity, saliency is calculated using statistical or norm-based estimators derived from feature activations. The query vector is then used to cheaply approximate which partitions contribute the highest potential attention weight or information content, thus dictating which cache blocks or features are loaded or processed further (Tang et al., 2024, Liu et al., 28 Aug 2025, Luo et al., 8 Oct 2025, Khaki et al., 20 Oct 2025, Liao et al., 21 Aug 2025).

2. Algorithmic Mechanisms and Variants

Page-Based Query-Aware Sparsity

Partitioning the KV cache into fixed-size pages is central to Quest, TinyServe, and related systems. For each page, metadata—typically the minima and maxima along each key dimension—are stored. Given a query $q_t$, the upper-bound of the maximal inner-product between $q_t$ and all keys in page $j$ is efficiently computed by:

$$r(q_t, \phi(\mathcal{K}_j)) = \sum_{i=1}^d \begin{cases} q_{t,i}\, M_{j,i}, & q_{t,i} \geq 0 \ q_{t,i}\, m_{j,i}, & q_{t,i} < 0 \end{cases}$$

The top-$K$ pages by these scores are selected; only their content is loaded from memory, dramatically reducing the memory traffic required for each decode step (Tang et al., 2024, Liu et al., 28 Aug 2025).

Token and Channel-Level Query-Aware Sparsity

AsyncSpade predicts the upcoming query vector by regressing over a short window of previous queries and then applies token-level relevance scoring asynchronously to further reduce cache bandwidth and enable cross-rank pipeline overlap (Luo et al., 8 Oct 2025). SparK advances the concept along the feature channel axis: it prunes KV cache channels with low query-dependent saliency, calculated as the product $$\|\overline{\mathbf{q}_i^j}\|_2 \cdot \|\mathcal{K}_i[:,j]\|_2$$ across a window of queries, and dynamically recovers pruned dimensions as needed at attention time (Liao et al., 21 Aug 2025).

Multimodal and Cross-Modal Extensions

SparseVILA generalizes query-aware sparsity to Vision-LLMs (VLMs), combining query-agnostic token pruning during visual encoder prefill with query-aware token retrieval in decoding. This enables aggressive visual token reduction, maintains high retrieval accuracy for long video contexts, and adapts to multimodal attention patterns (Khaki et al., 20 Oct 2025).

3. Empirical Evaluation and Efficiency Gains

Multiple implementations of query-aware sparsity have been evaluated across language modeling, question-answering, reasoning, and multimodal tasks. Representative empirical findings include:

• Speedup: Quest and TinyServe report up to 2.23x to 3.4x faster self-attention or decoding at 32k context, with end-to-end decoding speedups of up to 7.03x at long sequence lengths (Tang et al., 2024, Liu et al., 28 Aug 2025).
• Memory savings: Both page-based and unstructured (channel) pruning yield 2x or more HBM bandwidth reduction, with SparK specifically reducing KV cache storage by over 30% relative to baseline eviction methods at equivalent or lower accuracy degradation (Liao et al., 21 Aug 2025, Liu et al., 28 Aug 2025).
• Accuracy retention: With adequate page or token budgets, query-aware sparse methods yield negligible drops in perplexity or retrieval accuracy—typically less than 1% for language and VLM tasks. Many approaches achieve accuracy on par with or even slightly better than dense-attention baselines across LongBench, MMLU, and specialized retrieval (Liu et al., 28 Aug 2025, Khaki et al., 20 Oct 2025, Tang et al., 2024).
• Token and page hit rates: Hit rates ≥90% for selected pages, indicating high predictive power of bounding-box or criticality scoring (Liu et al., 28 Aug 2025).
• Concurrency and scaling: AsyncSpade demonstrates that decoupling and pipelining query-aware token selection can yield >20% reduction in time-per-output-token (TPOT) versus page-level sparsity, sustaining near-constant latency even as batch size or context length increases (Luo et al., 8 Oct 2025).

Key ablations isolate components such as bounding-box scoring, fused kernels, and asynchronous selection, consistently demonstrating a trade-off frontier between memory footprint, computational latency, and output accuracy.

4. Implementation and Hardware Considerations

Efficient realization of query-aware sparsity requires careful kernel and memory system co-design:

• Metadata management: Page-wise or token-wise metadata (e.g., min/max vectors, saliency weights) should be compact (e.g., $2d$ floats per page) to reside in on-chip SRAM or L2 to avoid latency bottlenecks (Liu et al., 28 Aug 2025, Tang et al., 2024).
• Fused execution: TinyServe and Quest implement single-pass CUDA kernels that compute per-page scores, top-$K$ selection, sparse KV cache loading, and masked attention within the same kernel launch, eliminating kernel-launch and synchronization overheads (Tang et al., 2024, Liu et al., 28 Aug 2025).
• Asynchrony: AsyncSpade disaggregates inference and cache management ranks, fully overlapping KV selection with decoding logic, enabled by lightweight online ridge regression for query prediction and pipelined inter-rank dispatch (Luo et al., 8 Oct 2025).
• Quantization and compression: Query-aware sparsity is complementary to quantization schemes (e.g., AWQ, SmoothQuant), further reducing memory requirements and compute load without retraining or model calibration (Khaki et al., 20 Oct 2025, Liao et al., 21 Aug 2025).
• Compatibility: The mechanisms are architecture-agnostic, requiring only access to KV cache structure and attention projections. They generalize to any Transformer variant—encoder, decoder, encoder-decoder, and mixture-of-experts (MoE) (Liu et al., 28 Aug 2025, Khaki et al., 20 Oct 2025).

5. Query-Aware Sparsity in Reasoning and Database Approximation

Beyond autoregressive LLM inference, query-aware sparsity arises in probabilistic query approximation for large sparse binary datasets (Pavlov et al., 2013). In this domain, model capacity is directed toward subspaces most frequently queried by users, such as selecting only those Markov random field (MRF) parameters for itemsets appearing in high-probability queries. This principle focuses estimation and memory usage on relevant, high-T(Q)-mass patterns, thereby enhancing accuracy for query distributions characteristic of the application.

In reasoning tasks, the RaaS algorithm extends query-aware sparsity by identifying and retaining milestone tokens (i.e., critical intermediate results such as lemmas) in the KV cache, until they are no longer relevant, providing both $O(L)$ time and memory complexity with accuracy on par with page-based query-aware approaches (Hu et al., 16 Feb 2025).

6. Limitations, Challenges, and Future Directions

While query-aware sparsity has established itself as a practical method for scaling model inference, several limitations are noted:

• Critical dependency on tuning: Selection of page or channel budgets (e.g., $q_t$0, $q_t$1, pruning fractions $q_t$2/$q_t$3) directly affects the trade-off curve; overly aggressive sparsity can incur rare but significant misses of long-range or rare-dependency tokens (Tang et al., 2024, Liu et al., 28 Aug 2025, Liao et al., 21 Aug 2025).
• Early layer effectiveness: Page-based sparsity is generally only viable from intermediate transformer layers and above, as early layers exhibit lower attention sparsity (Tang et al., 2024).
• Recovery and value pruning: For fine-grained (channel) sparsity, heuristics for value cache pruning and stochastic recovery remain simplistic; more principled or learned recovery mechanisms are an open area (Liao et al., 21 Aug 2025).
• Engineering overhead: Custom kernels (Triton/CUDA) and system stack modifications are generally required for efficient integration, increasing engineering complexity (Khaki et al., 20 Oct 2025).
• Asynchrony hardware dependencies: Asynchronous frameworks (e.g., AsyncSpade) require multi-rank or multi-device serving infrastructure, which may limit deployment in some environments (Luo et al., 8 Oct 2025).

Ongoing research targets finer-grained scheduling of sparsity, more expressive recoverability functions, unification with data-dependent query distributions, and generalization to non-transformer models and large-scale database settings.

7. Representative Systems and Empirical Benchmarks

System	Sparsity Granularity	Efficiency Gain	Accuracy
Quest/TinyServe	Page-level (KV-cache)	2.2–3.4× speedup, 2× mem	<1% drop on LongBench
AsyncSpade	Token-level (KV-cache)	>20% lower TPOT than Quest	Matches full attention
SparK	Channel-level (KV, unstruct)	>30% lower KV storage	≤2% drop at 80% prune
SparseVILA	Token-level (visual KV)	2.5× decode, 2.6× end-to-end	≤1% drop, VQA/reasoning

These empirical results establish query-aware sparsity as the leading paradigm for efficient long-context and multimodal inference in LLMs and VLMs, providing a tunable balance between efficiency objectives and model quality (Tang et al., 2024, Liu et al., 28 Aug 2025, Khaki et al., 20 Oct 2025, Liao et al., 21 Aug 2025, Luo et al., 8 Oct 2025).