Progressive Sparse Local Attention (PSLA)

• PSLA is a family of attention mechanisms that adaptively focuses computation on localized, high-importance regions in both video and language models.
• It reduces computational complexity and memory usage by replacing dense operations with dynamic, sparsity-driven attention over local neighborhoods or KV blocks.
• Benchmarks show PSLA improves detection accuracy and inference throughput while requiring fewer parameters, making it a practical choice for modern pipelines.

Progressive Sparse Local Attention (PSLA) is a family of attention mechanisms designed to increase computational and memory efficiency in both video understanding and LLM serving, by dynamically focusing computation on informative local neighborhoods or high-importance memory blocks. PSLA variants adaptively allocate computation based on data-driven sparsity, either in space (image/video coordinates) or memory (token/key-value blocks), and have demonstrated superior accuracy-efficiency tradeoffs over dense or static sparse baselines. Two major instantiations are found in video object detection (Guo et al., 2019) and LLM inference (Zhou et al., 1 Mar 2025).

1. Motivation and Conceptual Foundations

Conventional attention mechanisms, whether in vision or LLMs, typically carry prohibitive computational or memory overheads when applied densely. For video object detection, optical-flow-based feature propagation incurs significant model bloat and can misalign features at high semantic levels due to the granularity mismatch between pixel-level flows and deep feature representations. In long-context LLMs, dense key-value (KV) cache attention leads to linear scaling in memory with sequence length, often exceeding the model size and limiting serving throughput. Empirical analysis shows that, in both domains, the majority of useful correspondences—whether motion in video or token dependencies in text—are localized and sparse, with most information concentrated in a fraction of space or memory. PSLA addresses this sparsity through adaptively sampled local attention neighborhoods (vision) or cumulative-attention-mass-driven KV selection (language), omitting the need for inert computations or large storage overheads (Guo et al., 2019, Zhou et al., 1 Mar 2025).

2. Mathematical Formulation and Algorithmic Details

Video Domain: Local Feature Alignment

Given high-level feature maps $F^t \in \mathbb{R}^{C\times H\times W}$ (source) and $F^{t+\epsilon} \in \mathbb{R}^{C\times H\times W}$ (target), and embedding functions $f(\cdot)$, $g(\cdot)$ (typically $1\times1$ convolutions to reduce dimension to $c$), PSLA operates as follows:

• Define a local neighborhood $\Phi(x, y) = \bigcup_{s=0}^d \Phi_s(x, y)$ around each spatial position $(x, y)$ in $g(F^{t+\epsilon})$, where $\Phi_0$ is the center and each $\Phi_s$ for $1 \leq s \leq d$ forms a sparse “shell” at stride $s$. This yields a dense grid near the center, becoming sparser with increasing $s$.
• Compute inner-product affinity between embeddings for each $(p_1, p_2) \in (x, y)\times\Phi(x, y)$:

$$c_{(p_1, p_2)} = \langle g(F^{t+\epsilon}_{x, y}), f(F^t_{x_2, y_2}) \rangle$$

• Apply softmax normalization over the neighborhood:

$$\hat{c}_{(p_1, p_2)} = \frac{\exp(c_{(p_1, p_2)})}{\sum_{q \in \Phi(x, y)} \exp(c_{(p_1, q)})}$$

• Aggregate features:

$$\hat{F}^{t+\epsilon}_{x, y} = \sum_{p_2 \in \Phi(x, y)} \hat{c}_{(p_1, p_2)} F^t_{x_2, y_2}$$

This operation is denoted $$\hat{F}^{t+\epsilon} = \mathrm{PSLA}(F^{t+\epsilon}, F^t)$$ (Guo et al., 2019).

LLM Serving: Attention with Adaptive KV Cache Budgets

For query $q$ and candidate KV block indices $\mathcal{B}$ with block-level metadata $M$, PSLA adaptively selects blocks with the highest “criticality” scores, defined as $CS[b] = q^\top M[b]$, and sorts them descendingly. Attention is computed over microbatches, and accumulation proceeds until the sum of softmax-normalized attention weights (exp-scores) covers a predefined fraction $\epsilon$ (e.g., 0.95) of total estimated attention mass:

$$B_{\ell, t} = \min \{ m \mid \sum_{j=1}^m A_{t, \pi(j)} \geq \epsilon \}$$

where $\pi(\cdot)$ is the descending attention-weight order (Zhou et al., 1 Mar 2025).

Pipelined asynchronous loading and kernel execution, together with a global LRU-managed memory pool, enables efficient dynamic allocation and cache usage.

3. Complexity and Efficiency Benefits

PSLA reduces both parameter count and per-query computational complexity:

• Video PSLA: For maximum displacement $d$, per-output complexity is $O(HWd c)$; dense local attention is $O(HW(2d+1)^2)$, and non-local attention is $O((HW)^2 c)$. Optical-flow-based systems necessitate extra models (e.g., FlowNet, +37M parameters), while PSLA adds just two $1\times1$ convolutions ($\sim$0.5M parameters). Empirical results show model compactness ($\sim$63.7M vs. 96.6M for comparable backbone) and higher accuracy/speed (Guo et al., 2019).
• LLM PSLA: KV memory footprint is reduced by up to 2.4$\times$ vs. dynamic sparse attention (DSA) and up to 8.8$\times$ vs. dense. Throughput is improved by up to 2.0$\times$ under relaxed SLOs. Memory granularity is efficiently managed by global pooling, further improving hit rates and reducing overhead (Zhou et al., 1 Mar 2025).

4. Integration in Downstream Pipelines

Video Object Detection

PSLA is utilized in two principal modules:

• Recursive Feature Updating (RFU): Maintains a “temporal feature” $F_t$ for key frames. $F_t$ is aligned and updated using PSLA and an “Update Net” with learned spatial weight maps, enabling temporal feature enrichment.
• Dense Feature Transforming (DenseFT): For non-key frames, low-level features are encoded, aligned via PSLA to the nearest key frame's temporal feature, and fused using “Quality Net” for detection (Guo et al., 2019).

LLM Inference

PSLA is instantiated during incremental decoding by dynamically adjusting per-token, per-layer budgets for KV block access, intertwined with system-level optimizations:

• Pipelined Iteration Execution: CPU loads KV blocks asynchronously while the GPU computes attention and verifies threshold coverage, reducing CPU–GPU synchronization overhead.
• Unified GPU Memory Management: A single global KV block pool is used across all layers, automatically redistributing capacity according to layer-sparse attention distributions (Zhou et al., 1 Mar 2025).

5. Empirical Evaluation

Video Object Detection (ImageNet VID, ResNet-101 Backbone, $l=10$)

Method	mAP (%)	Runtime (fps, TITAN V)	Params (M)
R-FCN (framewise)	73.9	4.05	59.6
DFF (FlowNet)	73.1	20.3	96.6
FGFA (FlowNet)	76.3	1.36	100.4
PSLA+RFU+DenseFT	77.1	30.8	63.7
PSLA+DCN Backbone	80.0	26.0	72.2
+Seq-NMS	81.4	6.31	—

PSLA improves mAP (+4.0% over DFF), reduces parameters (–34%), and achieves higher throughput (Guo et al., 2019).

LLM Serving (LWM-Text-7B, Llama-3.1-8B, LongBench)

Baseline	KV Usage Reduction (×)	Max QPS Increase (×)
DSA (block-level)	up to 2.4	1.3–1.5
Dense (vllm)	up to 8.8	up to 2.0
Token-level DSA	~1.8	—

Accuracy loss is negligible (within ≈98% of dense), with configurable thresholds via $\epsilon$ (Zhou et al., 1 Mar 2025).

6. Implementation Notes and Extensions

• The threshold $\epsilon$ trades accuracy for memory/compute efficiency and is typically set in the $[0.90, 0.98]$ range.
• Metadata (e.g., cuboid-mean, mean-key) can be modified without affecting PSLA semantics.
• Block and microbatch size must be tuned to balance granularity and kernel launch overhead.
• The global LRU KV block pool can be replaced by more advanced policies (e.g., adaptive replacement cache), and multi-GPU extension is feasible via peer-to-peer memory and distributed metadata (Zhou et al., 1 Mar 2025).
• In vision, PSLA is orthogonal to prefetching or propagation strategies (e.g., RFU vs. DenseFT), and can be composed with deformable convolutions or post-processing for further improvement (Guo et al., 2019).

7. Context and Impact

PSLA has demonstrated clear merit in both vision and language domains:

• In video object detection, it replaces optical-flow modules for temporally coherent feature propagation, leading to reduced model size, elimination of flow estimation artifacts, and improved detection performance (Guo et al., 2019).
• In LLM serving, PSLA introduces an algorithm–system co-design for memory-adaptive, accuracy-preserving sparse attention, mitigating critical bottlenecks in throughput and GPU memory capacity for long-context inference (Zhou et al., 1 Mar 2025).

Key aspects include its data-driven adaptivity, parameter- and compute-efficiency, and straightforward integration with modern pipeline architectures. Deployment at scale is enabled by system-level innovations, including unify memory management and tight CPU–GPU pipelining. No evidence of significant accuracy–efficiency tradeoff under reasonable thresholds is reported, and the design is agnostic to backbone architecture and can coexist with other memory and acceleration schemes.