---
title: Progressive Sparse Local Attention (PSLA)
url: https://www.emergentmind.com/topics/progressive-sparse-local-attention-psla
type: topic
---

# Progressive Sparse Local Attention (PSLA)

Progressive Sparse Local Attention (PSLA) is a family of attention mechanisms designed to increase computational and memory efficiency in both video understanding and large language model (LLM) serving, by dynamically focusing computation on informative local neighborhoods or high-importance memory blocks. PSLA variants adaptively allocate computation based on data-driven sparsity, either in space (image/video coordinates) or memory (token/key-value blocks), and have demonstrated superior accuracy-efficiency tradeoffs over dense or static sparse baselines. Two major instantiations are found in video object detection [1903.09126] and LLM inference [2503.00392].

## 1. Motivation and Conceptual Foundations

Conventional attention mechanisms, whether in vision or language models, typically carry prohibitive computational or memory overheads when applied densely. For video object detection, optical-flow-based feature propagation incurs significant model bloat and can misalign features at high semantic levels due to the granularity mismatch between pixel-level flows and deep feature representations. In long-context LLMs, dense key-value (KV) cache attention leads to linear scaling in memory with sequence length, often exceeding the model size and limiting serving throughput. Empirical analysis shows that, in both domains, the majority of useful correspondences—whether motion in video or token dependencies in text—are localized and sparse, with most information concentrated in a fraction of space or memory. PSLA addresses this sparsity through adaptively sampled local attention neighborhoods (vision) or cumulative-attention-mass-driven KV selection (language), omitting the need for inert computations or large storage overheads [1903.09126, 2503.00392].

## 2. Mathematical Formulation and Algorithmic Details

### Video Domain: Local Feature Alignment

Given high-level feature maps $F^t \in \mathbb{R}^{C\times H\times W}$ (source) and $F^{t+\epsilon} \in \mathbb{R}^{C\times H\times W}$ (target), and embedding functions $f(\cdot)$, $g(\cdot)$ (typically $1\times1$ convolutions to reduce dimension to $c$), PSLA operates as follows:

- Define a local neighborhood $\Phi(x, y) = \bigcup_{s=0}^d \Phi_s(x, y)$ around each spatial position $(x, y)$ in $g(F^{t+\epsilon})$, where $\Phi_0$ is the center and each $\Phi_s$ for $1 \leq s \leq d$ forms a sparse “shell” at stride $s$. This yields a dense grid near the center, becoming sparser with increasing $s$.
- Compute inner-product affinity between embeddings for each $(p_1, p_2) \in (x, y)\times\Phi(x, y)$:
  $$
  c_{(p_1, p_2)} = \langle g(F^{t+\epsilon}_{x, y}), f(F^t_{x_2, y_2}) \rangle
  $$
- Apply softmax normalization over the neighborhood:
  $$
  \hat{c}_{(p_1, p_2)} = \frac{\exp(c_{(p_1, p_2)})}{\sum_{q \in \Phi(x, y)} \exp(c_{(p_1, q)})}
  $$
- Aggregate features:
  $$
  \hat{F}^{t+\epsilon}_{x, y} = \sum_{p_2 \in \Phi(x, y)} \hat{c}_{(p_1, p_2)} F^t_{x_2, y_2}
  $$
This operation is denoted $\hat{F}^{t+\epsilon} = \mathrm{PSLA}(F^{t+\epsilon}, F^t)$ [1903.09126].

### LLM Serving: Attention with Adaptive KV Cache Budgets

For query $q$ and candidate KV block indices $\mathcal{B}$ with block-level metadata $M$, PSLA adaptively selects blocks with the highest “criticality” scores, defined as $CS[b] = q^\top M[b]$, and sorts them descendingly. Attention is computed over microbatches, and accumulation proceeds until the sum of softmax-normalized attention weights (exp-scores) covers a predefined fraction $\epsilon$ (e.g., 0.95) of total estimated attention mass:
$$
B_{\ell, t} = \min \{ m \mid \sum_{j=1}^m A_{t, \pi(j)} \geq \epsilon \}
$$
where $\pi(\cdot)$ is the descending attention-weight order [2503.00392].

Pipelined asynchronous loading and kernel execution, together with a global LRU-managed memory pool, enables efficient dynamic allocation and cache usage.

## 3. Complexity and Efficiency Benefits

PSLA reduces both parameter count and per-query computational complexity:

- **Video PSLA:** For maximum displacement $d$, per-output complexity is $O(HWd c)$; dense local attention is $O(HW(2d+1)^2)$, and non-local attention is $O((HW)^2 c)$. Optical-flow-based systems necessitate extra models (e.g., FlowNet, +37M parameters), while PSLA adds just two $1\times1$ convolutions ($\sim$0.5M parameters). Empirical results show model compactness ($\sim$63.7M vs. 96.6M for comparable backbone) and higher accuracy/speed [1903.09126].
- **LLM PSLA:** KV memory footprint is reduced by up to 2.4$\times$ vs. dynamic sparse attention (DSA) and up to 8.8$\times$ vs. dense. Throughput is improved by up to 2.0$\times$ under relaxed SLOs. Memory granularity is efficiently managed by global pooling, further improving hit rates and reducing overhead [2503.00392].

## 4. Integration in Downstream Pipelines

### Video Object Detection

PSLA is utilized in two principal modules:

- **Recursive Feature Updating (RFU):** Maintains a “temporal feature” $F_t$ for key frames. $F_t$ is aligned and updated using PSLA and an “Update Net” with learned spatial weight maps, enabling temporal feature enrichment.
- **Dense Feature Transforming (DenseFT):** For non-key frames, low-level features are encoded, aligned via PSLA to the nearest key frame's temporal feature, and fused using “Quality Net” for detection [1903.09126].

### LLM Inference

PSLA is instantiated during incremental decoding by dynamically adjusting per-token, per-layer budgets for KV block access, intertwined with system-level optimizations:

- **Pipelined Iteration Execution:** CPU loads KV blocks asynchronously while the GPU computes attention and verifies threshold coverage, reducing CPU–GPU synchronization overhead.
- **Unified GPU Memory Management:** A single global KV block pool is used across all layers, automatically redistributing capacity according to layer-sparse attention distributions [2503.00392].

## 5. Empirical Evaluation

### Video Object Detection (ImageNet VID, ResNet-101 Backbone, $l=10$)

| Method            | mAP (%) | Runtime (fps, TITAN V) | Params (M)       |
|-------------------|---------|------------------------|------------------|
| R-FCN (framewise) | 73.9    | 4.05                   | 59.6             |
| DFF (FlowNet)     | 73.1    | 20.3                   | 96.6             |
| FGFA (FlowNet)    | 76.3    | 1.36                   | 100.4            |
| PSLA+RFU+DenseFT  | 77.1    | 30.8                   | 63.7             |
| PSLA+DCN Backbone | 80.0    | 26.0                   | 72.2             |
| +Seq-NMS          | 81.4    | 6.31                   | —                |

PSLA improves mAP (+4.0% over DFF), reduces parameters (–34%), and achieves higher throughput [1903.09126].

### LLM Serving (LWM-Text-7B, Llama-3.1-8B, LongBench)

| Baseline             | KV Usage Reduction (×) | Max QPS Increase (×) |
|----------------------|-----------------------|----------------------|
| DSA (block-level)    | up to 2.4             | 1.3–1.5              |
| Dense (vllm)         | up to 8.8             | up to 2.0            |
| Token-level DSA      | ~1.8                  | —                    |

Accuracy loss is negligible (within ≈98% of dense), with configurable thresholds via $\epsilon$ [2503.00392].

## 6. Implementation Notes and Extensions

- The threshold $\epsilon$ trades accuracy for memory/compute efficiency and is typically set in the $[0.90, 0.98]$ range.
- Metadata (e.g., cuboid-mean, mean-key) can be modified without affecting PSLA semantics.
- Block and microbatch size must be tuned to balance granularity and kernel launch overhead.
- The global LRU KV block pool can be replaced by more advanced policies (e.g., adaptive replacement cache), and multi-GPU extension is feasible via peer-to-peer memory and distributed metadata [2503.00392].
- In vision, PSLA is orthogonal to prefetching or propagation strategies (e.g., RFU vs. DenseFT), and can be composed with deformable convolutions or post-processing for further improvement [1903.09126].

## 7. Context and Impact

PSLA has demonstrated clear merit in both vision and language domains:

- In video object detection, it replaces optical-flow modules for temporally coherent feature propagation, leading to reduced model size, elimination of flow estimation artifacts, and improved detection performance [1903.09126].
- In LLM serving, PSLA introduces an algorithm–system co-design for memory-adaptive, accuracy-preserving sparse attention, mitigating critical bottlenecks in throughput and GPU memory capacity for long-context inference [2503.00392].

Key aspects include its data-driven adaptivity, parameter- and compute-efficiency, and straightforward integration with modern pipeline architectures. Deployment at scale is enabled by system-level innovations, including unify memory management and tight CPU–GPU pipelining. No evidence of significant accuracy–efficiency tradeoff under reasonable thresholds is reported, and the design is agnostic to backbone architecture and can coexist with other memory and acceleration schemes.

Source: https://www.emergentmind.com/topics/progressive-sparse-local-attention-psla