Buffering for Spatial Sparsity (BSS)

• Buffering for Spatial Sparsity (BSS) is a pruning technique that reweights token similarity using normalized spatial distances to balance redundancy reduction with spatial coverage.
• It employs a centrifugal, parallel greedy selection algorithm with channel screening and selective feature fusion, dynamically adjusting thresholds for orderly token selection.
• Empirical evaluations demonstrate that BSS retains over 95% accuracy at extreme sparsity while achieving significant inference speedups across various vision-language models.

Buffering for Spatial Sparsity (BSS) is a criterion introduced to address the challenge of efficiently pruning visual tokens in vision-LLMs (VLMs) while maintaining both redundancy reduction and adequate spatial coverage of semantic content. BSS operates within the VLM-Pruner framework, a training-free, centrifugal (near-to-far) token pruning algorithm that employs a parallel greedy strategy and selective feature fusion to achieve high inference speed with minimal performance degradation at extreme sparsity (Wu et al., 2 Dec 2025).

1. Mathematical Formulation of Buffering for Spatial Sparsity

Suppose an image yields a feature map of size $H \times W$, resulting in $N = H \cdot W$ visual tokens indexed by $i \in \{0,\dots,N-1\}$. Each token $i$ contains a $d_k$-dimensional key vector $K_i \in \mathbb{R}^{d_k}$ and a $d$-dimensional hidden state $H_i \in \mathbb{R}^d$. The algorithm first projects $H$ onto its $q$ highest-variance channels (channel screening), producing $N = H \cdot W$0.

The pairwise cosine similarity is

$N = H \cdot W$1

Define each token's 2D grid coordinate $N = H \cdot W$2, with $N = H \cdot W$3, $N = H \cdot W$4. The spatial distance is

$N = H \cdot W$5

Let $N = H \cdot W$6 denote the active retained token set and $N = H \cdot W$7 its complement. For $N = H \cdot W$8, define the normalized nearest-neighbor spatial distance

$N = H \cdot W$9

BSS modulates similarity as

$i \in \{0,\dots,N-1\}$0

with buffering strength $i \in \{0,\dots,N-1\}$1 (authors set $i \in \{0,\dots,N-1\}$2). This augmentation increases the apparent redundancy of candidates far from $i \in \{0,\dots,N-1\}$3, deferring their selection. The surrogate selection score is

$i \in \{0,\dots,N-1\}$4

with candidates accepted if $i \in \{0,\dots,N-1\}$5, with a scheduled threshold described below.

2. Algorithmic Workflow and Parallel Greedy Selection

The pruning process follows a centrifugal paradigm:

Step	Description	Hyperparameters
1	Channel screening	$i \in \{0,\dots,N-1\}$6
2	Cosine similarity $i \in \{0,\dots,N-1\}$7
3	Pivot initialization	$i \in \{0,\dots,N-1\}$8 pivots; Eq. 3.6
4	Compute all $i \in \{0,\dots,N-1\}$9, $i$0
5	Set threshold/epoch	$i$1, $i$2, $i$3
6	Main pruning loop	$i$4, buffer parameter $i$5

Candidates in $i$6 are ranked by $i$7 and processed in parallel batches (size $i$8). For each batch, candidates are analyzed with up-to-date $i$9; those meeting the threshold condition are added. If no additions occur in a round, the threshold is incremented: $d_k$0. The process continues until $d_k$1 or failsafe ($d_k$2).

Discarded tokens are clustered by nearest pivot ($d_k$3), and final representation aggregation uses similarity-weighted aggregation (SWA) with

$d_k$4

with $d_k$5.

3. Spatial Buffering and Redundancy Modulation

The central principle of BSS is the spatial modulation of redundancy. The minimum spatial distance $d_k$6, normalized to $d_k$7 by $d_k$8, directly controls the penalization term $d_k$9. Candidates spatially further from any retained token are more aggressively up-weighted in redundancy:

• Early, strict $K_i \in \mathbb{R}^{d_k}$0 favors tokens in local neighborhoods of current pivots (low redundancy, dense detail).
• As $K_i \in \mathbb{R}^{d_k}$1 increases, acceptance of more remote tokens allows progressive spatial coverage.
• The buffer parameter $K_i \in \mathbb{R}^{d_k}$2 mediates the speed and strength of outward expansion: higher $K_i \in \mathbb{R}^{d_k}$3 defers far tokens more strongly.

Geometrically, the newly added tokens are preferentially those that are simultaneously low-redundancy and spatially proximal, balanced by the threshold annealing schedule. The approach enforces a principled “buffer” around $K_i \in \mathbb{R}^{d_k}$4, yielding orderly near-to-far token selection, as confirmed by specific qualitative examples (e.g., Fig. 3, (Wu et al., 2 Dec 2025)).

4. Trade-Offs: Redundancy Versus Spatial Coverage

Prior methods based purely on importance tend to over-select from small regions, redundantly wasting capacity. Conversely, pure diversity-based selection over-disperses, missing critical local structure.

BSS provides a calibrated compromise:

• Centrifugal, near-to-far selection enables rapid local detail preservation near pivots.
• Gradually reduced buffering (via increasing $K_i \in \mathbb{R}^{d_k}$5) ensures global spatial coverage.
• The resulting token set covers both fine object details and broader image context.

This design demonstrably mitigates the classic trade-off between redundancy reduction and spatial sparsity: local features are preserved before moving to semantically/visually distinct, distant regions.

5. Empirical Effects and Comparative Evaluation

Empirical evaluation (Wu et al., 2 Dec 2025) shows that BSS-equipped VLM-Pruner achieves high accuracy under extreme sparsity and significant improvements relative to baselines. For instance:

• On LLaVA-1.5-7B at 88.9% pruning (retain 64/576 tokens), VLM-Pruner with BSS retains 95.61% of model accuracy, outperforming DivPrune (93.68%) and DART (92.71%).
• On OCRBench, VLM-Pruner yields a 1.19× end-to-end inference speedup (FLOPs reduced to 22.1% of baseline) while maintaining accuracy; comparable DART speedup is 1.22× but with lower accuracy.
• On Qwen2-VL-7B, equivalent sparsity yields 92.58% retained accuracy and a 1.60× speedup.
• Across five VLMs and thirteen benchmarks, BSS consistently outperforms both importance-based and diversity-based approaches, with particularly strong gains at high sparsity.

6. Implementation Notes and Hyperparameters

Key hyperparameters for practical implementation include:

• Buffering strength: $K_i \in \mathbb{R}^{d_k}$6
• Pruning threshold schedule: $K_i \in \mathbb{R}^{d_k}$7, increment $K_i \in \mathbb{R}^{d_k}$8
• Channel screening dimension: $K_i \in \mathbb{R}^{d_k}$9
• Initial pivot count: $d$0
• Batch size for parallelism: $d$1
• Similarity-weighted aggregation: $d$2

All required operations (channel screening, nearest-neighbor search, batching, aggregation) are suitable for efficient parallelization. The process is amenable to integration into existing token-pruning pipelines for vision-LLMs.

7. Summary and Significance

Buffering for Spatial Sparsity acts as a sub-graph-modular re-weighting scheme, modifying pairwise visual token similarity with normalized spatial distance, embedded within an annealed-threshold greedy pruning framework. This approach results in orderly, centrifugal token selection that maintains both semantic and spatial coverage, addressing key shortcomings of previous redundancy- or diversity-guided pruning designs. Comprehensive evaluations affirm its effectiveness at high sparsity in multiple VLM architectures and tasks (Wu et al., 2 Dec 2025).