--- title: Buffering for Spatial Sparsity (BSS) url: https://www.emergentmind.com/topics/buffering-for-spatial-sparsity-bss type: topic --- # Buffering for Spatial Sparsity (BSS) Buffering for Spatial Sparsity (BSS) is a criterion introduced to address the challenge of efficiently pruning visual tokens in vision-language models (VLMs) while maintaining both redundancy reduction and adequate spatial coverage of semantic content. BSS operates within the VLM-Pruner framework, a training-free, centrifugal (near-to-far) token pruning algorithm that employs a parallel greedy strategy and selective feature fusion to achieve high inference speed with minimal performance degradation at extreme sparsity [2512.02700]. ## 1. Mathematical Formulation of Buffering for Spatial Sparsity Suppose an image yields a feature map of size $H \times W$, resulting in $N = H \cdot W$ visual tokens indexed by $i \in \{0,\dots,N-1\}$. Each token $i$ contains a $d_k$-dimensional key vector $K_i \in \mathbb{R}^{d_k}$ and a $d$-dimensional hidden state $H_i \in \mathbb{R}^d$. The algorithm first projects $H$ onto its $q$ highest-variance channels (channel screening), producing $\tilde H \in \mathbb{R}^{N \times q}$. The pairwise cosine similarity is \[ M_{ij} = \frac{\tilde H_i^\top \tilde H_j}{\|\tilde H_i\|_2 \|\tilde H_j\|_2} \] Define each token's 2D grid coordinate $p_i = (x_i, y_i)$, with $x_i = i \bmod W$, $y_i = \lfloor i/W\rfloor$. The spatial distance is \[ D_{ij}^{(\rm sp)} = \|p_i - p_j\|_2, \qquad D_{\max} = \sqrt{H^2 + W^2} \] Let $S \subseteq \{0,\ldots,N-1\}$ denote the active retained token set and $C$ its complement. For $i \in C$, define the normalized nearest-neighbor spatial distance \[ \delta_i(S) = \min_{j \in S} D_{ij}^{(\rm sp)},\qquad \bar\delta_i(S) = \frac{\delta_i(S)}{D_{\max}} \] BSS modulates similarity as \[ \widetilde M_{ij} = M_{ij} \bigl(1+\lambda\,\bar\delta_i(S)\bigr) \] with buffering strength $\lambda \geq 0$ (authors set $\lambda=0.5$). This augmentation increases the apparent redundancy of candidates far from $S$, deferring their selection. The surrogate selection score is \[ r_i = 1 - \max_{j\in S} \widetilde M_{ij} \] with candidates accepted if $\max_{j\in S}\widetilde M_{ij} < \tau^{(t)}$, with a scheduled threshold described below. ## 2. Algorithmic Workflow and Parallel Greedy Selection The pruning process follows a centrifugal paradigm: | Step | Description | Hyperparameters | |------|----------------------------|------------------------| | 1 | Channel screening | $q$ | | 2 | Cosine similarity $M$ | | | 3 | Pivot initialization | $\kappa$ pivots; Eq. 3.6 | | 4 | Compute all $D_{ij}^{(\rm sp)}$, $D_{\max}$ | | | 5 | Set threshold/epoch | $\tau^{(0)}=0.8$, $\Delta\tau=0.1$, $t=0$ | | 6 | Main pruning loop | $|S| 1+\lambda$). Discarded tokens are clustered by nearest pivot ($\arg\max_{j\in S} M_{uj}$), and final representation aggregation uses similarity-weighted aggregation (SWA) with \[ \alpha_{u\to j} = \frac{M_{u j}}{\sum_{u' \in D_j} M_{u' j} + \epsilon}, \quad E_j = \sum_{u\in D_j} \alpha_{u\to j} H_u, \quad H_j \leftarrow \beta H_j + (1-\beta)E_j \] with $\beta=0.3$. ## 3. Spatial Buffering and Redundancy Modulation The central principle of BSS is the spatial modulation of redundancy. The minimum spatial distance $\delta_i(S)$, normalized to $[0,1]$ by $D_{\max}$, directly controls the penalization term $1+\lambda\bar\delta_i(S)$. Candidates spatially further from any retained token are more aggressively up-weighted in redundancy: - Early, strict $\tau$ favors tokens in local neighborhoods of current pivots (low redundancy, dense detail). - As $\tau$ increases, acceptance of more remote tokens allows progressive spatial coverage. - The buffer parameter $\lambda$ mediates the speed and strength of outward expansion: higher $\lambda$ defers far tokens more strongly. Geometrically, the newly added tokens are preferentially those that are simultaneously low-redundancy and spatially proximal, balanced by the threshold annealing schedule. The approach enforces a principled “buffer” around $S$, yielding orderly near-to-far token selection, as confirmed by specific qualitative examples (e.g., Fig. 3, [2512.02700]). ## 4. Trade-Offs: Redundancy Versus Spatial Coverage Prior methods based purely on importance tend to over-select from small regions, redundantly wasting capacity. Conversely, pure diversity-based selection over-disperses, missing critical local structure. BSS provides a calibrated compromise: - Centrifugal, near-to-far selection enables rapid local detail preservation near pivots. - Gradually reduced buffering (via increasing $\tau^{(t)}$) ensures global spatial coverage. - The resulting token set covers both fine object details and broader image context. This design demonstrably mitigates the classic trade-off between redundancy reduction and spatial sparsity: local features are preserved before moving to semantically/visually distinct, distant regions. ## 5. Empirical Effects and Comparative Evaluation Empirical evaluation [2512.02700] shows that BSS-equipped VLM-Pruner achieves high accuracy under extreme sparsity and significant improvements relative to baselines. For instance: - On LLaVA-1.5-7B at 88.9% pruning (retain 64/576 tokens), VLM-Pruner with BSS retains 95.61% of model accuracy, outperforming DivPrune (93.68%) and DART (92.71%). - On OCRBench, VLM-Pruner yields a 1.19× end-to-end inference speedup (FLOPs reduced to 22.1% of baseline) while maintaining accuracy; comparable DART speedup is 1.22× but with lower accuracy. - On Qwen2-VL-7B, equivalent sparsity yields 92.58% retained accuracy and a 1.60× speedup. - Across five VLMs and thirteen benchmarks, BSS consistently outperforms both importance-based and diversity-based approaches, with particularly strong gains at high sparsity. ## 6. Implementation Notes and Hyperparameters Key hyperparameters for practical implementation include: - Buffering strength: $\lambda=0.5$ - Pruning threshold schedule: $\tau^{(0)}=0.8$, increment $\Delta\tau=0.1$ - Channel screening dimension: $q$ - Initial pivot count: $\kappa$ - Batch size for parallelism: $B$ - Similarity-weighted aggregation: $\beta=0.3$ All required operations (channel screening, nearest-neighbor search, batching, aggregation) are suitable for efficient parallelization. The process is amenable to integration into existing token-pruning pipelines for vision-language models. ## 7. Summary and Significance Buffering for Spatial Sparsity acts as a sub-graph-modular re-weighting scheme, modifying pairwise visual token similarity with normalized spatial distance, embedded within an annealed-threshold greedy pruning framework. This approach results in orderly, centrifugal token selection that maintains both semantic and spatial coverage, addressing key shortcomings of previous redundancy- or diversity-guided pruning designs. Comprehensive evaluations affirm its effectiveness at high sparsity in multiple VLM architectures and tasks [2512.02700]. Source: https://www.emergentmind.com/topics/buffering-for-spatial-sparsity-bss