Sample-Adaptive Visual Token Pruning (SAP)

• SAP is a family of dynamic token reduction methods that condition pruning on sample complexity, redundancy, and relevance.
• It employs varied criteria such as reconstruction objectives, entropy measures, mutual information, and diversity to optimize token selection.
• These techniques enable vision and vision-language models to reduce computational costs while maintaining high task performance.

Searching arXiv for papers on sample-adaptive visual token pruning and closely related methods. Sample-Adaptive visual token Pruning (SAP) denotes a family of token-reduction strategies in which the subset of visual tokens retained by a vision transformer, vision-LLM (VLM), VideoLLM, or related retriever is determined conditionally on the current sample rather than by a fixed global pruning ratio or a fixed layer-wise schedule. Across recent work, SAP has been instantiated through reconstruction objectives, mutual-information estimates, entropy-based image-complexity measures, spatio-temporal dissimilarity, register-anchored relative evidence tests, and attention-diversity hybrids. The common objective is to reduce the computational and memory cost induced by large visual token sets while preserving task performance, retrieval fidelity, or segmentation quality under sample-dependent budgets and selection criteria (Lee et al., 17 Jun 2026, Qin et al., 11 Nov 2025, Wang et al., 28 Sep 2025, Liu et al., 27 Jan 2026).

1. Conceptual scope and terminology

In the recent literature, SAP functions both as a generic label for sample-adaptive visual token pruning and as an overloaded acronym for specific methods. The generic usage appears in works on efficient VLM inference, VideoLLM reasoning, long-context multimodal inputs, and adaptive image representation. At the same time, the acronym also names "Structural Anchor Pruning" for scalable Visual RAG indexing and "Spatial Augmented Pruning" inside ATP-LLaVA, so the exact meaning depends on the paper’s local context (Liu et al., 27 Jan 2026, Ye et al., 2024).

As a generic paradigm, SAP differs from fixed-budget pruning in that the pruning policy is conditioned on sample complexity, redundancy, or relevance. AutoPrune states that previous methods typically apply a fixed schedule uniformly to all input samples and tasks, whereas its Complexity-Adaptive Pruning tailors logistic retention curves to varying sample and task complexities under a predefined computational constraint (Wang et al., 28 Sep 2025). ATP-LLaVA similarly argues that pruning ratio varies across LLM layers and instances and therefore requires a layer-wise and instance-wise strategy (Ye et al., 2024). TrimTokenator-LC extends the same logic to long-context multi-image settings, where redundancy is decomposed into intra-image and inter-image components and token budgets are allocated dynamically across images (Zhang et al., 28 Dec 2025).

Historically, adaptive token pruning predates the recent VLM wave. In vision-only transformers, "SaiT: Sparse Vision Transformers through Adaptive Token Pruning" introduces adaptive token pruning based on input image complexity within a unified dense/sparse training framework, and "Adaptive Sparse ViT" learns stage-wise thresholds that induce sample-adaptive token counts while matching target FLOPs on average (Li et al., 2022, Liu et al., 2022). Recent multimodal work generalizes that principle from image-only token sparsification to text-conditioned relevance, KV-cache pruning, retrieval indexing, and video information flow.

2. Selection objectives and adaptive signals

The defining technical feature of SAP is the choice of an adaptive signal that determines which tokens are retained. The literature exhibits several distinct objective families.

A reconstruction-based family treats token selection as preserving the original visual feature space. SPARE reformulates visual token pruning as a column subset selection problem and minimizes the Frobenius-norm reconstruction error of the visual embedding matrix,

$$\min_{S:|S|=k} \| X_v^\top - \Pi_S X_v^\top \|_F^2,$$

where $\Pi_S$ is the projector onto the span of the selected tokens. Greedy selection is driven by projection residuals

$r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$

and the paper further introduces an anti-relevance term derived from negative CLIP image-text cosine similarity, yielding the joint score $score_n = r_n \cdot \rho_n$ (Lee et al., 17 Jun 2026). The same paper explicitly argues that cosine-normalized diversity discards magnitude information and can therefore fail to approximate the original feature representation faithfully.

A complexity-estimation family uses scalar proxies for sample difficulty. ERASE computes Shannon entropy on image patches, aggregates patch entropies by the median,

$C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$

and uses a lookup table learned once via Bayesian optimization to choose both the stage-1 retention ratio and the decoder layer used for stage-2 context-aware pruning (Lee et al., 11 May 2026). AutoPrune instead uses mutual information between visual and text tokens, computed from early cross-modal attention,

$$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$

and maps this signal to a budget-constrained logistic retention function whose shape varies per sample (Wang et al., 28 Sep 2025).

A dissimilarity and diversity family emphasizes coverage, novelty, or representativeness. SharpV defines normalized Euclidean dissimilarity

$$I(v_1,v_2)=\left\|\frac{v_1}{\|v_1\|_2}-\frac{v_2}{\|v_2\|_2}\right\|_2=\sqrt{2-2\cos\theta},$$

and uses spatial importance, temporal importance, and an adaptive per-frame threshold to decide how many video tokens to keep in each frame (Qin et al., 11 Nov 2025). EVTP-IVS formulates token pruning as a $k$-center coverage problem, augments each token with normalized 2D coordinates, and sets the spatial scaling $\lambda$ to the feature variance so that pruning in joint feature-coordinate space retains a compact but spatially representative subset (Zhu et al., 16 Aug 2025). AgilePruner uses effective rank,

$$\mathrm{erank}(E)=\exp\!\Bigl(-\sum_i q_i\log q_i\Bigr),$$

to quantify image complexity and modulates diversity filtering strength accordingly (Baek et al., 1 Mar 2026).

A relevance-threshold family uses relative comparisons rather than absolute ranking. OccamToken introduces a register token and asks whether a visual token provides stronger evidence than that reference under the current attention distribution. Stage-I pruning keeps tokens satisfying $\Pi_S$0, and stage-II pruning inside the LLM keeps tokens satisfying $\Pi_S$1 (Li et al., 28 May 2026). This directly contrasts with fixed top-$\Pi_S$2 schemes. ATP, in a different sense, fuses ViT CLS attention and CLIP text-image similarity into a hybrid score

$\Pi_S$3

and then retains the top-$\Pi_S$4 patches for the sample (Li et al., 14 Dec 2025).

3. Algorithmic architectures

Recent SAP methods are predominantly organized as one-stage greedy selectors or two-stage pipelines.

SPARE is a deterministic greedy CSSP procedure. It initializes residuals with token norms, iteratively selects the token with largest residual or re-weighted residual, orthonormalizes the selected vector by Gram-Schmidt, and updates the remaining residuals in $\Pi_S$5 time each. Its dominant cost is $\Pi_S$6 when $\Pi_S$7, and its memory overhead stores residuals, anti-relevance weights, and the basis $\Pi_S$8 (Lee et al., 17 Jun 2026). The method is fully training-free.

ERASE is explicitly two-stage. Stage 1 performs image-level pruning directly from raw-pixel entropy, retaining the top $\Pi_S$9 patches with highest entropy. Stage 2 enters the multimodal decoder, computes text-to-vision cross-attention at a single decoder layer $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$0, sums attention over text heads and positions to obtain a relevance score for each remaining token, and keeps the top $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$1 tokens while evicting removed tokens from the KV cache up through layer $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$2 (Lee et al., 11 May 2026). Both $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$3 and $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$4 are chosen per sample through the same entropy-conditioned lookup configuration.

SharpV also uses two stages, but in a video setting. "Visual SharpV" prunes pre-LLM visual tokens using per-frame spatial and temporal information, while "Memory SharpV" prunes degraded visual features from the KV cache inside the decoder through a self-calibration rule based on similarity to the original visual features. The two stages together implement end-to-end sample-adaptive pruning without requiring exposed attention scores and are stated to be fully compatible with FlashAttention and FlashAttention-2 (Qin et al., 11 Nov 2025).

Long-context multimodal reasoning introduces a distinct two-stage structure. TrimTokenator-LC first allocates a content-aware intra-image budget using intra-image diversity and inter-image variation, then performs inter-image pruning by global diversity filtering followed by a Pareto selection that balances diversity with text alignment. The final candidate token set is therefore conditioned simultaneously on per-image richness and cross-image novelty (Zhang et al., 28 Dec 2025).

Not all SAP variants are training-free. ATP-LLaVA inserts ATP modules between decoder layers and predicts per-layer redundancy and spatial thresholds with a small MLP, using differentiable masks and target-token regularization during training (Ye et al., 2024). SaiT alternates dense and sparse training epochs and uses token importance scores from self-attention to support multiple token densities at inference time (Li et al., 2022). Adaptive Sparse ViT similarly learns stage-wise thresholds under a budget-aware FLOPs loss and uses a straight-through approximation for differentiable hard pruning (Liu et al., 2022). This division between training-free inference-time SAP and learned budget-aware SAP is one of the central axes of the field.

4. Representative formulations across application domains

The breadth of SAP is evident from the range of application settings in which it appears.

Method	Adaptive signal	Application
SPARE	Projection residual + anti-relevance	Efficient VLMs
SharpV	Spatial-temporal information + self-calibrated degradation	VideoLLMs
ERASE	Patch entropy median + cross-attention saliency	High-resolution VLMs
AutoPrune	Mutual information + logistic retention curve	VLMs and VLA
OccamToken	Register-anchored relative thresholds	Training-free VLM inference
Structural Anchor Pruning	Middle-layer self-attention centrality	Visual RAG indexing

SPARE represents a subspace-reconstruction interpretation of SAP, where the retained subset is intended to preserve the global visual subspace and complementary context for the prompt (Lee et al., 17 Jun 2026). SharpV shows that sample adaptation can extend beyond visual token selection to hierarchical KV-cache pruning driven by observed visual information degradation inside decoder layers (Qin et al., 11 Nov 2025). ERASE shows that raw-image statistics alone can drive an initial pruning stage before any multimodal interaction, while later context-aware pruning reintroduces text-conditioned selectivity (Lee et al., 11 May 2026).

AutoPrune broadens the scope further by coupling per-sample retention curves to an exact token or FLOPs budget through area-under-curve renormalization of a logistic schedule (Wang et al., 28 Sep 2025). OccamToken departs from absolute token ranking and instead uses a register token as a stable sample-specific reference, thereby making the surviving token count itself sample-adaptive rather than fixed a priori (Li et al., 28 May 2026). Structural Anchor Pruning transfers the SAP principle from generative VLM inference to Visual Document Retrieval, where top patches are selected query-agnostically from middle-layer self-attention centrality to compress the index vector size per page (Liu et al., 27 Jan 2026).

This distribution of methods suggests that SAP is not a single algorithmic family but a systems pattern: adapt the number and identity of visual tokens to the input’s internal structure, temporal dynamics, textual conditioning, or retrieval geometry, and do so early enough to reduce the dominant compute and memory costs.

5. Empirical behavior and efficiency regimes

Across the cited literature, SAP is primarily evaluated through retained accuracy, retrieval fidelity, segmentation quality, FLOPs, latency, prefill time, or KV-cache memory. Although the exact metrics vary by task, a consistent empirical pattern is that adaptive policies outperform fixed or single-cue pruning under aggressive compression.

SPARE reports that on LLaVA-1.5-7B, pruning from $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$5 to $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$6 tokens, corresponding to $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$7 pruning, retains $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$8 overall relative accuracy, compared with CDPruner’s $r_i = \|x_i\|_2^2 - \sum_{q\in Q}(q^\top x_i)^2,$9 and DART’s $score_n = r_n \cdot \rho_n$0. On TextVQA, retaining $score_n = r_n \cdot \rho_n$1 tokens yields a $score_n = r_n \cdot \rho_n$2 speed-up in prefill time and a $score_n = r_n \cdot \rho_n$3 reduction in KV-cache memory versus the full model. The same paper summarizes its headline regime as removing up to $score_n = r_n \cdot \rho_n$4 of visual tokens while retaining $score_n = r_n \cdot \rho_n$5 of baseline performance in a fully training-free manner (Lee et al., 17 Jun 2026).

SharpV reports average final token budgets around $score_n = r_n \cdot \rho_n$6 through the product of visual-token pruning and memory pruning, and on LLaVA-OV-0.5B obtains $score_n = r_n \cdot \rho_n$7 accuracy versus dense $score_n = r_n \cdot \rho_n$8 with a $score_n = r_n \cdot \rho_n$9 faster time to first token. Memory SharpV discards approximately $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$0 of visual-token layers on average without accuracy loss, leading to roughly $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$1 smaller cache and about $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$2 faster per-token decode (Qin et al., 11 Nov 2025).

ERASE reports that on Qwen2.5-VL-7B, at a token pruning ratio of $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$3, it retains $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$4 of the original model accuracy, whereas the best prior method retains $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$5. It also reports corresponding retained-accuracy gains on Qwen3-VL-8B at $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$6 pruning and on InternVL3-8B at $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$7 pruning (Lee et al., 11 May 2026). AutoPrune reports that on LLaVA-1.5-7B, pruning to $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$8 tokens out of $C(x)=\bar H=\mathrm{median}\{H_1,\dots,H_M\},$9 reduces inference FLOPs by $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$0 while averaging $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$1 of original accuracy, and that on LLaVA-NeXT-7B with $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$2 tokens it preserves $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$3 of original performance (Wang et al., 28 Sep 2025).

The same adaptive pattern appears in nonstandard domains. Structural Anchor Pruning reports over $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$4 index-vector reduction with robust retrieval fidelity on ViDoRe and adds only $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$5-$$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$6 ms per page, compared with approximately $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$7 ms for K-means clustering (Liu et al., 27 Jan 2026). EVTP-IVS reports that using only $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$8 of tokens yields up to $$I(V,T)=\sum_{i,j}p(v_i,t_j)\log\frac{p(v_i,t_j)}{p(v_i)p(t_j)},$$9 speed-up on image tasks and up to $$I(v_1,v_2)=\left\|\frac{v_1}{\|v_1\|_2}-\frac{v_2}{\|v_2\|_2}\right\|_2=\sqrt{2-2\cos\theta},$$0 on video IVS while maintaining comparable accuracy relative to the unpruned model (Zhu et al., 16 Aug 2025). These results indicate that SAP is effective not only for generative VLMs but also for retrieval and segmentation workloads.

6. Controversies, failure modes, and open directions

Several recurrent debates structure the SAP literature. One concerns whether attention alone is a reliable pruning signal. AdaptPrune argues that raw attention in LVLMs exhibits a nearly fixed motif across samples and layers, with higher scores clustering toward the end of the patch sequence, which the paper attributes to positional bias inherited from left-to-right token processing. It therefore augments attention with spatial distance and key-vector similarity inside an adaptive non-maximum suppression procedure (Luan et al., 11 Mar 2025). OccamToken advances a related criticism, stating that absolute-ranking paradigms are brittle because attention sinks distort token importance rankings and fixed token budgets are unreliable across inputs (Li et al., 28 May 2026).

A second debate concerns diversity. AgilePruner reports that many diversity-oriented pruning methods preserve substantially less feature diversity than intended, and that the diversity they do retain is closely tied to increased hallucination frequency compared to attention-based pruning on the CHAIR dataset. The same study also finds a conditional asymmetry: attention-based approaches are more effective on simple images where visual evidence is concentrated, while diversity-based methods better handle complex images with distributed features (Baek et al., 1 Mar 2026). This has motivated hybrid or image-aware mixtures rather than single-cue policies.

A third debate concerns where informative pruning signals reside in depth. Structural Anchor Pruning argues that final-layer signals are poor proxies for query-agnostic pruning in Visual RAG because structural signals dissipate during alignment, whereas middle layers exhibit a "Structural Plateau" in which semantic structural anchor patches persist (Liu et al., 27 Jan 2026). ATP-LLaVA likewise reports that static early-layer pruning can cause severe accuracy drops, while static late-layer pruning recovers accuracy but yields little efficiency gain; its image-complexity-aware layer selection is intended to navigate that trade-off (Ye et al., 2024).

SPARE introduces a more counterintuitive finding: tokens with lower image-text relevance score can better preserve contextual information, which it terms an anti-relevance phenomenon (Lee et al., 17 Jun 2026). This directly challenges the widespread assumption that higher text relevance should dominate selection. A plausible implication is that future SAP methods will continue moving away from single-score relevance ranking toward multi-criterion selection, reconstruction objectives, or relative evidence tests.

Open directions are explicitly identified in several works. TrimTokenator-LC raises the question of how to jointly optimize visual token pruning with dynamic token compression in the LLM itself for end-to-end long-context acceleration (Zhang et al., 28 Dec 2025). SharpV frames KV-cache pruning through an information-bottleneck perspective, suggesting that token selection and memory selection may be treated as parts of a unified information-flow problem inside VideoLLMs (Qin et al., 11 Nov 2025). Taken together, these directions indicate that SAP is evolving from a front-end token-dropping heuristic into a broader framework for adaptive multimodal computation allocation.