Dynamic Token Pooling Techniques

Updated 4 July 2026

Dynamic token pooling is a set of adaptive methods that reduce redundancy by merging, weighting, or budgeting tokens based on input content.
It employs techniques such as dynamic grouping, boundary prediction, and attention-driven weighting to optimize computational efficiency and maintain model performance.
These methods enhance efficiency in autoregressive language models, vision transformers, and video LLMs by preserving critical information while reducing unnecessary computations.

Dynamic token pooling denotes a set of token-reduction or token-reweighting mechanisms in which the pooling pattern is determined by input-dependent features, attention structure, sequence segmentation, or denoising stage rather than by a fixed window or stride. In the recent literature, the term covers several distinct regimes: variable-length contiguous segmentation in autoregressive language modeling (Nawrot et al., 2022), per-image feature-adaptive token merging in vision transformers (Marin et al., 2021), instance-aware switching between pruning and pooling in ViTs (Wu et al., 2023), boundary-based patch merging in visual encoders (Peng et al., 29 Oct 2025), attention-driven token weighting for text embeddings (Pan et al., 31 Aug 2025), and norm-based spatial pooling with hierarchical temporal gridding in Video LLMs (Luo et al., 21 May 2026). A central theme across these variants is that token redundancy is treated as nonuniform: some tokens can be merged, some should be preserved, and some should receive larger weights, but the appropriate decision depends on the current representation rather than on a static geometry.

1. Conceptual scope and taxonomy

The literature does not use a single definition of dynamic token pooling. Instead, it spans dynamic grouping, dynamic weighting, dynamic token-count selection, and architecture-level token-density schedules. Some methods are dynamic only in which tokens are grouped, while the number of retained tokens is fixed; others are dynamic only in token weights; still others explicitly choose a per-input token budget.

Form of adaptivity	Representative mechanism	Characterization
Dynamic grouping / merging	Token Pooling in ViTs (Marin et al., 2021), DRIP (Peng et al., 29 Oct 2025)	Group membership depends on current features
Dynamic policy over prune vs pool	PPT (Wu et al., 2023)	Policy chosen per image and per layer
Dynamic weighting	ATA pooling (Pan et al., 31 Aug 2025), NSP in ST-GridPool (Luo et al., 21 May 2026)	Token influence changes per example
Dynamic token budget	AdaTok (Lu et al., 5 Jun 2026)	Token count is selected per image
Static contrast	Retrieval token pooling (Clavié et al., 2024)	Pooling is document-dependent but offline and query-independent

This comparison clarifies a recurring source of ambiguity. “Dynamic” may refer to content-adaptive grouping, weighting, or budgeting; it does not necessarily imply per-sample variation in the final token count. Token Pooling for Vision Transformers is explicitly dynamic in grouping structure but not in the per-layer counts $K_l$ , which are fixed offline for the model (Marin et al., 2021). ATA pooling is dynamic in attention-derived weights, but it does not reduce sequence length (Pan et al., 31 Aug 2025). By contrast, token pooling for ColBERT-like retrieval is explicitly described as static and query-independent, even though it uses clustering (Clavié et al., 2024).

2. Autoregressive sequence models and dynamic segmentation

A direct formulation of dynamic token pooling appears in autoregressive character-level language modeling, where the central difficulty is to shorten intermediate sequences without violating causality. “Efficient Transformers with Dynamic Token Pooling” introduces a Hourglass-like Transformer in which a boundary predictor operates on hidden states $\mathbf{h}_t$ after the first block, predicts segment boundaries $b_t$ , pools contiguous spans into segment representations $\mathbf{s}_m$ , processes the shortened sequence in the middle block, and then upsamples back to full length in a causality-preserving way (Nawrot et al., 2022). Boundary probabilities are predicted by a two-layer MLP,

$\hat{b}_t = \sigma(\mathrm{MLP}_\phi(\mathbf{h}_t)),$

and can be trained end-to-end with a hard Gumbel-Sigmoid relaxation, or supervised from Unigram tokenization, entropy spikes, or whitespace boundaries. Pooling is segment-wise averaging, and upsampling maps each position $t$ only to the last completed segment before or at $t$ , so future information is never exposed (Nawrot et al., 2022).

The empirical result is that dynamic pooling is both faster and more accurate than vanilla Transformers and fixed-length pooling within the same computational budget. On text8, for example, whitespace and Unigram segmentation reach 1.133 and 1.134 bits per character with shortening factors $5.7\times$ and $5.0\times$ , whereas the vanilla Transformer reports 1.143 and fixed pooling at shortening factor 4 reports 1.166 (Nawrot et al., 2022). The same study shows that average pooling within segments is better than sub-sampling the last token, and that dynamic pooling improves the efficiency–performance frontier across English, Finnish, Hebrew, and Vietnamese. This suggests that contiguous, content-aware segmentation can act as a computational hierarchy rather than merely as an approximation to full attention.

3. Vision transformers: clustering, stagewise reduction, and hybrid policies

In vision, dynamic token pooling emerged partly from the observation that attention is often not the dominant compute bottleneck. “Token Pooling in Vision Transformers” reports that, for typical ViT and DeiT settings, less than ~15% of computation is in softmax self-attention, while more than 80% is in fully connected operations, including QKV and output projections and especially the MLP blocks. The proposed remedy is to reduce the token count $N$ after attention blocks by clustering token embeddings into $\mathbf{h}_t$ 0 groups under a reconstruction objective,

$\mathbf{h}_t$ 1

which is the K-Means objective under nearest-neighbor reconstruction. The grouping is recomputed per image and per layer, optionally with significance-based weights, but the per-layer counts $\mathbf{h}_t$ 2 are fixed for the model. Applied to DeiT, the method achieves the same ImageNet top-1 accuracy with about 42% fewer FLOPs (Marin et al., 2021).

A distinct line, exemplified by PSViT, treats token pooling as a stage transition that decreases token count and increases channel dimension. In PSViT, the number of tokens in each stage and the placement of token pooling are selected by SPOS-based neural architecture search rather than by per-image routing; the resulting schedule is dynamic across depth but fixed at inference. The searched PSViT-2D models improve the speed–accuracy trade-off relative to DeiT, and the paper reports up to 6.6% accuracy improvement in ImageNet classification compared with DeiT (Chen et al., 2021). This is dynamic only at the architecture level, not at the sample level.

PPT makes the per-input distinction explicit. It argues that ViTs exhibit two different kinds of redundancy: inattentive redundancy, addressed by pruning, and duplicative redundancy, addressed by pooling. Its Adaptive Token Compression module computes token scores, measures their variance,

$\mathbf{h}_t$ 3

and chooses token pruning if the variance exceeds a threshold $\mathbf{h}_t$ 4, otherwise token pooling. Pooling uses ToMe-style bipartite soft matching and a size vector to merge similar tokens; pruning keeps top- $\mathbf{h}_t$ 5 tokens according to a class-attention-and-value-norm score. On DeiT-S, PPT reduces over 37% FLOPs and improves throughput by over 45% without any accuracy drop on ImageNet (Wu et al., 2023).

DRIP pushes the dynamic segmentation idea further into visual encoders by predicting boundaries between patch tokens. After $\mathbf{h}_t$ 6 pre-pooling transformer layers, a two-layer MLP predicts boundary probabilities, a Gumbel-Sigmoid produces boundary decisions, and tokens between consecutive boundaries are merged into pooled tokens before the remaining $\mathbf{h}_t$ 7 layers. The resulting boundary maps are visualized as aligning with object edges, semantic transitions, and background complexity, which is why the method is described as “Interpretable Pooling.” Quantitatively, DRIP-4x-16 with a 2+10 split reduces ViT-B-16 from 11.29 GFLOPs to 6.37 GFLOPs while moving from 76.79% to 75.80% top-1 on ImageNet, and it also reduces CLIP pretraining step time and GPU memory (Peng et al., 29 Oct 2025).

4. Dynamic weighting, token-density control, and self-budgeting

Not all dynamic token pooling methods reduce sequence length inside a backbone. In text embeddings, ATA pooling uses the final layer’s attention matrices to derive token weights,

$\mathbf{h}_t$ 8

and then forms the embedding as a weighted sum of hidden states. The method is parameter-free, used identically in training and inference, and is dynamic because the weights depend on each input sequence and its attention pattern. In a controlled MTEB ablation, ATA pooling reaches 65.87, compared with 65.41 for mean pooling, 64.97 for last pooling, and 65.80 for NV-Embed pooling (Pan et al., 31 Aug 2025). Here, the dynamic element is token weighting rather than token reduction.

Diffusion transformers introduce a different axis of adaptivity: token density can vary across both depth and denoising time. FlexDiT uses Poolingformer layers in the bottom segment, Sparse–Dense Token Modules in the middle segment, and dense tokens in the top segment; temporally, it adjusts sparsity across denoising steps so that early noisy steps use fewer sparse tokens and later steps use more. On DiT-XL at $\mathbf{h}_t$ 9, the paper reports a 55% reduction in FLOPs and a 175% improvement in inference speed with only a 0.09 increase in FID (Chang et al., 2024). This is not dynamic token scoring in the strict sense; it is a handcrafted but timestep-dependent token-density schedule.

For Video LLMs, ST-GridPool combines fixed multi-scale temporal gridding with dynamic spatial weighting. Pyramid Temporal Gridding partitions the frame sequence into hierarchical temporal segments and updates segment-end tokens with interpolated multi-frame summaries, while Norm-based Spatial Pooling computes local weights from token norms and uses them in a weighted pooling window. On LLaVA-Video-7B, the reported gains are consistent across benchmarks, including VideoMME 63.3 $b_t$ 0 64.2, Long VideoBench 58.2 $b_t$ 1 60.1, and MVBench 58.6 $b_t$ 2 59.8, all without retraining (Luo et al., 21 May 2026). The temporal component is structured rather than content-adaptive, but the spatial pooling weights are explicitly recomputed per frame and per window.

AdaTok shifts the question to the tokenizer itself. Instead of using a fixed visual token count, it trains a discrete 1D tokenizer so that prefixes remain decodable at budgets $b_t$ 3, and a lightweight GRPO policy then selects the budget per image. AdaTok-Full reaches rFID 1.31 at 256 tokens, while AdaTok-Adaptive attains rFID 1.50 using only ~118 tokens on average, yielding ~2.1x autoregressive throughput over a fixed 256-token decode (Lu et al., 5 Jun 2026). This is dynamic token budgeting rather than dynamic merging inside the encoder, but it is directly relevant to the broader dynamic-token-pooling problem because it makes token count an input-conditioned output.

5. Recurrent mathematical motifs and implementation patterns

Across modalities, three mathematical patterns recur. The first is reconstruction-preserving aggregation. In Vision Transformers, clustering-based pooling minimizes a Chamfer-style reconstruction error over token embeddings, which reduces dynamic pooling to K-Means or K-Medoids in feature space (Marin et al., 2021). In DRIP and the autoregressive LLM, pooling follows predicted boundaries and averages contiguous hidden states into segment-level representations (Peng et al., 29 Oct 2025, Nawrot et al., 2022). In PPT, merge decisions are similarity-driven and size-aware, whereas prune/pool selection itself is driven by the variance of token scores (Wu et al., 2023).

The second pattern is boundary or score prediction. Dynamic pooling in language and DRIP both use explicit boundary variables, predicted from token features with a small MLP and trained via Gumbel-Sigmoid or direct supervision (Nawrot et al., 2022, Peng et al., 29 Oct 2025). ATA and ST-GridPool replace boundary prediction with continuous weighting: ATA derives weights from attention aggregation patterns (Pan et al., 31 Aug 2025), while NSP in ST-GridPool uses

$b_t$ 4

inside each spatial pooling window (Luo et al., 21 May 2026). AdaTok again replaces boundaries with a categorical budget policy over a fixed candidate set (Lu et al., 5 Jun 2026). This suggests that “dynamic token pooling” is mathematically broader than boundary prediction alone; it includes any input-conditioned mechanism that changes how token mass is aggregated or how many tokens are retained.

The third pattern is separation between structural policy and downstream backbone. Several methods introduce no trainable parameters in the pooling operator itself: Token Pooling in ViTs recomputes clustering online but has no learnable pooling weights (Marin et al., 2021); PPT’s policy relies on existing attention and value tensors and adds no new trainable parameters (Wu et al., 2023); ATA pooling is parameter-free (Pan et al., 31 Aug 2025); ST-GridPool is training-free (Luo et al., 21 May 2026). Others use lightweight policy modules—boundary predictors in dynamic segmentation (Nawrot et al., 2022, Peng et al., 29 Oct 2025) and a small allocation policy in AdaTok (Lu et al., 5 Jun 2026)—while keeping the main transformer or tokenizer largely unchanged. Variable output lengths then require padding and masking in batch processing, a practical issue stated explicitly for DRIP and implicit in most per-sample dynamic-length methods (Peng et al., 29 Oct 2025).

6. Empirical regularities, misconceptions, and limitations

A consistent empirical pattern is that content-adaptive pooling usually dominates static pooling at matched compute. Dynamic segmentation in language beats fixed-length pooling at the same shortening factor (Nawrot et al., 2022). Feature-adaptive clustering in ViTs improves the cost–accuracy trade-off over random selection, importance sampling, and PoWER-BERT-style pruning (Marin et al., 2021). PPT outperforms pruning-only and pooling-only baselines across most FLOP regimes (Wu et al., 2023). DRIP improves over fixed global pooling and Swin-style local pooling at the same GFLOPs (Peng et al., 29 Oct 2025). These results support the claim that token redundancy is structured rather than uniform.

A common misconception is that token pooling and token pruning are interchangeable. The surveyed papers repeatedly distinguish them. Token Pooling in ViTs and DRIP merge multiple tokens into representative tokens rather than discarding them (Marin et al., 2021, Peng et al., 29 Oct 2025). PPT explicitly argues that inattentive redundancy and duplicative redundancy require different operations, with pruning better suited to the former and pooling to the latter (Wu et al., 2023). Another misconception is that “dynamic” necessarily means a variable number of tokens per input. This is not generally true: Token Pooling in ViTs uses fixed $b_t$ 5 per layer, ATA keeps the same sequence length, and ST-GridPool’s PTG uses fixed temporal segmentations (Marin et al., 2021, Pan et al., 31 Aug 2025, Luo et al., 21 May 2026). Dynamic token count appears only in a subset of methods, such as AdaTok’s self-budgeting tokenizer (Lu et al., 5 Jun 2026).

The main limitations are also recurrent. Many methods retain a fixed global budget or schedule even when the grouping is adaptive, so they do not realize full adaptive-computation routing (Marin et al., 2021). Boundary-based methods require discrete or relaxed segmentation decisions and can complicate batching and positional handling (Nawrot et al., 2022, Peng et al., 29 Oct 2025). Heuristic weighting rules can work well empirically without a strong theoretical derivation; ATA’s logarithmic attention aggregation and ST-GridPool’s norm-based saliency are explicit examples (Pan et al., 31 Aug 2025, Luo et al., 21 May 2026). Diffusion token-density schedules are often manually chosen rather than learned (Chang et al., 2024). AdaTok’s dynamic budget remains restricted to a discrete candidate set (Lu et al., 5 Jun 2026). A plausible implication is that the field is converging on a common view—token allocation should be content-conditioned—but has not converged on a single optimal control variable: boundaries, clusters, scores, local norms, or global budget policies all remain active design choices.

In that sense, dynamic token pooling is less a single algorithm than a design principle. It treats token representations as unevenly informative, and it uses learned or heuristic structure to decide which tokens to merge, which to weight more heavily, and, in some systems, how many to keep at all. The specific operational form varies by domain, but the underlying objective is stable: to reduce redundant computation while preserving the information most relevant to the current input and stage of processing.