Layer-wise Token Compression (LTC)

• LTC is an efficiency method that compresses tokens at designated transformer layers based on depth rather than uniformly across the network.
• It employs adaptive techniques such as pruning, merging, and reversible routing to minimize computational and memory overhead while preserving key representations.
• Empirical results across domains like reranking, ViTs, and LLM inference demonstrate significant speed-ups, memory savings, and maintained accuracy.

Layer-wise Token Compression (LTC) denotes a family of efficiency methods in which token reduction, token transformation, or token/KV budget allocation is conditioned on network depth rather than applied uniformly at the input or output interface. In the strict sense, LTC compresses hidden-state sequences at intermediate layers; in broader usage, it also includes methods that choose an adaptive selection layer, assign different budgets to different layers, or infer token importance from multi-layer behavior instead of a single-layer saliency snapshot. Across reranking, long-context language modeling, vision transformers, large vision-language models, and diffusion transformers, the shared objective is to reduce the computation and memory dominated by long token sequences while preserving the representations that later layers need most [2605.20683] [2404.04793] [2606.01756].

1. Conceptual basis and motivation

The central premise of LTC is that token usefulness is not constant across depth. Several papers make this point from different angles. In cross-encoder reranking, compressing token embeddings at the input layer may be ineffective because fine-grained query–document interactions emerge inside lower transformer layers; delaying compression to a middle layer preserves ranking quality while increasing inference QPS by up to 25% for passage ranking and up to 116% for document ranking [2605.20683]. In KV-cache compression for decoder-only LLMs, SqueezeAttention argues that attention layers are not equally important during inference and reports that the least-important cluster (G_3) typically accounts for around (50\%) to (70\%) of all layers, which is why a uniform per-layer KV budget is suboptimal [2404.04793].

A complementary geometric argument appears in the analysis of token correlation across transformer depth. That work reports an expansion-contraction pattern in which the initial 10% of layers diffuse token representations into a higher-dimensional working space and later layers progressively project them toward a lower-dimensional semantic manifold of order (10^1) dimensions [2503.22547]. This suggests that early layers may require representational freedom before later layers become naturally more compressible. RAPID operationalizes a similar intuition for ViTs by assigning redundancy-aware pruning to shallow-to-middle layers and importance-aware merging to deeper layers, rather than using a single reduction operator everywhere [2606.08156].

These arguments converge on the same LTC principle: token compression is not only a question of which tokens are redundant, but also of when in depth that redundancy becomes safely exploitable.

2. Operational forms of layer-wise compression

LTC is not a single mechanism. One explicit formulation compresses the hidden states at a chosen intermediate layer. For document reranking, a compression module (\mathcal{C}) is applied to layer-(l^*) hidden states (\mathbf{H}^{(l^*)}\in\mathbb{R}^{n\times h}), producing
[
\mathbf{H}^{(l^)}_{\text{compressed}}=\mathcal{C}(\mathbf{H}^{(l^)},r)\in\mathbb{R}^{n'\times h},
\qquad n'=\lfloor n\cdot r\rfloor,
]
after which later layers process the shortened sequence [2605.20683]. Here the defining layer-wise decision is where (l^*) is placed and how aggressive (r) is.

A more general formulation treats token reduction itself as a matrix transformation. Token Transforming writes compression as
[
\boldsymbol{Y}=\boldsymbol{W}\cdot\boldsymbol{X},
]
where (\boldsymbol{X}\in\mathbb{R}^{N\times d}), (\boldsymbol{Y}\in\mathbb{R}^{M\times d}), (M<N), and (\boldsymbol{W}\in\mathbb{R}^{M\times N}) [2506.05709]. Under this view, pruning corresponds to a special one-to-one selection matrix, merging to a many-to-one assignment matrix, and many-to-many transformation to a denser operator that allows one original token to contribute to more than one compressed token. This is a particularly broad algebraic interpretation of LTC because it defines layer-wise compression as repeated sequence-to-sequence maps with changing token counts.

Other LTC systems emphasize reversible routing rather than irreversible deletion. ALVTS uses a lightweight token selector that identifies important tokens for further processing while allowing less important tokens to skip the layer; the two streams are then reintegrated before subsequent layers, so skipped tokens are not permanently inaccessible [2606.14277]. This differs sharply from static pruning, in which once a token is removed at one layer it is unavailable to all later layers. Prune-and-Merge for ViTs offers another non-destructive variant: tokens are compressed before a block by a trainable merge matrix (\mathbf{M}), processed in reduced form, and then reconstructed by (\mathbf{M}^+) with a shortcut path for pruned-token information, so every layer can have its own compression structure without destroying the original spatial frame [2503.23455].

Sequence-level latent compression is adjacent rather than identical to LTC. (K)-Token Merging compresses each contiguous block of (K) input embeddings into one latent embedding before the transformer stack, and the resulting compressed prompt is then processed at fixed length through all layers [2604.15153]. Because the token count does not change with depth, this is better viewed as an input-level latent bottleneck than as strict LTC, but it is relevant as a boundary case that shows how much redundancy can be removed before layer-wise mechanisms are even introduced.

3. How layer-wise importance is estimated

A major LTC design axis is the scoring signal used to decide which tokens or layers to compress. EvoCut exemplifies a strict multi-layer token-scoring approach. Let (x_i^{(\ell)}\in\mathbb{R}^d) be token (i) at vision-encoder layer (\ell). EvoCut defines normalized evolution directions
[
\vec{\Delta}_i^{(\ell)}=\frac{x_i^{(\ell)}-x_i^{(\ell-1)}}{|x_i^{(\ell)}-x_i^{(\ell-1)}|_2},
]
clusters these directions into (M) groups with K-means, measures each token’s best cosine alignment (a_i^{(\ell)}) to any group direction, converts this into a deviation score (r_i^{(\ell)}=1-a_i^{(\ell)}), and accumulates evidence with an exponential moving average
[
s_i^{(\ell)}=\alpha s_i^{(\ell-1)}+(1-\alpha)r_i^{(\ell)}.
]
Tokens are retained by top-(K) on the final (s_i^{(L)}). The distinctive LTC claim is that token importance is better inferred from persistent cross-layer deviation than from a single-layer attention map or representation snapshot [2606.01756].

Other methods estimate layer-wise compressibility rather than token saliency alone. ASL monitors the variance of token ranks ordered by attention score across a sliding window of layers and chooses the selection layer online when the relative variance falls below a threshold, interpreting rank stabilization as evidence that the model has identified a trustworthy subset of important tokens [2601.07667]. SqueezeAttention uses cosine similarity between the hidden state before and after each self-attention layer, treating low cosine similarity as higher layer importance and reallocating KV budgets accordingly [2404.04793]. DiffCR learns a continuous compression ratio for each Diffusion Transformer layer, initialized at zero and regularized toward a target average compression, so layer budgets themselves become learnable objects rather than fixed heuristics [2412.16822].

In multimodal settings, query-conditioned attention often replaces purely visual saliency. OmniDrop scores audiovisual tokens by averaging text-to-audiovisual attention at each decoder layer and combines this with a sigmoid depth schedule
[
p_l=p_{init}+(p_{final}-p_{init})\cdot \sigma(l,t_{mid},\beta)
]
to prune little in early layers and more in later layers [2605.14458]. CoViPAL takes a different route: it trains a Plug-and-Play Pruning Module on visual and text tokens so that shallow pruning decisions can imitate deeper contextual signals, because shallow layers themselves “do not expose enough contextual information to prune them safely” [2508.17243]. Representation Shift replaces attention maps entirely by scoring a token from the magnitude of its representation change, with the default choice
[
s=\Delta \mathbf{x}=|F(\mathbf{x})-\mathbf{x}|_2,
]
which makes training-free token compression compatible with FlashAttention [2508.00367].

4. Major design patterns across domains

The LTC literature now spans several related but non-identical design patterns.

Setting	Representative methods	Layer-wise mechanism
ViT classification and dense prediction	PM-ViT [2503.23455], Token Transforming [2506.05709], RAPID [2606.08156]	Per-block prune-and-merge, many-to-many token transformation, or early pruning followed by late merging
LVLM and MLLM visual compression	EvoCut [2606.01756], LaCo [2507.02279], CoViPAL [2508.17243], ALVTS [2606.14277], OmniDrop [2605.14458]	Multi-layer trajectory scoring, internal encoder compression, contextualized pre-backbone pruning, reversible skip/reintegrate, or decoder-layer progressive pruning
LLM KV-cache and long-context inference	SqueezeAttention [2404.04793], ASL [2601.07667]	Layer-specific KV budgets or adaptive choice of the one-shot token-selection layer
Reranking	LTC reranking [2605.20683]	Intermediate-layer adaptive average pooling for pointwise and listwise rerankers
Diffusion transformers	DiffCR / DiffRatio-MoD [2412.16822]	Learnable layer-wise and timestep-wise compression ratios with top-(k) token routing
Adjacent, not strict internal LTC	token correlator analysis [2503.22547], (K)-Token Merging [2604.15153], Z-token compression [2603.25340]	Diagnose latent compressibility or compress at the sequence interface rather than progressively inside layers

One recurring divide concerns where compression occurs. Post-encoder methods shorten the sequence only for downstream modules, whereas internal methods shorten it for part of the encoder itself. LaCo formalizes internal visual token compression as
[
E_v^k=ENC_{1:k}(V),\qquad \hat E_v^k=PML(E_v^k,r),\qquad \hat E_v=ENC_{k+1:L}(\hat E_v^k),
]
so that layers (k+1,\dots,L) operate on only (|E_v^k|/r^2) tokens [2507.02279]. By contrast, EvoCut keeps the vision encoder unchanged and prunes only after its final layer, so its savings come from shortening the sequence for the multimodal projector and LLM rather than for the encoder itself [2606.01756].

A second divide concerns whether compression is destructive. Permanent pruning dominates one-shot LVLM and KV-cache methods, but ALVTS and PM-ViT show that LTC can also be reversible or reconstructive, either by bypassing tokens locally and reintegrating them later or by reconstructing compressed block outputs back to the original token lattice [2606.14277] [2503.23455]. This broadens LTC from pure elimination toward conditional participation and reconstruction.

5. Empirical performance and trade-offs

Empirical results show that LTC can be effective under aggressive budgets, but the gains are domain- and mechanism-dependent. EvoCut reports the headline LVLM result of retaining only 64 of 576 visual tokens on LLaVA-1.5-7B, which is (64/576\approx 11.1\%) of the original count, while preserving (94.4\%) of normalized average task performance; at 192 tokens it preserves (98.4\%), and at 128 tokens (97.0\%) [2606.01756]. ALVTS reports that with an 89% token compression ratio it retains 96.7% of the original model’s accuracy, while OmniDrop reports gains of up to 3.58 points over baselines together with prefill latency reduction of up to 40% and memory reduction of up to 14.7% [2606.14277] [2605.14458]. LaCo, which compresses inside the vision encoder, reports training-efficiency gains beyond 20% and inference throughput gains over 15% relative to external compression [2507.02279].

In ViTs, the same pattern appears: more layer-aware or reconstruction-aware methods dominate simpler plug-and-play baselines at similar budgets. PM-ViT reports that on DeiT-Small it achieves a (1.64\times) speed-up with only a (0.2\%) drop in accuracy on ImageNet-1k [2503.23455]. Token Transforming reports that it reduces 40% FLOPs and accelerates DeiT-S by (\times 1.5) with marginal 0.1% accuracy drop, while RAPID reports up to 4.29% higher accuracy than ToMe at extreme reduction rates by using pruning in shallow layers and merging in deep layers instead of a uniform operator [2506.05709] [2606.08156].

For LLM inference, the main measurable benefits are memory reduction, decoding throughput, and stability across tasks. SqueezeAttention reports around 30%–70% memory reduction and up to 2.2 times throughput improvement by jointly optimizing sequence-wise and layer-wise KV budgets [2404.04793]. In long-context LLM inference, ASL improves the accuracy of one-shot layer-wise token pruning by adapting the selection layer online, particularly on hard retrieval tasks, while keeping the same decoding-stage budget and near-identical memory to fixed-budget baselines [2601.07667]. In document reranking, layer-wise token compression at intermediate layers increases inference QPS by up to 25% for passage ranking and up to 116% for document ranking, and the same design transfers to listwise LLM rerankers with larger gains in long-context settings [2605.20683].

Diffusion transformers and FlashAttention-compatible pruning extend LTC beyond standard ViTs and decoder-only LLMs. DiffRatio-MoD-L improves generation quality over uniform-ratio MoD at the same 20% compression budget, with FID 12.28 versus 22.78 on text-to-image and 13.53 versus 18.34 on inpainting, while keeping latency and memory close to the uniform baseline [2412.16822]. Representation Shift shows that the importance metric itself can dominate realized speed: on UMT-B for video-text retrieval, attention-based pruning and representation-shift pruning have the same 156.4 GFLOPs, yet throughput differs from 57 to 175 because representation shift remains compatible with FlashAttention [2508.00367].

6. Misconceptions, limitations, and open questions

A common misconception is that LTC always means progressive internal token dropping at every layer. The literature is broader. EvoCut is layer-wise because it scores tokens from multi-layer evolution but prunes only once after the vision encoder; ASL is layer-wise because it adaptively chooses the one-shot selection layer; SqueezeAttention is layer-wise because it assigns different KV budgets to different layers; and LaCo is layer-wise because it inserts compression at an intermediate encoder depth [2606.01756] [2601.07667] [2404.04793] [2507.02279]. “Layer-wise” therefore refers to dependence on depth, not to a single fixed compression topology.

Another misconception is that LTC is synonymous with permanent pruning. ALVTS explicitly challenges that view by showing that only about 3% of tokens are never selected and by routing less important tokens to skip a layer rather than deleting them forever [2606.14277]. This suggests that many tokens are temporarily unimportant rather than globally useless. A related open direction is whether reversible routing, reconstruction operators, or memory-style token revival can systematically outperform irreversible deletion.

The placement of compression remains unresolved. The geometric token-correlation analysis suggests that the initial 10% of layers expand into a working space before contraction begins, while RAPID and LaCo both report that not all depths are equally suitable for compression [2503.22547] [2606.08156] [2507.02279]. This implies that early compression is neither universally good nor universally bad. Instead, the right onset depends on when local features have become stable enough for redundancy removal and when later modules dominate the remaining cost.

Several engineering limitations recur. Methods such as EvoCut require access to intermediate vision-encoder states and therefore cannot be applied to closed-source or API-only models [2606.01756]. Post-encoder methods do not reduce encoder FLOPs, whereas internal methods such as LaCo do, but often at greater optimization difficulty [2507.02279]. Attention-map-based training-free pruning can be awkward to combine with FlashAttention, which is why attention-free or representation-based scores such as Representation Shift matter operationally [2508.00367]. In special-token compression, positional geometry can also be a bottleneck: the designed position identifiers in context compression show that assigning compressed tokens positions uniformly spread across ((1,m)) rather than (m+1,\dots,m+n) can drastically improve reconstruction quality, which suggests that broader LTC systems that replace many tokens with a few representatives may need equally careful positional handling [2409.14364].

Finally, some adjacent methods point toward future LTC directions without being strict internal LTC themselves. (K)-Token Merging and Z-token compression show that variable-length or latent-space compression can preserve substantial downstream utility, which suggests a broader design space in which layer-wise token reduction might be combined with learned semantic bottlenecks, reconstruction-aware supervision, or compressed-space reasoning [2604.15153] [2603.25340]. The field is therefore moving from the narrow question “which tokens should be pruned?” toward a wider one: which information should survive, in what form, at which depth, and with what downstream interface.