Pixel Token Compaction

• Pixel Token Compaction is a set of algorithmic strategies that reorganize, merge, and prune redundant visual tokens from Vision Transformer outputs to reduce computational overhead.
• Techniques such as cluster aggregation, prune-and-merge modules, and token transforming achieve up to 99% FLOPs reduction with negligible accuracy loss.
• These methods enable efficient processing for dense prediction and multimodal tasks on mobile and edge devices while preserving critical spatial-temporal information.

Pixel Token Compaction refers to a spectrum of algorithmic strategies for reorganizing, merging, and selectively pruning pixel-level visual tokens, typically produced by Vision Transformers (ViTs) or hybrid vision-language encoders, with the objective of reducing computational and memory costs while preserving essential semantic and spatial-temporal information. Mechanisms for pixel token compaction operate at the level of patch embeddings, cluster centroids, superpatches, and dynamic token groups, yielding rapid inference and resource-efficient large-scale multimodal processing. This article surveys the methodological landscape, principled design choices, empirical benchmarks, and theoretical implications of pixel token compaction in contemporary research.

1. Foundations and Rationale

Vision Transformers tokenize visual inputs (images or video frames) into fixed-size patch embeddings. The resulting token sequence length $N$ is quadratic in spatial resolution, presenting significant computational barriers for dense prediction, long-horizon video understanding, and multimodal reasoning. Pixel token compaction targets redundancy at the level of patch tokens and seeks to minimize $M \ll N$ with negligible loss of accuracy on downstream tasks (Omri et al., 24 Apr 2025, Mao et al., 30 Mar 2025, Szczepanski et al., 17 Sep 2025, Zhang et al., 4 Jun 2025). Key motivations include:

• Quadratic scaling of self-attention: Reducing token count directly lowers FLOPs from $O(N^2d)$ to $O(M^2d)$, where $d$ is the embedding dimension.
• High correlation in pixel patches: Empirical studies demonstrate that many ViT/CLIP patch embeddings exhibit near-duplicate information, enabling aggressive compaction (Omri et al., 24 Apr 2025).
• Mobile and edge deployment: Memory and inference latency constraints necessitate adaptive, hardware-compatible compression modules (Mao et al., 30 Mar 2025).

2. Core Methodologies for Pixel Token Compaction

2.1 Cluster Aggregation

Cluster-based aggregation applies k-means++ clustering to token embeddings $X = \{x_1, ..., x_N\}$, forming $M$ clusters $\mu_1,...,\mu_M$ (Omri et al., 24 Apr 2025). Each compressed token $x'_j$ is the mean or (optionally) weighted mean of its assigned cluster:

$x'_j = \frac{1}{|C_j|} \sum_{i \in C_j} x_i$

Positional embeddings $p_i$ may be aggregated identically to maintain spatial context. Cluster aggregation is training-free and strictly non-lossy at the token level; all input tokens contribute, ensuring coverage and redundancy reduction. It outperforms attention-based saliency selection and importance-pruning in accuracy-per-compute trade-offs.

2.2 Prune-and-Merge Modules

The Prune & Merge strategy wraps each ViT block with a trainable merge matrix $\mathbf{M} \in \mathbb{R}^{M \times N}$ that implements both token pruning and merging. A reconstruct matrix $\mathbf{R} = \mathbf{M}^+$ restores spatial layout post-processing (Mao et al., 30 Mar 2025). Token importance $\mathcal{I}(i)$ is computed via gradient-weighted attention scores during training, enabling layerwise adaptive compaction. Pseudocode for merge matrix generation and global structure search is available in (Mao et al., 30 Mar 2025). Shortcut connections preserve pruned tokens for later recovery.

2.3 Token Transforming (Matrix Formulation)

Token Transforming recasts compaction as explicit matrix multiplication $Y = T X$ with $T \in \mathbb{R}^{M \times N}$ (Zeng et al., 6 Jun 2025). $T$ is constructed to generalize both:

• Pruning (one-to-one selectors)
• Merging (many-to-one block averages)

A many-to-many, soft assignment (column- and row-normalized) is built using attention/local similarity. Training-free and hardware-friendly, it achieves $>40\%$ FLOPs reduction and subpercent accuracy drop across ViT backbones, segmentation models, and vision-language pipelines.

2.4 Dynamic Patch Merging and Early Pruning (STEP)

The STEP framework employs a policy net (EfficientNet-Lite0) to decide, for each spatial window, whether to merge patches into superpatches, based on similarity thresholds $\tau$ (Szczepanski et al., 17 Sep 2025). Early-exit pruning further halts high-confidence tokens at intermediate encoder layers, using auxiliary classifier heads and softmax gating. Both mechanisms yield up to $4\times$ reduction in computational complexity, $75\%$ token drop, and control inference speed and accuracy trade-offs.

2.5 Extreme Token Reduction for Video (Token Dynamics)

Token Dynamics introduces object-level clustering and vector-quantization applied to video tokens, achieving token ratios as low as $0.07\%$ of the original sequence (Zhang et al., 21 Mar 2025). Separate hash tables and spatial-temporal key maps maintain positional coherence. A cross-dynamics attention mechanism integrates motion features into the concise token base without increasing token count. This methodology supports fixed-length and adaptive compression subtasks for long-horizon video LLMs.

2.6 Progressive Visual Token Compression (PVC)

PVC unifies image and video token compaction with progressive (causal) spatial-temporal encoding and framewise, AdaLN-conditioned adaptive compression (Yang et al., 2024). Images are repeated as static videos; each frame is compressed to a hard token budget using PixelShuffle pooling and conditioned MLP. The progressive sparse attention mechanism ensures each frame accumulates new spatial or temporal details, exploiting temporal redundancy.

3. Theoretical Analysis and Computational Complexity

All compaction modules exploit the quadratic scaling of self-attention:

$\text{ViT FLOPs:} \quad O(N^2d)$

$\text{Post-compaction:} \quad O(M^2d)$

With typical $r = M/N$ in $[0.07, 0.7]$, FLOPs reduction ranges from $80\%$ to $99\%$ depending on strategy and hardware (Mao et al., 30 Mar 2025, Zeng et al., 6 Jun 2025, Szczepanski et al., 17 Sep 2025, Zhang et al., 21 Mar 2025). Matching and merging overheads are typically negligible; for example, $Prune$ at $r=0.7$ yields a $1.55\times$ speedup and $-34.8\%$ FLOPs on DeiT-small, with $<0.2\%$ accuracy loss (Mao et al., 30 Mar 2025). Token Dynamics at $r=0.0007$ doubles throughput for video LLMs (Zhang et al., 21 Mar 2025).

4. Empirical Benchmarks and Comparative Results

Benchmark studies consistently demonstrate that cluster aggregation and joint prune-merge approaches outperform vanilla pruning, saliency methods, and random or spatial sampling:

Method	Accuracy (%)	Tokens Retained (%)	FLOPs Reduction (%)
Cluster Aggregate (Omri et al., 24 Apr 2025)	$68.96$ (SQA)	$15$	$88$
Token Dynamics (Zhang et al., 21 Mar 2025)	$57.72$ (NextQA-MC)	$0.07$	$>$99
Prune&Merge (Mao et al., 30 Mar 2025)	$1.64\times$ speed	$60$	$41.3$
STEP (Szczepanski et al., 17 Sep 2025)	$75.7 \rightarrow 73.8$ (mIoU)	$25$	$4\times$
PVC (Yang et al., 2024)	$80.0$ (TextVQA)	$25$ (image)	$75$
Token Transforming (Zeng et al., 6 Jun 2025)	$79.8\rightarrow79.7$	$60$	$43$

Cluster aggregation additionally achieves $1$-$3\%$ accuracy gains over VisionZip and other SoTA methods across multimodal benchmarks (Omri et al., 24 Apr 2025). PVC preserves fine-grained detail in detail-sensitive VQA, segmentation, and video classification (Yang et al., 2024). STEP maintains mIoU within $2$ percentage points for 4$\times$ compaction (Szczepanski et al., 17 Sep 2025).

5. Extensions: Dense Prediction, Multimodal, and Video Processing

Compaction strategies are highly modular. Dense prediction (segmentation, detection, depth estimation) is accommodated by "un-transforming" compressed tokens via nearest-neighbor assignment for per-pixel or bounding box heads (Zeng et al., 6 Jun 2025, Mao et al., 30 Mar 2025). Multimodal processing (e.g., LLaVA pipelines) compacts only the visual stream, leaving language tokens uncompressed (Zeng et al., 6 Jun 2025, Omri et al., 24 Apr 2025). Progressive compaction mechanisms enable unified handling of images and videos under a strict per-frame token budget (Yang et al., 2024). The cross-dynamics attention paradigm is suited for extreme video LLM contexts (Zhang et al., 21 Mar 2025).

6. Practical Trade-offs, Limitations, and Future Directions

Notable practical design choices and trade-offs include:

• Trade-off between compression ratio and accuracy: Aggressive compaction (e.g., $r < 0.1$) achieves massive compute savings but risks missing fine spatial context—object-level clustering and cross-modal attention augmentation can mitigate this (Zhang et al., 21 Mar 2025, Omri et al., 24 Apr 2025).
• Saliency-based selection limitations: Raw attention-based importance scoring is unreliable due to prompt-insensitivity and volatility across layers (Omri et al., 24 Apr 2025).
• Layerwise adaptive compaction: Insertion/removal of compression modules per-layer enables granular control of throughput and context preservation (Mao et al., 30 Mar 2025).
• Hardware and parallelism compatibility: All methods report hardware-specific speed-ups (NVIDIA H100, LLaMA2-7B), confirming practical scalability.

Future developments outlined include learnable or hierarchical clustering for arbitrary sequence length, extension of compaction strategies to video/audio frames, integration of co-attention for multimodal saliency, and quantification of energy/memory savings in green AI deployments (Omri et al., 24 Apr 2025). PVC and Token Dynamics frameworks suggest that causal and progressive compaction may become standard in unified multimodal transformers (Yang et al., 2024, Zhang et al., 21 Mar 2025).

• (Omri et al., 24 Apr 2025) Token Sequence Compression for Efficient Multimodal Computing
• (Mao et al., 30 Mar 2025) Efficient Token Compression for Vision Transformer with Spatial Information Preserved
• (Zeng et al., 6 Jun 2025) Token Transforming: A Unified and Training-Free Token Compression Framework for Vision Transformer Acceleration
• (Szczepanski et al., 17 Sep 2025) Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
• (Zhang et al., 21 Mar 2025) Token Dynamics: Towards Efficient and Dynamic Video Token Representation for Video LLMs
• (Yang et al., 2024) PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-LLMs
• (Zhang et al., 4 Jun 2025) DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
• (Wei et al., 2023) Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers