AiluRus: Token Reduction via Adaptive Clustering

• Token reduction via adaptive clustering aggregates redundant tokens into representative clusters, reducing computational and memory complexity.
• It employs density-based methods like density-peaks clustering and k-means to adaptively process intermediate feature representations in vision and multimodal models.
• This approach enhances efficiency and speed while maintaining or even improving prediction accuracy across applications such as image prediction, video-language models, and remote sensing.

Token Reduction via Adaptive Clustering (AiluRus) refers to a family of approaches for drastically reducing the effective sequence length of vision and multimodal transformer models by replacing spatially and semantically redundant tokens with a compact set of representative cluster tokens. Leveraging density- or similarity-driven clustering in feature space, these techniques operate adaptively at the intermediate representation level—whether within image, video, or general visual encoders—thereby reducing both computational and memory complexity, mitigating quadratic scaling, and often enhancing downstream prediction or retrieval accuracy. AiluRus-style modules are now established in domains ranging from dense image prediction and high-resolution remote sensing to video-language modeling and diffusion generation.

1. Core Principles and Mathematical Foundations

AiluRus techniques are predicated on the explicit identification and aggregation of tokens that are either feature-homogeneous or informationally redundant. The canonical operational flow consists of four stages:

1. Feature Extraction: At a specified layer of the backbone (e.g., ViT, CLIP, ResNet), obtain a sequence of $N$ tokens, each a $d$-dimensional embedding.
2. Clustering: Use a content-adaptive clustering algorithm—typically a variant of density-peaks clustering (DPC) or k-means—to select $M \ll N$ cluster centers and partition remaining tokens by minimum feature-space distance or affinity to these centers.
   • Density-peaks clustering estimates a local density for each token $x_i$:

   $$\rho_i = \exp\left(-\frac{1}{k}\sum_{j \in \mathrm{kNN}(x_i)} \|x_i - x_j\|^2\right)$$

• A "distance to higher-density neighbors" $\delta_i$ is computed:

  $$\delta_i = \min_{j: \rho_j > \rho_i} \|x_i - x_j\| \quad \text{(or max if none exists)}$$

• The peak score $\gamma_i = \rho_i \delta_i$ is used to select cluster centers (Xie et al., 2022, Li et al., 2023).

1. Token Aggregation: For each cluster, a (possibly attention- or importance-weighted) representative token is computed by averaging or summing features, often with trainable or predicted weights:

$h_j = \sum_{x_i \in G_j} w_i x_i$

where $w_i$ can be a learned or attention-derived scalar.

2. Sequence Replacement and Attention: Subsequent modules process the $d$0-token sequence, with attention layers adjusted appropriately. Some AiluRus variants modify the denominator of attention to account for the cluster size, ensuring correct weighting of merged tokens (Li et al., 2023).

Extensions, such as multi-scale clustering, concatenation of coarse and fine cluster representatives, or the use of cross-scale attention further enhance the representative signal and trade off spatial detail for computational savings (Xie et al., 2022, Li et al., 2023).

2. Algorithmic Instantiations and Variations

Implementations of AiluRus modules diverge primarily in the definition and scheduling of clustering and how cluster counts are determined:

• Density-Peaks Clustering (DPC): Used in ClusTR, TCFormer, AiluRus itself, and similar, DPC efficiently produces content-adaptive clusters by leveraging kNN distances and local density criteria (Xie et al., 2022, Zeng et al., 2024, Li et al., 2023).
• K-means, k-medoids, Spectral Clustering: For token sets where the DPC cost is prohibitive, or where deterministic center selection is desired, k-means clustering (in ClusCa for diffusion (Zheng et al., 12 Sep 2025) and CenterCLIP (Zhao et al., 2022)) or spectral clustering (CenterCLIP, segment-level video token reduction) are used with analogous objective formulations.
• Locality Constraints and Spatial Bias: Spatially-aware variants incorporate location metadata into clustering, either by biasing feature distances or by restricting allowable assignments within a local window (AiluRus spatial-aware DPC, SCSA (Li et al., 13 Apr 2026)).
• Progressive and Multi-Stage Merging: Hierarchical clustering and merging after each transformer stage (TCFormer (Zeng et al., 2022, Zeng et al., 2024)) yields a pyramidal progression from high token counts (fine spatial granularity) to compact global context, culminating in multi-scale aggregation heads.
• Adaptive Cluster Count: Some variants statically fix reduction ratios per stage; others dynamically adapt the number of clusters per sample or segment based on measured importance, redundancy, or an explicit router (DualComp (Li et al., 13 Apr 2026), PruMerge (Shang et al., 2024)).

3. Domain-Specific Adaptations

AiluRus-style modules have been adopted and extended across a wide spectrum of applications:

Domain	Clustering/Efficiency Modifications	Representative Results/Metrics
Image Dense Prediction	Spatial-aware DPC, reweighted self-attention	Up to 48% FPS increase, <0.1 mIoU drop (Li et al., 2023)
Human-Centric Analysis	Progressive DPC-kNN, token-shaped clusters	+3.7% AP (COCO-WB); state-of-art NME (Zeng et al., 2022, Zeng et al., 2024)
Video LLMs	Multi-segment medoid/k-means, spectral	35% mem, 14% speedup, +2% retrieval (Zhao et al., 2022, Wang et al., 5 Aug 2025)
Multimodal LLMs	CLS-attention pruning + kNN merge	18× token reduction, -6.5% VQA accuracy (Shang et al., 2024)
Diffusion Generators	Per-timestep k-means, feature caching	92% token reduction, up to 6.2× speedup (Zheng et al., 12 Sep 2025)
Remote Sensing	Size-adaptive, cluster scoring by [CLS]	42× compression, accuracy ↑ (semantic+geo) (Li et al., 13 Apr 2026)

Algorithmic details often include training-free operation (e.g., post-hoc clustering), integration at specific intermediate layers, and application to both training and inference. Some variants optimize for memory, others for wall-clock speed or FLOP count, with measured metrics directly tied to chosen cluster count and implementation specifics.

4. Complexity Analysis and Scaling Behavior

The principal computational advantage of AiluRus methods arises from reducing the $d$1 cost of dense self-attention to $d$2 or $d$3, with $d$4 the number of clusters:

• Clustering Overhead: DPC and k-means operate at $d$5 or $d$6; local patch partitioning ameliorates the scaling to $d$7 (P=number of spatial blocks).
• Self-Attention Savings: After token reduction, each attention block costs $d$8; at $d$9, the reduction is ∼16× per block.
• End-to-End Gains: Image segmentation with AiluRus achieves up to 48% FPS gains and 2.5× training speedup with negligible performance drop (Li et al., 2023). ClusCa achieves 4.96× FLOPs reduction and $M \ll N$0 ImageReward (Zheng et al., 12 Sep 2025). DualComp in remote sensing yields a $M \ll N$1 effective compression (Li et al., 13 Apr 2026).

Ablation studies confirm that extremely aggressive token reduction can degrade fine boundary or small-part accuracy; optimal $M \ll N$2 typically lies between $M \ll N$3 and $M \ll N$4 depending on the application.

5. Empirical Performance and Behavioral Patterns

A spectrum of studies across AiluRus-type methods demonstrate:

• Negligible drop—often an increase—in key task metrics versus dense or grid-token baselines, provided reduction is not too severe (Li et al., 2023, Zeng et al., 2022, Zeng et al., 2024).
• Clustering-based token selection outperforms naive spatial pooling, uniform sampling, or fixed grid reductions, especially in highly nonuniform and structured visual contexts (Zhao et al., 2022, Shang et al., 2024).
• Dense prediction benefits from preserving token granularity around boundaries and high-information regions, consistent with token-importance or attention-driven selection (Li et al., 2023, Li et al., 13 Apr 2026).
• Token redundancy in video, and even in iterative diffusion steps, can be exploited for order-of-magnitude FLOPs reductions with minimal or no observable performance degradation (Wang et al., 5 Aug 2025, Zheng et al., 12 Sep 2025).
• For MLLMs and remote-sensing, instruction- or class-conditional routing for dynamic allocation between semantic and geometric clusters yields substantial efficiency and accuracy gains over uniform token reduction (Li et al., 13 Apr 2026).

6. Limitations, Extensions, and Theoretical Insights

Limitations of current AiluRus-type modules include:

• Clustering Overhead: For extremely large $M \ll N$5, clustering itself is nontrivial; approximations (local CTM, approximate KNN, hashing) are effective in mitigating prohibitive cost (Li et al., 2023, Zeng et al., 2022).
• Boundary Sensitivity: Hard assignments may induce artifacts for spatially adjacent but semantically distinct regions; soft or differentiable clustering (e.g., DIFFPOOL-style) is a plausible extension (Zeng et al., 2022).
• Dynamic Adaptivity: Static reduction ratios can be suboptimal for nonstationary or highly anomalous inputs; adaptive/learned clustering schedule or per-sample routing (as in DualComp) enhances flexibility (Li et al., 13 Apr 2026).
• Tasks Requiring Fine Granularity: Overaggressive reduction may irreparably damage pixel/point-wise prediction tasks or very fine object structures, requiring fallback to higher cluster counts (Shang et al., 2024, Li et al., 2023).

Open extensions and research directions include soft and differentiable clustering for end-to-end learnable architectures, cross-scale cluster attention, integration with quantization or pruning, and principled routing between semantic (object-centric) and spatial (contextual/topological) features (Li et al., 2023, Li et al., 13 Apr 2026).

7. Connections and Comparative Landscape

AiluRus and related adaptive token clustering mechanisms draw direct lineage from and extend:

• Sparse Attention and Token Pruning (DynamicViT, PnP-DETR): Where the focus is on discarding low-salience tokens, AiluRus instead replaces groups of redundant tokens by abstracted cluster tokens, maintaining coverage of high- and low-information regions (Zeng et al., 2022).
• Multi-Scale ViTs: Classic image pyramids use fixed spatial layouts for resolution; AiluRus modules provide content-aware, region-adaptive scaling, outperforming grid-based methods in dense prediction and compactness (Xie et al., 2022, Zeng et al., 2024).
• Temporal and Video Compression: Segment-level and hierarchical clustering in CenterCLIP and AFP directly address both spatial and temporal redundancy, providing stronger alignment and semantic recall than frame sampling or pooling (Zhao et al., 2022, Wang et al., 5 Aug 2025).
• Token-Efficient Diffusion: Cluster-driven feature caching (ClusCa) is orthogonal to temporal feature reuse and enables unprecedented speedups with stable generation FID (Zheng et al., 12 Sep 2025).

Benchmark comparisons show that token clustering consistently yields better speed, memory, and accuracy trade-offs than uniform downsampling, random token dropping, or deterministic pruning, corroborating its role as a foundational primitive for scalable vision and multimodal transformers.