Hierarchical Token Deduplication

• Hierarchical token deduplication aims to reduce semantically redundant tokens in large-scale models.
• It utilizes natural topologies such as hardware hierarchies for efficient computation and communication.
• This technique accelerates tasks like mixture-of-experts model training, vision transformer inference, and video understanding.

Hierarchical token deduplication encompasses algorithmic frameworks and practical systems that remove or merge semantically redundant tokens in structured, multi-stage or multi-level fashion, typically organized along natural topologies such as hardware hierarchies, model architectures, or input modalities. This approach contrasts with flat, one-shot, or uniform token reduction, targeting both communication/computation minimization and preservation of task-critical information. Hierarchical token deduplication methodologies now play a crucial role in accelerating large-scale mixture-of-experts (MoE) model training, vision/language transformer inference, and streaming video understanding in both academic and industrial settings.

1. Foundational Concepts and Motivation

Hierarchical token deduplication finds its roots in inefficiencies caused by token-level redundancy during distributed computation, vision/language fusion, or sequential video processing.

• In sparse Mixture-of-Experts (MoE) LLMs, each token routes to $K$ experts possibly colocated on the same or across different devices, leading to extraneous transfer of duplicate token data and load imbalances across the compute topology. Token deduplication aims to minimize communication bandwidth by sending each token only once per expert-group or communication partition, especially across slow links (e.g., InfiniBand, QPI) (Lin et al., 13 Aug 2025).
• In vision transformers, image and video models, and multimodal LLMs, many tokens (image patches, frame patches, etc.) are spatially/temporally redundant. Progressive, layer-aware deduplication exploits the hierarchical nature of model architectures and data structure to prune non-essential tokens in an information-preserving manner (Haurum et al., 2024, Guo et al., 2 Apr 2026, Wang et al., 30 Nov 2025, Wu et al., 27 Feb 2026).

The central objective is to select—at each appropriate level—the minimal set of unique, informative tokens sufficient for downstream operations, while adapting to both the data and the hardware or model architecture.

2. Formal Models and Algorithmic Strategies

Implementations of hierarchical token deduplication vary across application domains, but share structural commonalities.

Hierarchical Deduplication in MoE Communication (HierMoE) (Lin et al., 13 Aug 2025):

• The token dispatch/combination pattern in expert parallel MoE is formalized as a two-stage AlltoAll operation. The system cluster is abstracted as a $D$-level hierarchy (e.g., data center → node → socket → PCIe → GPU), partitioning all $G$ GPUs into expert-groups at each level $U[0]=1, U[1],...,U[D]=G$.
• At each hierarchy stage, duplicate tokens for experts on the same group are removed before inter-group transfer. The process is mathematically modelled via per-level "duplicate-free" token allocation vectors $p^{(i)}\in \mathbb{R}^{U[i]}$, and an overall communication time function:

$$t_d = \sum_{i=1}^{d-1} [\alpha^{\text{Inter}}_i + \beta^{\text{Inter}}_i n^{\text{Inter}}_i] + [\alpha^{\text{Intra}}_{d-1} + \beta^{\text{Intra}}_{d-1} n^{\text{Intra}}_{d-1}]$$

The optimal hierarchy depth $d^*$ is selected to minimize $t_d$ given the current routing and hardware topology.

Hierarchical Agglomerative Token Clustering (ATC) (Haurum et al., 2024):

• Token embeddings $\{\phi(t_i)\}_{i=1}^N$ are treated as points in feature space. Deduplication proceeds via bottom-up agglomerative clustering, successively merging the most similar token pairs under average, complete, or single linkage, until the desired retention ratio $r$ is met.
• Hierarchical clustering adaptively finds and merges local redundancies before global ones, respecting the semantic content of spatial structure.

Multi-level Hierarchical Pruning in Video and Multimodal Models:

• HieraVid (Guo et al., 2 Apr 2026) applies segment-level (temporal), frame-level (diversity via DPP), and layer-level (progressive pruning along the transformer stack) deduplication, exploiting both visual and language cues.
• STC (Wang et al., 30 Nov 2025) for streaming video LLMs employs a two-stage pipeline: intra-ViT-layer caching of static token features, then inter-frame and intra-frame saliency-weighted token pruning, using both spatial and temporal anchor context for scoring.
• HiDrop (Wu et al., 27 Feb 2026) decomposes the vision–language transformer into shallow/propagator, mid/fusion, and deep/language-only blocks. Visual tokens are injected only at the effective fusion layer, and then pruned according to a concave schedule optimized using layer-pair attention similarity metrics.

3. Representative Algorithms and Procedural Details

A selection of canonical hierarchical token deduplication pipelines is outlined below:

HierMoE HierD-AlltoAll:

1. Compute the routing mask $D$0 indicating token-to-expert assignment.
2. For each hierarchy level $D$1, aggregate duplicate tokens per expert-group, simulating token flow after each communication step, and derive the deduplicated token count vectors $D$2.
3. Evaluate communication cost $D$3 for possible hierarchy depths; select $D$4.
4. Execute multi-level AlltoAll, deduplicating tokens across slow communication levels before entering finer grain, lower-latency exchange (Lin et al., 13 Aug 2025).

Agglomerative Token Clustering (ATC):

• Initialize each token as its own cluster.
• Iteratively merge nearest clusters according to a chosen linkage criterion, recalculating cluster centroids as weighted means (often using attention mass).
• Output $D$5 clusters, each representing a deduplicated token for downstream encoder layers (Haurum et al., 2024).

HieraVid Three-Level Pruning:

1. Segment-level: partition video frames based on patchwise similarity and merge static patches.
2. Frame-level: for each segment, select tokens using determinantal point process (DPP) maximizing both diversity and relevance to text instructions.
3. Layer-level: divide transformer layers into stages; apply geometric schedule of sparse spatial merges and progressive token dropping.

• Quantitatively, with $D$6 token retention, HieraVid maintains $D$7 average performance of the unpruned model at $D$8 of original FLOPs on LLaVA-Video-7B (Guo et al., 2 Apr 2026).

STC-Cacher and STC-Pruner:

• For streaming input, cache per-layer per-token features at each reference frame.
• For non-reference frames, compute similarity and recompute only the lowest-similarity (dynamic) tokens; others are reused from cache, at each transformer layer.
• Before LLM input, compute a convex-combination saliency $D$9 for each token, reflecting both dissimilarity to "temporal" and "spatial" anchors; retain only the top-$G$0 saliency tokens (Wang et al., 30 Nov 2025).

4. Communication and Computational Complexity Reduction

Hierarchical token deduplication analytically and empirically delivers major gains in bandwidth and compute reduction.

• In HierMoE, deduplication decreases the expected duplication factor on slowest communication links from $G$1 to $G$2 (where $G$3 is the number of expert-groups at the highest topology level), yielding up to $G$4–$G$5 reduction in inter-node traffic (Lin et al., 13 Aug 2025).
• Agglomerative hierarchical clustering achieves greater preservation of semantic content at aggressive reduction (low retain rate $G$6) compared to one-shot or flat merging, with no additional learnable parameters (Haurum et al., 2024).
• HieraVid and STC demonstrate that hierarchical approaches can reduce transformer FLOPs and wall-clock latency by factors of $G$7–$G$8, while sacrificing under $G$9 in task accuracy on large-scale video-language understanding (Guo et al., 2 Apr 2026, Wang et al., 30 Nov 2025).
• In HiDrop, vision token FLOPs are reduced by nearly $U[0]=1, U[1],...,U[D]=G$0 with $U[0]=1, U[1],...,U[D]=G$1 accuracy retention across $U[0]=1, U[1],...,U[D]=G$2 benchmarks, enabled by layer-aware pruning and selective injection strategies (Wu et al., 27 Feb 2026).

5. Empirical Results and Comparative Performance

Reported results underscore the effectiveness of hierarchical token deduplication:

Paper/Method	Task Domain	Main Gains	Token Retention
HierMoE (Lin et al., 13 Aug 2025)	MoE LLM train	$U[0]=1, U[1],...,U[D]=G$3–$U[0]=1, U[1],...,U[D]=G$4 comm speedup; $U[0]=1, U[1],...,U[D]=G$5–$U[0]=1, U[1],...,U[D]=G$6 E2E speedup	Aggressive; up to 55% token dedup
HieraVid (Guo et al., 2 Apr 2026)	VideoLLM	$U[0]=1, U[1],...,U[D]=G$7–$U[0]=1, U[1],...,U[D]=G$8 FLOP reduction, $U[0]=1, U[1],...,U[D]=G$9 accuracy at $p^{(i)}\in \mathbb{R}^{U[i]}$0 tokens	Flexible (10–30%)
STC (Wang et al., 30 Nov 2025)	Streaming VideoLLM	$p^{(i)}\in \mathbb{R}^{U[i]}$1 lower ViT latency, $p^{(i)}\in \mathbb{R}^{U[i]}$2 lower prefill, $p^{(i)}\in \mathbb{R}^{U[i]}$3 accuracy retention	User-specified
HiDrop (Wu et al., 27 Feb 2026)	MLLM (vision)	$p^{(i)}\in \mathbb{R}^{U[i]}$4–$p^{(i)}\in \mathbb{R}^{U[i]}$5 vision token pruned, $p^{(i)}\in \mathbb{R}^{U[i]}$6 faster run; $p^{(i)}\in \mathbb{R}^{U[i]}$7 accuracy	~10% retained
ATC (Haurum et al., 2024)	ViT (vision)	$p^{(i)}\in \mathbb{R}^{U[i]}$8–$p^{(i)}\in \mathbb{R}^{U[i]}$9 pp accuracy over prior merging at low keep rates; best FID @ $$t_d = \sum_{i=1}^{d-1} [\alpha^{\text{Inter}}_i + \beta^{\text{Inter}}_i n^{\text{Inter}}_i] + [\alpha^{\text{Intra}}_{d-1} + \beta^{\text{Intra}}_{d-1} n^{\text{Intra}}_{d-1}]$$0	Any (parametrized)

Task-dependent ablation studies consistently indicate the additive nature of hierarchical pruning stages, with layer-level and segment/frame-level deduplication yielding compounded benefits. For both vision and video models, early hierarchical merging boosts diversity preservation compared to flat, global approaches.

6. Implementation Considerations and Limitations

Implementation details are highly system- and workload-dependent:

• HierMoE replaces standard AlltoAll collectives with a stack of deduplicated, topology-aware AlltoAlls, using NCCL for communication and custom fusing for buffer management. Deduplication requires $$t_d = \sum_{i=1}^{d-1} [\alpha^{\text{Inter}}_i + \beta^{\text{Inter}}_i n^{\text{Inter}}_i] + [\alpha^{\text{Intra}}_{d-1} + \beta^{\text{Intra}}_{d-1} n^{\text{Intra}}_{d-1}]$$1 computation per layer (Lin et al., 13 Aug 2025).
• ATC relies on precomputed similarity matrices and external clustering libraries (CPU-bound for large $$t_d = \sum_{i=1}^{d-1} [\alpha^{\text{Inter}}_i + \beta^{\text{Inter}}_i n^{\text{Inter}}_i] + [\alpha^{\text{Intra}}_{d-1} + \beta^{\text{Intra}}_{d-1} n^{\text{Intra}}_{d-1}]$$2), highlighting bottlenecks for batch and scale. GPU-native agglomerative routines are necessary for high-throughput pipelines (Haurum et al., 2024).
• Pruning schedules and budget allocations must be tuned for data domain and compute topology; errant hyperparameter selection or linkage criterion may degrade performance.
• The memory overhead for per-layer statistics and caches (as in STC) may be significant for long streaming input or very deep models (Wang et al., 30 Nov 2025).
• In models where information fusion is not staged (or when redundancy is not hierarchically structured), non-hierarchical approaches may still suffice or outperform.

Hierarchical deduplication integrates naturally with auxiliary techniques such as expert-swap in MoE, differential top-$$t_d = \sum_{i=1}^{d-1} [\alpha^{\text{Inter}}_i + \beta^{\text{Inter}}_i n^{\text{Inter}}_i] + [\alpha^{\text{Intra}}_{d-1} + \beta^{\text{Intra}}_{d-1} n^{\text{Intra}}_{d-1}]$$3 selection in pruning, and dynamic feedback control in video streaming.

7. Applications and Research Significance

Hierarchical token deduplication is now integral to efficient distributed LLM training (especially large MoE models), vision transformer acceleration, multimodal and streaming video-LLMs.

• Research groups developing distributed LLM systems leverage these methods for bandwidth and end-to-end training time reduction at scale (e.g., HierMoE atop Megatron-LM).
• Multimodal models exploit hierarchy-aligned token processing (e.g., late injection, progressive pruning) for state-of-the-art resource-accuracy trade-offs.
• Practical deployment of video-LLMs in streaming, latency-constrained, or edge environments depends on such acceleration frameworks to eliminate unnecessary computation without loss of fine-grained prediction quality.

The continuing evolution of hierarchical token deduplication is marked by tighter integration with hardware topology, dynamic scheduling, differentiable selection, and deeper theoretical analysis of redundancy in both model and data.