---
title: Hierarchical Token Deduplication
url: https://www.emergentmind.com/topics/hierarchical-token-deduplication
type: topic
---

# Hierarchical Token Deduplication

Hierarchical token deduplication encompasses algorithmic frameworks and practical systems that remove or merge semantically redundant tokens in structured, multi-stage or multi-level fashion, typically organized along natural topologies such as hardware hierarchies, model architectures, or input modalities. This approach contrasts with flat, one-shot, or uniform token reduction, targeting both communication/computation minimization and preservation of task-critical information. Hierarchical token deduplication methodologies now play a crucial role in accelerating large-scale mixture-of-experts (MoE) model training, vision/language transformer inference, and streaming video understanding in both academic and industrial settings.

## 1. Foundational Concepts and Motivation

Hierarchical token deduplication finds its roots in inefficiencies caused by token-level redundancy during distributed computation, vision/language fusion, or sequential video processing. 

- In sparse Mixture-of-Experts (MoE) LLMs, each token routes to $K$ experts possibly colocated on the same or across different devices, leading to extraneous transfer of duplicate token data and load imbalances across the compute topology. Token deduplication aims to minimize communication bandwidth by sending each token only once per expert-group or communication partition, especially across slow links (e.g., InfiniBand, QPI) [2508.09591].
- In vision transformers, image and video models, and multimodal LLMs, many tokens (image patches, frame patches, etc.) are spatially/temporally redundant. Progressive, layer-aware deduplication exploits the hierarchical nature of model architectures and data structure to prune non-essential tokens in an information-preserving manner [2409.11923, 2604.01881, 2512.00891, 2602.23699].

The central objective is to select—at each appropriate level—the minimal set of unique, informative tokens sufficient for downstream operations, while adapting to both the data and the hardware or model architecture.

## 2. Formal Models and Algorithmic Strategies

Implementations of hierarchical token deduplication vary across application domains, but share structural commonalities.

**Hierarchical Deduplication in MoE Communication (HierMoE) [2508.09591]:**
- The token dispatch/combination pattern in expert parallel MoE is formalized as a two-stage AlltoAll operation. The system cluster is abstracted as a $D$-level hierarchy (e.g., data center → node → socket → PCIe → GPU), partitioning all $G$ GPUs into expert-groups at each level $U[0]=1, U[1],...,U[D]=G$.
- At each hierarchy stage, duplicate tokens for experts on the same group are removed before inter-group transfer. The process is mathematically modelled via per-level "duplicate-free" token allocation vectors $p^{(i)}\in \mathbb{R}^{U[i]}$, and an overall communication time function:
  $$
  t_d = \sum_{i=1}^{d-1} [\alpha^{\text{Inter}}_i + \beta^{\text{Inter}}_i n^{\text{Inter}}_i] + [\alpha^{\text{Intra}}_{d-1} + \beta^{\text{Intra}}_{d-1} n^{\text{Intra}}_{d-1}]
  $$
  The optimal hierarchy depth $d^*$ is selected to minimize $t_d$ given the current routing and hardware topology.

**Hierarchical Agglomerative Token Clustering (ATC) [2409.11923]:**
- Token embeddings $\{\phi(t_i)\}_{i=1}^N$ are treated as points in feature space. Deduplication proceeds via bottom-up agglomerative clustering, successively merging the most similar token pairs under average, complete, or single linkage, until the desired retention ratio $r$ is met.
- Hierarchical clustering adaptively finds and merges local redundancies before global ones, respecting the semantic content of spatial structure.

**Multi-level Hierarchical Pruning in Video and Multimodal Models:**
- HieraVid [2604.01881] applies segment-level (temporal), frame-level (diversity via DPP), and layer-level (progressive pruning along the transformer stack) deduplication, exploiting both visual and language cues.
- STC [2512.00891] for streaming video LLMs employs a two-stage pipeline: intra-ViT-layer caching of static token features, then inter-frame and intra-frame saliency-weighted token pruning, using both spatial and temporal anchor context for scoring.
- HiDrop [2602.23699] decomposes the vision–language transformer into shallow/propagator, mid/fusion, and deep/language-only blocks. Visual tokens are injected only at the effective fusion layer, and then pruned according to a concave schedule optimized using layer-pair attention similarity metrics.

## 3. Representative Algorithms and Procedural Details

A selection of canonical hierarchical token deduplication pipelines is outlined below:

**HierMoE HierD-AlltoAll:**
1. Compute the routing mask $I_{\text{route}}^{(1,E)}$ indicating token-to-expert assignment.
2. For each hierarchy level $\ell=1...D$, aggregate duplicate tokens per expert-group, simulating token flow after each communication step, and derive the deduplicated token count vectors $p^{(\ell)}$.
3. Evaluate communication cost $t_d$ for possible hierarchy depths; select $d^* = \arg \min t_d$.
4. Execute multi-level AlltoAll, deduplicating tokens across slow communication levels before entering finer grain, lower-latency exchange [2508.09591].

**Agglomerative Token Clustering (ATC):**
- Initialize each token as its own cluster.
- Iteratively merge nearest clusters according to a chosen linkage criterion, recalculating cluster centroids as weighted means (often using attention mass).
- Output $K=\lceil rN\rceil$ clusters, each representing a deduplicated token for downstream encoder layers [2409.11923].

**HieraVid Three-Level Pruning:**
1. Segment-level: partition video frames based on patchwise similarity and merge static patches.
2. Frame-level: for each segment, select tokens using determinantal point process (DPP) maximizing both diversity and relevance to text instructions.
3. Layer-level: divide transformer layers into stages; apply geometric schedule of sparse spatial merges and progressive token dropping.
- Quantitatively, with $30\%$ token retention, HieraVid maintains $98.4\%$ average performance of the unpruned model at $24.5\%$ of original FLOPs on LLaVA-Video-7B [2604.01881].

**STC-Cacher and STC-Pruner:**
- For streaming input, cache per-layer per-token features at each reference frame.
- For non-reference frames, compute similarity and recompute only the lowest-similarity (dynamic) tokens; others are reused from cache, at each transformer layer.
- Before LLM input, compute a convex-combination saliency $S(z_j)$ for each token, reflecting both dissimilarity to "temporal" and "spatial" anchors; retain only the top-$k$ saliency tokens [2512.00891].

## 4. Communication and Computational Complexity Reduction

Hierarchical token deduplication analytically and empirically delivers major gains in bandwidth and compute reduction.

- In HierMoE, deduplication decreases the expected duplication factor on slowest communication links from $K$ to $K/U[1]$ (where $U[1]$ is the number of expert-groups at the highest topology level), yielding up to $50$–$70\%$ reduction in inter-node traffic [2508.09591].
- Agglomerative hierarchical clustering achieves greater preservation of semantic content at aggressive reduction (low retain rate $r$) compared to one-shot or flat merging, with no additional learnable parameters [2409.11923].
- HieraVid and STC demonstrate that hierarchical approaches can reduce transformer FLOPs and wall-clock latency by factors of $2$–$3\times$, while sacrificing under $3\%$ in task accuracy on large-scale video-language understanding [2604.01881, 2512.00891].
- In HiDrop, vision token FLOPs are reduced by nearly $90\%$ with $\geq 96.5\%$ accuracy retention across $11$ benchmarks, enabled by layer-aware pruning and selective injection strategies [2602.23699].

## 5. Empirical Results and Comparative Performance

Reported results underscore the effectiveness of hierarchical token deduplication:

| Paper/Method      | Task Domain     | Main Gains                                                | Token Retention      |
|-------------------|----------------|-----------------------------------------------------------|----------------------|
| HierMoE [2508.09591] | MoE LLM train   | $1.55\times$–$3.32\times$ comm speedup; $1.18\times$–$1.27\times$ E2E speedup     | Aggressive; up to 55% token dedup |
| HieraVid [2604.01881] | VideoLLM        | $2\times$–$4\times$ FLOP reduction, $>98\%$ accuracy at $30\%$ tokens             | Flexible (10–30%)    |
| STC [2512.00891]      | Streaming VideoLLM | $24.5\%$ lower ViT latency, $45.3\%$ lower prefill, $97\%$ accuracy retention   | User-specified       |
| HiDrop [2602.23699]   | MLLM (vision)     | $89$–$92\%$ vision token pruned, $1.72\times$ faster run; $\geq96.5\%$ accuracy | ~10% retained        |
| ATC [2409.11923]      | ViT (vision)      | $\sim2$–$5$ pp accuracy over prior merging at low keep rates; best FID @ $r\leq0.5$ | Any (parametrized)   |

Task-dependent ablation studies consistently indicate the additive nature of hierarchical pruning stages, with layer-level and segment/frame-level deduplication yielding compounded benefits. For both vision and video models, early hierarchical merging boosts diversity preservation compared to flat, global approaches.

## 6. Implementation Considerations and Limitations

Implementation details are highly system- and workload-dependent:

- HierMoE replaces standard AlltoAll collectives with a stack of deduplicated, topology-aware AlltoAlls, using NCCL for communication and custom fusing for buffer management. Deduplication requires $O(D N K)$ computation per layer [2508.09591].
- ATC relies on precomputed similarity matrices and external clustering libraries (CPU-bound for large $N$), highlighting bottlenecks for batch and scale. GPU-native agglomerative routines are necessary for high-throughput pipelines [2409.11923].
- Pruning schedules and budget allocations must be tuned for data domain and compute topology; errant hyperparameter selection or linkage criterion may degrade performance.
- The memory overhead for per-layer statistics and caches (as in STC) may be significant for long streaming input or very deep models [2512.00891].
- In models where information fusion is not staged (or when redundancy is not hierarchically structured), non-hierarchical approaches may still suffice or outperform.

Hierarchical deduplication integrates naturally with auxiliary techniques such as expert-swap in MoE, differential top-$k$ selection in pruning, and dynamic feedback control in video streaming.

## 7. Applications and Research Significance

Hierarchical token deduplication is now integral to efficient distributed LLM training (especially large MoE models), vision transformer acceleration, multimodal and streaming video-large language models.

- Research groups developing distributed LLM systems leverage these methods for bandwidth and end-to-end training time reduction at scale (e.g., HierMoE atop Megatron-LM).
- Multimodal models exploit hierarchy-aligned token processing (e.g., late injection, progressive pruning) for state-of-the-art resource-accuracy trade-offs.
- Practical deployment of video-language models in streaming, latency-constrained, or edge environments depends on such acceleration frameworks to eliminate unnecessary computation without loss of fine-grained prediction quality.

The continuing evolution of hierarchical token deduplication is marked by tighter integration with hardware topology, dynamic scheduling, differentiable selection, and deeper theoretical analysis of redundancy in both model and data.

Source: https://www.emergentmind.com/topics/hierarchical-token-deduplication