Dynamic Hierarchical Merging

• Dynamic hierarchical merging is a computational paradigm that recursively combines elementary units or model components using adaptive, similarity-driven measures.
• It is widely applied in deep learning, graph algorithms, image segmentation, and physical simulations to enhance efficiency and scalability.
• The approach balances computational efficiency with accuracy through adaptive pruning and dynamic scheduling, while addressing challenges in optimality and interpretability.

Dynamic hierarchical merging refers to a broad class of algorithmic and modeling paradigms in which entities, structures, or representations are recursively and adaptively combined in a tiered, often tree-structured fashion, with merging decisions and operations varying as a function of local data, task requirements, and hierarchical context. Deployed across disparate fields—including deep learning, graph algorithms, computer vision, high-dimensional clustering, and physical simulations—dynamic hierarchical merging exploits recursive partitioning, selective aggregation, adaptive pruning, and dynamic scheduling to achieve computational scalability, representation efficiency, or domain-specific performance guarantees.

1. Formal Definition and Scope

A dynamic hierarchical merging algorithm operates by (i) decomposing the data, model, or environment into elementary units (tokens, regions, models, clusters, etc.), (ii) recursively merging pairs or groups of these units at multiple, progressively coarser, levels, and (iii) adaptively determining merging actions and criteria as a function of local representations, similarity, or optimization objectives, often under runtime or memory constraints. The operational nature of "dynamic" here specifies that merging steps, groupings, or reductions are not statically predetermined but depend on intermediate results, ongoing measurements, or auxiliary optimization (e.g., attention scores, clustering outputs, model similarity, etc.).

Underlying this paradigm is a hierarchical structure: the merging process typically defines a tree or DAG, with leaves representing elementary units and internal nodes corresponding to composite entities produced by previous merges. The depth and width of the hierarchy, as well as the specific merging rules at each level, are typically data-, context-, or task-dependent.

2. Domain-Specific Methodologies

2.1. Neural Context and Token Merging

For transformer architectures, dynamic hierarchical merging is exemplified by schemes such as HOMER (Song et al., 2024), DyTo (Zhang et al., 2024), and MergeDNA (Li et al., 17 Nov 2025):

• HOMER (Hierarchical cOntext MERging): Long input sequences are divided into context chunks, each processed independently by early layers. At each successive tree level, pairs of adjacent chunks undergo a data-driven token reduction (using an attention-based significance score), are concatenated, and further processed together. This process is recursively applied, yielding a computational and memory scaling of $O(L\,C\,M\,\log(N/C))$ rather than $O(LN)$, with $L$ the depth, $C$ the maximal chunk size, and $M$ per-token storage. All merging steps are dynamically determined based on intermediate attention activations and recursively propagated pruning masks. Memory and accuracy improvements are empirically demonstrated over long-range benchmarks.
• DyTo for Video Understanding: Combines a hierarchical (connected components in 1-NN CLS-embedding graph) frame selection, reducing a sequence of $N$ frames to $K\ll N$ key frames, with a per-frame, bipartite maximum-similarity token merging down to a global budget $Z$. All merging operations—frame selection and token reduction within frames—are guided by adaptive, similarity-based selections, not static assignments. This enables efficient and faithful zero-shot video processing by maintaining diversity and concentrating representation on semantically salient subregions (Zhang et al., 2024).
• MergeDNA (Genomics): Employs a multi-layer, local-window-constrained token merging module to learn variable-length "DNA words" by dynamically fusing tokens in small context windows, stacking multiple such layers to build up a hierarchy of motifs. Global token merging is applied downstream. Each merging phase is data-driven, with pairs selected by similarity measures computed within local context; the overall merging schedule adapts to the information density and task structure of the underlying DNA sequence (Li et al., 17 Nov 2025).

2.2. Model and Adapter Merging

• Hierarchical Re-Basin: In model consolidation, dynamic hierarchical merging structures the merging of $N = 2^K$ models as a balanced tree: at each stage, pairs of models are permutation-aligned (Re-Basin) and then weights are interpolated. At every merge, Re-Basin matching is dynamically solved based on current parameter statistics, and the hierarchy enables implicit regularization and robustness, outperforming direct sum-averaging on both adversarial and clean accuracy (Franke et al., 10 Oct 2025).
• Hierarchical Adapter Merging (HAM): For continual learning with efficient parameter usage, task adapters—parameter-efficient, low-rank updates—are dynamically grouped based on cosine similarity in the parameter space. When a group exceeds thresholds, selective pruning and adaptive weighted averaging produce a new, consolidated representation. Grouping and merging are dynamically controlled according to the evolution of the task sequence and similarity structure, ensuring both continual adaptation and controlled parameter explosion (Coleman et al., 16 Sep 2025).
• Merging Sparse Mixture-of-Experts (HC-SMoE): Experts are clustered based on average output features measured over diverse data, via agglomerative hierarchical clustering with average linkage. The hierarchy is dynamically constructed from inter-expert distances and can be cut at any desirable point. For each resulting cluster, expert weights are merged, and the routing mechanism exploits the merged structure for memory efficiency (Chen et al., 2024).
• Hierarchical Multi-Objective Merging (HM3): In multi-model multi-objective settings, dynamic hierarchical merging is realized as an RL-based search through a space of model architectures and parameter-space combined models, conditioned on user task preferences. At each inference path, the merging policy dynamically chooses model-layer transitions, using a PPO-trained actor-critic with additional per-layer learnable transformations (Zhou et al., 2024).

2.3. Graphs, Clustering, and Physical Systems

• Merging HNSW Graphs: Three dynamic hierarchical merging algorithms (NGM, IGTM, CGTM) consolidate multi-level HNSW graphs by iteratively selecting unprocessed vertices, collecting candidates via graph traversal or local search, constructing neighborhoods based on relative proximity, and determining the next candidates dynamically. Traversal order and candidate sets in IGTM/CGTM are selected adaptively to reduce global distance computations by up to 70% while preserving search accuracy (Ponomarenko, 21 May 2025).
• Hierarchical Co-Clustering (LAMC): Parallelizes large-scale co-clustering by probabilistically partitioning the input matrix into subblocks, running co-clustering in parallel, and hierarchically merging atomic clusters using an iterative similarity-maximizing dynamic merge routine (union of clusters with large overlaps), with merge thresholds and progress regulated at runtime. Both partitioning schemes and merging hierarchies are constructed adaptively to balance computational cost and recovery of underlying structure (Wu et al., 2024).
• Region Merging for Image Segmentation: Starting from primitive superpixels, dynamic region merging (DRM) iteratively and hierarchically merges neighboring image regions. Merge predicates are data- and context-sensitive: only mutual nearest-neighbor pairs passing both a similarity threshold and a sequential probability ratio test (likelihood-ratio-based homogeneity assessment) are merged. The process proceeds in rounds, dynamically grouping and stopping according to accumulated evidence, which can be mathematically shown to yield optimality properties related to global consistency (Peng et al., 2010).
• Astrophysics (Hierarchical Star Cluster Formation): In N-body simulations of young star clusters, subcluster (clump) structure is not erased primarily by clump–clump mergers, but via hierarchical and dynamic two-body relaxation—stars are scattered out of their initial associations, populating higher levels of a nascent hierarchical structure, ultimately yielding a smooth spherical distribution. The efficiency is set by the clump internal relaxation time and virial ratio, both governing the effective merging rate parameter $\eta$ (Smith et al., 2011).

3. Key Algorithmic Principles

Dynamic hierarchical merging frameworks are characterized by multiple shared algorithmic principles:

• Divide-and-Conquer + Hierarchical Recursion: Problems are subdivided into independent or semi-independent units, processed in parallel or sequentially at the leaves, and successively merged in a hierarchical fashion (binary, multiway, or variable branching).
• Adaptive Pruning or Reduction: At each level, merging involves selective pruning, aggregation, or compression, driven by learned or data-driven measures (e.g. attention, similarity, likelihood ratio, or model parameter alignment).
• Similarity-Driven or Data-Dependent Scheduling: Merging order, grouping, or path selection is geared by dynamic measurements (cosine similarity, distance, statistical tests, RL-based policy outputs), not static structure.
• Resource-Aware Processing: Hierarchical order and merging granularity are set to balance memory, communications, or computational cost (e.g., logarithmic memory scaling in HOMER), and merging policies may be modified dynamically according to online resource feedback.
• Stopping and Optimality Conditions: Merging halts upon satisfaction of certain criteria—no further similar neighbors (image segmentation), negligible gain in similarity (graph merging), or user-specified approximation or resource thresholds.

4. Comparative Algorithmic Table

Domain	Elementary Units	Merge Criterion	Dynamicity/Hierarchy	Key Reference
Transformer LLMs	Context chunks/tokens	Attention-based score, importance mask	Layer-wise, tree recursion	(Song et al., 2024)
MoE Models	Experts	Output similarity (avg-link clustering)	Dendrogram, offline merge	(Chen et al., 2024)
Adapter Merging	Task adapters	Cosine similarity in param. space	Grouping, hierarchical pool	(Coleman et al., 16 Sep 2025)
HNSW Graphs	Graph vertices	Proximity in metric space/neighbor sets	Traversal-order hierarchy	(Ponomarenko, 21 May 2025)
Co-Clustering	Submatrix co-clusters	Jaccard/overlap similarity	Iterative cluster merging	(Wu et al., 2024)
Image Segmentation	Superpixels/regions	Mutual min-similarity, homogeneity SPRT	Parallel sequential merges	(Peng et al., 2010)
Star Cluster Sim.	Stellar clumps	Scattering/relaxation, GTM thresholds	Bottom-up smoothing	(Smith et al., 2011)

5. Theoretical and Practical Implications

Dynamic hierarchical merging supports computational scalability ($O(\log N)$ memory scaling for long LLM contexts (Song et al., 2024)), robust consolidation of knowledge or resources (adapter and model merging (Coleman et al., 16 Sep 2025, Franke et al., 10 Oct 2025)), improved sparsity and efficiency in large models (expert clustering (Chen et al., 2024)), and domain-optimal integration of complex data (image, video, graph, or co-clustering tasks).

In deep learning, these mechanisms enable retraining-free adaptation, continual learning with controlled parameter growth, and efficient deployment. Hierarchical model merging strategies (e.g. Re-Basin, cosine-OT interpolation) have been empirically shown to deliver superior robustness and only modest performance trade-offs compared to naive averaging or complex pruning, as in medical LLMs (Timilsina et al., 17 Nov 2025) and multi-objective benchmarks (Zhou et al., 2024).

Rapid, application-driven adaptation is a direct consequence of dynamic scheduling: whether the problem calls for memory-efficient long-context reasoning (HOMER), ultra-fast graph index compaction (IGTM/CGTM for HNSW), or scalable model fusion in resource-limited IoT or healthcare settings (cosine-OT interpolation), the system can tailor its merging strategy to specific runtime signals and accuracy constraints.

6. Limitations and Open Challenges

While dynamic hierarchical merging generally leads to efficient and adaptive aggregation, several enduring challenges arise:

• Trade-offs between approximation and optimality: Aggressive pruning or groupwise merging can induce information loss; optimal settings of merging depth, pruning thresholds, or similarity measures are usually task- and data-dependent.
• Resource scheduling: In online or streaming contexts, dynamic hierarchical merging must avoid degenerate or unbalanced merges that may bottleneck lower levels or introduce load disparities (especially in distributed graph or co-cluster merges).
• Semantic or interpretability constraints: In domains where semantic structure is crucial (e.g., genomic modeling or image region merging), purely similarity-driven dynamic merges may occasionally overmerge or split functionally distinct units, requiring domain-informed correction or constraint.
• Extending to non-binary/data-fusion scenarios: Many algorithms are tree or pairwise by design; more general DAG hierarchies or multi-way merges may yield improved efficiency but add algorithmic complexity.
• Adversarial robustness: For model merging (especially in adversarial settings), the extent to which hierarchical mixing induces or degrades robust generalization is still under empirical and theoretical investigation (Franke et al., 10 Oct 2025).

7. Representative Applications and Empirical Performance

Across application domains, dynamic hierarchical merging achieves state-of-the-art or near best-in-class results:

• In transformers, HOMER enables long-context inference up to 64k tokens while reducing peak memory requirements by 73% and matching or exceeding SOTA performance in context-dependent tasks (Song et al., 2024).
• In model merging, hierarchical interpolation with attention-head OT alignment provides robust consolidation nearly matching the best simple interpolations, while outperforming all complex pruning-based merges for medical LLMs (Timilsina et al., 17 Nov 2025).
• For expert reduction, HC-SMoE enables up to 50% parameter reduction with negligible loss in zero-shot accuracy, outperforming all retraining-free pruning baselines (Chen et al., 2024).
• In large-scale co-clustering, LAMC attains up to 83% runtime reduction on dense datasets without loss in NMI/ARI, leveraging its probabilistic dynamic partitioning and hierarchical merging (Wu et al., 2024).

This diversity of settings attests to the generality and power of dynamic hierarchical merging as a unifying computational paradigm for scalable, adaptive, and resource-efficient integration across both symbolic and subsymbolic domains.