Prune-then-Merge: Simplification Before Fusion

• Prune-then-Merge is a strategy that removes low-importance parameters, embeddings, or activations before merging to reduce conflicts and preserve domain-specific signals.
• It applies to various settings such as delta merging in LLMs, visual document retrieval, and multimodal safety, each utilizing tailored pruning to enhance fusion quality.
• Empirical evidence shows that pruning before fusion can improve metrics like domain preservation and compression efficiency, making the method effective under specific deployment constraints.

Searching arXiv for the cited works and closely related "prune-then-merge" papers. Prune-then-Merge denotes an operation-ordering principle in which a model, representation set, or activation stream is first simplified and only then fused into a consolidated object. In the recent literature, the principle appears in several technically distinct forms: pruning domain deltas before model merging in LLMs, adaptive pruning of patch embeddings before hierarchical merging in visual document retrieval, token-level pruning followed by selective downstream recombination in multimodal jailbreak defense, and a low-rank analogue in which a rank bottleneck is imposed before LoRA merging rather than after full-space fusion (Zhu et al., 2024, Yan et al., 23 Feb 2026, Chen et al., 2 Jul 2025, He et al., 2 Jun 2026). The common question is whether future deployment constraints should be enforced upstream of fusion, but the answer is domain-dependent rather than universal, as some adjacent literatures explicitly favor merge-first or pruning-aware merge-first strategies (Li et al., 2023, He et al., 2019).

1. Conceptual scope

In the narrowest sense, prune-then-merge refers to pipelines that remove low-importance components before combining models or representations. "DPPA" studies this directly for LLM merging: several models are independently fine-tuned from the same base model, their domain-specific deltas are pruned to remove less important changes, the surviving parameters are rescaled by significance, and the processed deltas are merged back into the base model (Zhu et al., 2024).

In a broader but still explicit usage, "Prune-then-Merge" is a query-agnostic, offline compression framework for efficient multi-vector visual document retrieval. There the object being simplified is not a parameter delta but a set of patch embeddings: low-information patches are filtered out first, and the remaining high-signal embeddings are then hierarchically clustered and centroid-merged (Yan et al., 23 Feb 2026).

A related safety literature uses the ordering in activation space rather than parameter space. "SafePTR" is presented as a prune-then-restore mechanism: harmful multimodal tokens are selectively removed in vulnerable layers, and benign features are restored immediately afterward by combining representations from a pruned branch and a standard branch. The paper explicitly notes that this restore stage is operationally a gated branch merge over token indices, so the method is close in spirit to a broader prune-then-merge paradigm even though its own terminology is "restore" rather than "merge" (Chen et al., 2 Jul 2025).

A structurally different but conceptually parallel line appears in LoRA merging. "Compress then Merge" argues that if the target artifact is a single deployable rank-$r$ LoRA, then the rank bottleneck should be imposed before merge-time interference is resolved. The paper describes this as a low-rank, subspace-based analogue of a prune-then-merge principle: simplification is not sparse masking but a shared low-dimensional constraint enforced before fusion (He et al., 2 Jun 2026).

2. Formal patterns and mathematical structure

In delta-merging for LLMs, the basic object is the domain-specific parameter change

$\Delta^i = W^B - W^i,$

where $W^B$ is the base-model weight and $W^i$ is the fine-tuned weight for domain $i$. DPPA applies Dynamically Pruning (DP) to $\Delta^i$, then Dynamically Partition Amplification (DPA), and finally merges the processed deltas through

$W^m = W^B + \Sigma_{i=1}^k \text{DPPA}(\Delta^i).$

Its distinctive feature is that pruning rates are modulated at both the model-layer level and the linear-layer level, with the local pruning rate

$$\Theta_{lj} = \alpha + norm(dif(\Delta_l)) + norm(dif'(\Delta_{lj})),$$

where significance is defined by summing magnitudes of parameters larger than $N$ times the average magnitude, with $N=5$, and $\Delta^i = W^B - W^i,$0 with $\Delta^i = W^B - W^i,$1 (Zhu et al., 2024).

In low-rank LoRA consolidation, a pretrained linear layer $\Delta^i = W^B - W^i,$2 receives task-specific low-rank updates

$\Delta^i = W^B - W^i,$3

CtM seeks shared orthonormal bases

$\Delta^i = W^B - W^i,$4

and per-task coordinate matrices $\Delta^i = W^B - W^i,$5 by minimizing

$\Delta^i = W^B - W^i,$6

The original LoRAs are then projected into the shared subspaces,

$\Delta^i = W^B - W^i,$7

merged in the reduced $\Delta^i = W^B - W^i,$8 space,

$\Delta^i = W^B - W^i,$9

and reconstructed as

$W^B$0

which guarantees rank at most $W^B$1 by construction (He et al., 2 Jun 2026).

In multi-vector visual document retrieval, a query is represented by token embeddings

$W^B$2

a document page by patch embeddings

$W^B$3

and late interaction uses

$W^B$4

Pruning scores come from head-averaged final-layer attention,

$W^B$5

with document-specific threshold

$W^B$6

so the retained set is

$W^B$7

Merging then reduces the pruned set to

$W^B$8

centroids

$W^B$9

and retrieval proceeds with

$W^i$0

The framework is training-free, query-agnostic, and offline (Yan et al., 23 Feb 2026).

In SafePTR, the manipulated object is the hidden-state stream. At layer $W^i$1,

$W^i$2

and harmful token pruning operates on token hidden states at vulnerable layers. The pruning operator is written as

$W^i$3

Benign Feature Restoration then recombines the pruned and unpruned branches by keeping sanitized states at harmful indices and restoring original states elsewhere (Chen et al., 2 Jul 2025).

3. Empirical evidence for the ordering effect

In the model-merging setting studied by DPPA, the main empirical claim is that reducing conflict before merge substantially improves both sparse domain preservation and downstream merging. At 80% pruning, DPPA reports Domain-Ratio values of 97.08 for math, 96.65 for finance, and 116.02 for law; at 90% pruning, the corresponding values are 86.85, 92.11, and 110.55. For three-domain merging at 90% pruning, DARE reports Math 7.89, Fin 51.48, Law 53.86, whereas DPPA reports Math 89.95, Fin 85.24, Law 122.08. At 80% pruning, DARE reports Math 32.61, Fin 74.49, Law 78.11, whereas DPPA reports Math 91.28, Fin 95.20, Law 146.23 (Zhu et al., 2024).

In visual document retrieval, the explicit Prune-then-Merge framework is evaluated on 29 datasets spanning ViDoRe-V1, ViDoRe-V2, JinaVDR-Bench, REAL-MM-RAG, ViDoSeek, and MMLongBench-Doc. The paper states that the near-lossless compression range is extended by about 10 percentage points on average from roughly 50–60% to 60–70% compression. On ViDoRe-V1, Prune-then-Merge maintains ColQwen2.5’s base nDCG@5 of 0.87 even after about 68% compression; at high compression, Prune-then-Merge at 84% achieves 0.86, whereas DocPruner around 87% achieves 0.77. On REAL-MM-RAG with ColQwen2.5, Prune-then-Merge reports 0.65 at 84% compression, compared with 0.56 for DocPruner and 0.61 for Sem-Cluster (Yan et al., 23 Feb 2026).

In LoRA consolidation, the main result is that compression-first consistently outperforms merge-then-compress baselines when the final object must remain rank-constrained. On the 8-task ViT benchmark, the best CtM result is DARE-TIES + CtM = 70.68, ahead of TIES + CtM = 69.75, Iso-CTS = 66.34, RobustMerge = 64.31, and LoRA-LEGO = 62.36. On the 6-task LLaMA NLI benchmark, DARE-TIES + CtM = 95.63 and TIES + CtM = 94.15. The paper also identifies a truncation gap for merge-then-compress: on ViT-B/32 with CoreSpace coordinates, TIES drops from $W^i$4 to $W^i$5, DARE-TIES from $W^i$6 to $W^i$7, and Iso-C from $W^i$8 to $W^i$9 after SVD truncation (He et al., 2 Jun 2026).

In multimodal jailbreak defense, SafePTR reports that only a narrow contiguous window of layers is especially vulnerable: for LLaVA-1.5-7B and MiniGPT-4-7B the vulnerable span is $i$0, and for DeepSeek-VL2 it is $i$1. The harmful token ratios reported are 0.62% and 0.56% for LLaVA-1.5, 0.93% and 0.81% for MiniGPT-4, and 1.66% and 1.25% for DeepSeek-VL2. On JailbreakV-28K, SafePTR is summarized as 0.98 vs 2.10 vs 14.36 on LLaVA-1.5, 6.48 on MiniGPT-4-7B, and 2.46 on DeepSeek-VL2. The component ablation on LLaVA shows that HTP only yields FIG 3.8, MSB 3.06, MM-Vet 24.5, and MME 1428.11, whereas adding BFR yields FIG 1.6, MSB 1.29, MM-Vet 32.3, and MME 1538.11 (Chen et al., 2 Jul 2025).

4. Why pruning before merging can help

The mechanism is most explicit in delta merging. DPPA assumes that domain expertise is not spread uniformly across all parameters in the delta model. Pruning less important delta entries before merging reduces overlapping changed coordinates across domains, decreases conflict opportunities, and leaves coordinates that are more likely to represent salient domain knowledge. DPA then rescales the surviving partitions in relation to their significance levels rather than using one-shot uniform rescaling (Zhu et al., 2024).

In retrieval, the ordering effect is formulated as a noise-filtering argument. Pruning-only methods are described as precise but brittle at high compression, whereas merging-only methods can blur important semantics because centroids are formed over noisy and low-information patches. The two-stage pipeline first creates a refined, high-signal set of embeddings and only then summarizes it, thereby avoiding the noise-induced feature dilution seen in single-stage methods (Yan et al., 23 Feb 2026).

In low-rank adapter merging, the key claim is that full-parameter merging may destroy the low-rank structure, making it difficult for subsequent compression to recover an effective rank-$i$2 LoRA. CtM therefore computes shared $i$3-dimensional subspaces using only the LoRA weights, projects each adapter into these subspaces to obtain $i$4 coordinates, and applies standard merging rules in the reduced space. This suggests that if the final deployable object must satisfy a hard structural constraint, then enforcing that constraint before fusion can preserve shared structure more faithfully than post-hoc truncation (He et al., 2 Jun 2026).

In SafePTR, the ordering effect arises from causal localization rather than from parameter sparsity. Harmful multimodal tokens are sparse and concentrated in early-middle transformer layers, so pruning those token pathways at vulnerable layers can sharply reduce attack success rate. BFR then selectively recombines representations from the sanitized and unpruned branches, preserving benign information that would otherwise be lost. This suggests that prune-then-merge can also operate over intermediate representations when unsafe behavior is mediated by a small subset of activations rather than by a dense parameter subspace (Chen et al., 2 Jul 2025).

5. Counterpositions and non-equivalent orderings

Prune-then-Merge is not a universally preferred ordering. In sparsely activated Mixture-of-Experts, "Merge, Then Compress" argues for the opposite sequence. M-SMoE first uses routing statistics to identify dominant experts, aligns neurons by permutation matching, groups non-dominant experts under dominant ones by router-logits similarity, and merges each group by activation-frequency-weighted averaging. MC-SMoE then compresses the merged experts into low-rank and structural sparse alternatives. The paper reports that MC-SMoE achieves up to 80% memory and a 20% FLOPs reduction, with the central interpretation that redundancy should be consolidated before compression rather than pruned away before consolidation (Li et al., 2023).

A second counterposition appears in multitask inference. "Pruning-Aware Merging" does not endorse prune-then-merge or arbitrary merge-then-prune. Its argument is that merging should be designed so that later pruning is effective for every possible task combination. PAM constructs multitask networks intended to satisfy information-theoretic conditions under which multitask pruning reduces to non-conflicting single-task-like problems, and then applies existing pruning methods. The reported gains are up to 4.87x less computation against the baseline without network merging and up to 2.01x less computation against the baseline with a state-of-the-art network merging scheme (He et al., 2019).

A further boundary condition is that not every pruning method yields merge-compatible outputs. "Bonsai" is a forward-pass-only structured pruning method that removes self-attention heads and dimensions of the fully connected layers, producing physically smaller dense models. The paper is explicit that this changes tensor shapes, and the technical summary notes that it raises architecture-compatibility issues for later merging unless multiple models are pruned to aligned structures. The same summary characterizes shared-mask pruning before merge as a plausible implication rather than an evaluated result (Dery et al., 2024).

Low-rank compression should also not be conflated with sparse pruning. CtM itself states that the paper is not about pruning in the usual sparse-mask sense and does not directly answer whether magnitude pruning, structured channel pruning, or lottery-ticket-style sparsification should happen before merging. Its result is specific to a structured linear bottleneck represented by learned subspaces $i$5 and linear projections $i$6 (He et al., 2 Jun 2026).

6. Open questions, limitations, and practical significance

Several limitations recur across the literature. DPPA states that it performs less effectively than DARE on fine-tuned models with minimal differences compared to the original model, and that DPA requires a longer time to find the optimal ratio. SafePTR is not black-box, depends on hidden-state access and a safety-aligned reference prompt, uses a fixed Top-$i$7 strategy with $i$8 even though the harmful-token analysis is much sparser, and requires model-specific vulnerable layers discovered empirically. The VDR framework depends on attention quality and preset hyperparameters $i$9 and $\Delta^i$0, and hierarchical clustering adds offline latency. Bonsai is slow, with a reported runtime of about 40 hours for the main forward-only 50% LLaMA-2 7B result, and its structured dimension removal creates the strongest caveat for subsequent merging (Zhu et al., 2024, Chen et al., 2 Jul 2025, Yan et al., 23 Feb 2026, Dery et al., 2024).

The broader theoretical question is therefore not simply whether one should always prune before merging. The available evidence is more specific. When interference is created by low-value delta parameters, by low-information retrieval patches, by sparse harmful tokens in vulnerable layers, or by full-space LoRA fusion that destroys a required low-rank structure, simplification-before-fusion can be strongly beneficial. When the primary issue is expert redundancy that can be consolidated by routing statistics, or when later pruning across arbitrary task subsets must be anticipated during network construction, merge-first or pruning-aware merge-first can be preferable (Zhu et al., 2024, Yan et al., 23 Feb 2026, Chen et al., 2 Jul 2025, He et al., 2 Jun 2026, Li et al., 2023, He et al., 2019).

A plausible implication is that "Prune-then-Merge" is best understood not as a single algorithm but as a family of constrained-fusion strategies. What unifies the family is the decision to expose common structure only after low-value, noisy, or harmful components have been removed from the object being fused. What separates the methods is the nature of the simplification: sparse delta pruning, adaptive embedding filtering, token-pathway suppression, or low-rank subspace restriction. The current literature therefore supports prune-then-merge as a powerful but conditional design principle rather than as a universally dominant rule.