Router-weighted Expert Activation Merging

• The paper demonstrates that REAM reduces memory footprint by merging experts based on router-weighted activation saliency, preserving model performance better than pruning.
• REAM applies pseudo-pruning by aligning experts with centroids via the Hungarian algorithm and merging them through saliency-weighted averaging.
• The method achieves robust trade-offs in multi-choice and generative tasks across various MoE architectures without requiring retraining.

Searching arXiv for the cited REAM and adjacent MoE compression/routing papers to ground the article. Router-weighted Expert Activation Merging (REAM) is a one-shot compression method for Mixture-of-Experts (MoE) LLMs that reduces memory footprint by grouping experts and merging their weights rather than removing them outright. In the formulation reported for REAM, the method inherits the saliency notion of Router-weighted Expert Activation Pruning (REAP)—namely, router-weighted expert activation magnitude—but replaces expert deletion with saliency-guided merging into retained centroids. The result is a per-layer, static compression procedure intended to preserve more of the original model behavior than pruning, while remaining compatible with retraining-free deployment workflows (Jha et al., 6 Apr 2026).

1. Definition and architectural setting

REAM is defined for standard sparse MoE layers in which the router produces gate logits

$g(\mathbf{x})=\mathbf{x}W_g \in \mathbb{R}^N,$

followed by softmax probabilities

$\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$

and top-$k$ masking to obtain sparse routing weights

$$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$

The MoE output is

$$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$

where $E_i$ is expert $i$ (Jha et al., 6 Apr 2026).

Within this setting, REAM addresses the deployment bottleneck that sparse MoE reduces active computation but not total parameter storage. The method is therefore a memory-oriented expert compression scheme rather than a new routing architecture. It operates independently at each MoE layer, compressing $N$ experts to a smaller target count $N'$, typically corresponding to 25% or 50% expert reduction in the reported experiments (Jha et al., 6 Apr 2026).

A central distinction in the literature is between pruning, editing, and merging. One later taxonomy places REAM squarely in the “Expert Merging” class, characterized by reducing the number of experts from $N$ to $\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$0 by combining multiple original experts into synthesized experts (Hyeon et al., 10 Feb 2026). This suggests that REAM is best understood not as a routing method in the narrow sense, but as a retraining-free parameter-level compression method whose expert grouping criteria are explicitly router-aware.

2. Router-weighted saliency and the rationale for merging

REAM inherits its core importance score from REAP. A simple baseline importance measure is routing frequency,

$\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$1

which counts how often an expert is selected. REAP, and then REAM, instead define expert saliency as

$\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$2

where $\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$3 is the set of calibration tokens on which expert $\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$4 is active (Jha et al., 6 Apr 2026).

This score is router-weighted in a literal sense: it multiplies the activation norm of expert $\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$5 by the sparse top-$\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$6 gate weight that expert receives on each token. REAM uses this quantity to identify experts that should survive compression as centroids. The rationale is that pruning low-saliency experts entirely may discard useful information, whereas merging them into similar high-saliency experts may preserve more of the original function (Jha et al., 6 Apr 2026).

The motivation is also informed by a contrasting line of work. REAP argues that, for one-shot compression on generative tasks, pruning is often preferable to merging because summed-gate merging destroys the router’s independent, input-dependent control and introduces irreducible error through what that paper calls “functional subspace collapse” (Lasby et al., 15 Oct 2025). REAM explicitly departs from that conclusion by retaining REAP’s saliency signal while changing the compression operator from deletion to grouped saliency-weighted merging (Jha et al., 6 Apr 2026). A plausible implication is that REAM can be read as an attempt to preserve the strengths of router-aware importance estimation while avoiding pure expert removal.

3. Similarity metrics, pseudo-pruning, and group formation

REAM does not merge experts solely by saliency. It also computes pairwise similarity using an aggregated metric

$\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$7

where $\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$8 is gate-logit similarity and $\sigma(\mathbf{x})=\text{Softmax}(g(\mathbf{x})),$9 is a router-aware expert-output similarity (Jha et al., 6 Apr 2026).

Gate-logit similarity is defined as

$k$0

while a standard output similarity from earlier merging work is

$k$1

REAM modifies the latter into a gated expert similarity,

$k$2

using softmax probabilities $k$3, not top-$k$4-masked weights $k$5 (Jha et al., 6 Apr 2026).

The grouping procedure is called “pseudo-pruning.” For each layer $k$6, REAM selects the $k$7 highest-saliency experts as centroids,

$k$8

ordered by decreasing $k$9. It then greedily assigns non-centroid experts to centroids in descending centroid saliency order, up to a per-centroid group-size hyperparameter $$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$0, using $$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$1 as the similarity criterion (Jha et al., 6 Apr 2026).

This grouping strategy is described as “pseudo-pruning” because most centroids remain singleton groups while only a subset absorb additional experts. That behavior differs from classical clustering-based merging, which tends to force all experts into medium-sized clusters. In REAM, the intent is to preserve salient experts largely intact while absorbing only nearby, less important experts into them (Jha et al., 6 Apr 2026).

A later compression analysis gives a useful lens for interpreting this choice. In a merged model with cluster map

$$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$2

the original top-$$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$3 expert set $$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$4 projects into cluster space as

$$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$5

If $$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$6, some information loss is structurally unavoidable because the merged router cannot activate enough distinct clusters (Hyeon et al., 10 Feb 2026). This suggests that REAM’s preference for grouping low-saliency experts around salient centroids is not only a saliency heuristic but also a way to limit projected-cluster fragmentation.

4. Alignment, weighted merging, and sequential compression

Before averaging weights, REAM aligns each assigned expert to its centroid to address neuron permutation symmetry. For centroid $$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$7 and assigned expert $$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$8, the alignment cost matrix is

$$\pi(\mathbf{x})=\text{Mask}\big(\sigma(\mathbf{x}), \text{top-}k\big)\in\mathbb{R}^N.$$9

with entries

$$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$0

and

$$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$1

Here $$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$2 and $$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$3 are normalized calibration-token activation vectors, and $$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$4 are neuron weights. The Hungarian algorithm is then used to permute the assigned expert relative to the centroid (Jha et al., 6 Apr 2026).

After alignment, REAM merges each centroid group by saliency-weighted averaging. The paper’s method description makes clear that the weights are based on REAP saliency rather than routing frequency. An implementable form consistent with the text is

$$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$5

where $$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$6 is centroid $$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$7’s group and $$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$8 are aligned expert weights (Jha et al., 6 Apr 2026).

REAM also updates the router in a pruning-like way rather than keeping all gate rows and summing them within groups. After compression, rows of $$\mathbf{y}(\mathbf{x})=\sum\nolimits_{i=1}^{N}\pi(\mathbf{x})_i\,E_i(\mathbf{x}),$$9 corresponding to non-centroid experts are removed, and only the centroid rows are retained (Jha et al., 6 Apr 2026). This is notable because REAP had argued that summed-gate one-shot merging introduces a structural error term associated with policy variability (Lasby et al., 15 Oct 2025). REAM therefore avoids one common merged-gate construction and instead compresses the expert bank to a smaller retained router.

A further component is sequential merging. Rather than collecting all layer statistics once from the original model and compressing every layer independently from stale downstream activations, REAM compresses layer $E_i$0, reruns a forward pass through the updated layer, and then collects statistics for layer $E_i$1 from the compressed model state (Jha et al., 6 Apr 2026). On Qwen3-30B-A3B-Instruct-2507, the reported one-time overhead is about 1 hour for non-sequential merging versus about 1.5 hours for sequential merging, at roughly 30 GB VRAM (Jha et al., 6 Apr 2026). This suggests that REAM treats activation drift across layers as operationally significant during compression, even though the procedure remains one-shot and fine-tuning-free.

5. Empirical behavior and calibration-data trade-offs

REAM is evaluated on Qwen3-30B-A3B-Instruct-2507, Qwen3-Next-80B-A3B-Instruct, Qwen3-Coder-Next, and GLM-4.5-Air. Compression settings include $E_i$2, $E_i$3, and $E_i$4 experts per layer, corresponding to 25% and 50% reduction in the main settings (Jha et al., 6 Apr 2026).

Calibration uses 3072 sequences of 512 tokens drawn from mixtures of C4, NuminaMath, and The-Stack-Smol, with ten mixture ratios studied. Evaluation is split between eight multiple-choice tasks and six generative tasks, reported as separate MC and GEN averages (Jha et al., 6 Apr 2026).

Several reported results establish the method’s operating regime. On Qwen3-30B-A3B-Instruct-2507 at 25% compression ($E_i$5), using each method’s best GEN-favoring calibration ratio, the original model scores GEN 70.9; Freq scores 67.6; HC-SMoE 67.4; REAP 68.6; and REAM 69.8. In that setting REAM is 1.1 points below the uncompressed model and 1.2 points above REAP (Jha et al., 6 Apr 2026). On Qwen3-80B-A3B compressed $E_i$6, REAP scores GEN 69.6 while REAM reaches 71.5 against the original 72.9. On GLM-4.5-Air compressed $E_i$7, REAP scores 71.9 and REAM 73.9 against the original 77.1. On Qwen3-Coder-Next compressed $E_i$8, REAM matches the original GEN average at 72.9, while REAP reports 70.7 (Jha et al., 6 Apr 2026).

The paper also highlights a strong calibration-dependent MC–GEN trade-off. For saliency-based methods including REAM, increasing the C4 fraction improves MC but hurts GEN, while increasing code data improves GEN but harms MC; the reported appendix correlations are approximately $E_i$9 to $i$0 between C4 fraction and MC, and $i$1 to $i$2 between C4 fraction and GEN, with code fraction showing the opposite tendency (Jha et al., 6 Apr 2026). This suggests that REAM’s router-weighted activation statistics act as a workload prior: the calibration mix implicitly decides which experts are preserved.

The Pareto analysis formalizes this calibration sensitivity. At 25% compression, HC-SMoE reports hypervolume 853.3 with $i$3 Pareto-optimal configurations, Freq reports 429.7 with $i$4, REAP reports 878.0 with $i$5, and REAM reports 920.3 with $i$6 (Jha et al., 6 Apr 2026). The intended interpretation is that REAM produces the strongest MC–GEN trade-off surface among the compared one-shot methods.

6. Relation to adjacent methods and unresolved issues

REAM sits at the intersection of several lines of MoE research. It is directly motivated by REAP, which defines the router-weighted expert activation saliency that REAM adopts, but REAP itself argues against one-shot merging for generative compression because static merged experts lose the router’s independent, input-dependent control (Lasby et al., 15 Oct 2025). REAM therefore constitutes a direct counter-position: it treats router-weighted activation information as sufficiently rich to support grouping and merging, provided grouping, weighting, and calibration are done carefully (Jha et al., 6 Apr 2026).

It also differs from activation-time architectural methods. ReXMoE expands each layer’s candidate pool by allowing routers to access experts from adjacent layers and then combines selected expert activations by the usual sparse weighted sum

$i$7

where $i$8 may come from adjacent-layer expert pools (Tan et al., 20 Oct 2025). This is conceptually adjacent to REAM because it is router-weighted expert activation aggregation across multiple expert pools, but it is still an in-model forward-pass routing mechanism rather than a static post-training compression operator.

PASs-MoE provides another neighboring idea by replacing a separate router with weights derived from expert-internal low-rank activation energies,

$i$9

followed by dense weighted expert combination

$N$0

(Hou et al., 19 Jan 2026). That mechanism is REAM-like in the sense that expert activations themselves generate the merge coefficients, but it is aimed at continual instruction tuning rather than compression.

A distinct issue raised after REAM is router–expert mismatch. A later compression study argues that retraining-free compression often underperforms because experts are changed while the router is left optimized for the pre-compression expert landscape. It proposes Router Knowledge Distillation, which updates only router parameters by matching the original model’s next-token distribution on unlabeled calibration data (Hyeon et al., 10 Feb 2026). This suggests that REAM’s strongest deployment form may not be pure one-shot merging alone, but router-aware merging followed by lightweight router recalibration. That implication is not part of REAM’s original procedure, but it is a natural extension suggested by subsequent evidence.

Finally, REAM remains calibration-sensitive and static. It does not adapt its merge structure online, does not expose a global merge operator decoupled from layer structure, and does not define a generic post hoc activation-fusion primitive. Its grouping is fixed per layer; its quality depends heavily on calibration-data composition; and some benchmarks, such as AIME25 with only 30 problems, are explicitly noted as high variance (Jha et al., 6 Apr 2026). The broader literature therefore frames REAM as strong evidence that router-weighted expert activation statistics can support practical one-shot expert merging, but not as a final resolution of the trade-off between pruning, merging, routing fidelity, and post-compression calibration (Hyeon et al., 10 Feb 2026).