Adjugate Experts in Grove MoE

• Adjugate experts are group-shared components in Grove MoE that add a scaled output to base experts, enabling dynamic activation and heterogeneous capacity.
• They reuse computation by evaluating the adjugate expert only once per expert group, reducing redundant processing and overall computational cost.
• Inspired by big.LITTLE CPU design, this architecture enhances model capacity while controlling compute overhead in sparse large language models.

Searching arXiv for the referenced work and closely related context. Attempting arXiv lookup for “Grove MoE adjugate experts”. Adjugate experts are a component of the Grove MoE architecture, where experts are partitioned into groups and each group is assigned a shared parallel expert whose output is added to the outputs of the activated experts in that group. In the formulation introduced with Grove MoE, adjugate experts support heterogeneous expert sizes and a dynamic activation mechanism, with the stated objective of expanding model capacity while maintaining manageable computational overhead in Mixture of Experts LLMs (Wu et al., 11 Aug 2025).

1. Definition and architectural setting

Within a conventional MoE layer with experts $\{E_i\}_{i=1}^n$, router $R$, routing scores $\vec{\rho}=R(\vec{x})$, and top-$k$ activation, the token output is

$$\vec{y}=\sum_{i\in \arg \mathrm{top}k(\vec{\rho})}\rho_i \vec{e}_i, \qquad \vec{e}_i=E_i(\vec{x}).$$

Grove MoE modifies this arrangement by dividing the experts into $g$ disjoint groups $G_j$, each of size $n/g$, and attaching to every group a shared adjugate expert $A_j$. For an expert $E_i$ belonging to group $R$0, the adjugate-augmented output is defined as

$R$1

with $R$2 a scaling factor. The final output becomes

$R$3

where $R$4 (Wu et al., 11 Aug 2025).

This construction retains the standard top-$R$5 routing interface while altering what an activated expert contributes. The defining feature is not a new router, but a group-shared augmentation path. The paper presents this as an architecture inspired by heterogeneous big.LITTLE CPU design: small experts provide sparse specialist computation, while the group-level adjugate expert contributes additional shared capacity.

2. Group sharing, compute reuse, and dynamic activation

The operational distinction of adjugate experts is that they are shared within expert groups. If multiple activated experts belong to the same group, the adjugate expert for that group is evaluated once and its contribution is reused. For experts $R$6, the shared adjugate term contributes

$R$7

Accordingly, only groups that contain at least one activated expert require adjugate-expert computation, and the number of active adjugate experts per layer lies between

$R$8

for a token (Wu et al., 11 Aug 2025).

The paper characterizes this as a dynamic activation mechanism because the amount of adjugate computation depends on the routing pattern of the token rather than being fixed a priori. This differs from homogeneous MoE in two respects. First, the additional computation is group-conditional rather than expert-conditional. Second, the compute budget can vary with token complexity through the realized grouping of the top-$R$9 experts. A common misunderstanding is that every activated expert necessarily triggers its own additional adjugate computation; the Grove formulation explicitly states the opposite when several selected experts fall in the same group.

3. Heterogeneous capacity and scaling behavior

Adjugate experts are also the mechanism through which Grove MoE introduces heterogeneous expert sizes. The paper states that the adjugate experts $\vec{\rho}=R(\vec{x})$0 may use intermediate dimensions $\vec{\rho}=R(\vec{x})$1 different from those of the base experts $\vec{\rho}=R(\vec{x})$2, permitting combinations of “big” and “small” experts within one layer. This is the architectural basis for the analogy to big.LITTLE cores and for the claim that the design enables model capacity expansion without proportional dense-style compute growth (Wu et al., 11 Aug 2025).

The scaling factor $\vec{\rho}=R(\vec{x})$3 is included to preserve weight normalization and avoid overwhelming the base expert contribution. The reported guideline is typically $\vec{\rho}=R(\vec{x})$4, and the empirical summary states that low values of $\vec{\rho}=R(\vec{x})$5, specifically $\vec{\rho}=R(\vec{x})$6–$\vec{\rho}=R(\vec{x})$7, yield the best results. The architecture study further reports that more groups $\vec{\rho}=R(\vec{x})$8, corresponding to smaller groups, save more compute.

Aspect	Traditional MoE	Grove MoE with adjugate experts
Expert size	Homogeneous	Heterogeneous
Per-expert output	$\vec{\rho}=R(\vec{x})$9	$k$0
Extra compute	Expert-level	Group-shared
Routing changes	Not applicable	No router modification required

This arrangement implies a separation between capacity growth and activated compute growth. The paper’s interpretation is that adjugate experts act as a shared higher-capacity supplement for grouped specialists, while the final activated parameter count remains controlled by sparse routing and reuse of the group-level computation.

4. Integration into Grove MoE models

The adjugate-expert mechanism is instantiated in two 33B-parameter models, GroveMoE-Base and GroveMoE-Inst, developed from Qwen3-30B-A3B-Base through an upcycling strategy applied during mid-training and post-training. The description emphasizes compatibility with weight reuse: pre-trained expert weights are reused, while adjugate experts are initialized with zero-initialized down-projection and small random weights elsewhere (Wu et al., 11 Aug 2025).

GroveMoE-Base is the mid-trained model under the Grove MoE architecture, and GroveMoE-Inst is its instruction-tuned derivative. This training pathway is significant because it presents adjugate experts not only as a from-scratch architectural variant but as a post hoc expansion mechanism for an existing MoE backbone. A plausible implication is that the design is meant to lower the cost of capacity expansion relative to retraining an entirely new model, although the paper’s factual emphasis is on upcycling compatibility rather than on a formal cost comparison.

5. Empirical profile and systems characteristics

The reported parameter and compute profile is central to the practical identity of adjugate experts. GroveMoE models expand total parameters to 33B from the 30B Qwen3-30B-A3B-Base model, while dynamically activating 3.14–3.28B parameters per token. The paper states that this active-parameter range remains comparable to standard MoE, and that the resulting models achieve performance comparable to state-of-the-art open-source models of similar or even larger size (Wu et al., 11 Aug 2025).

The architecture ablation summary reports that with reasonable group sizes, specifically $k$1 for $k$2 experts, compute drops by about $k$3, and with fewer groups compute can drop about $k$4. At the implementation level, the paper notes that the current system is $k$5 slower than Qwen3-30B-A3B, mainly because of generic fused MoE kernels, and states that a unified GroveMoE kernel would reduce this to about $k$6 overhead, aligning with the theoretical expectation.

The benchmark summary is likewise framed comparatively rather than with exhaustive per-task numbers. Grove MoE is reported to achieve higher or comparable scores across general, reasoning, mathematical, and coding benchmarks relative to both dense and MoE baselines of similar or even larger size. This establishes adjugate experts as an efficiency-oriented modification whose intended benefit is not merely parameter inflation but improved capacity-to-compute tradeoff under sparse activation.

6. Conceptual scope, misconceptions, and terminological context

Several points delimit what adjugate experts are and are not. They do not require a new routing algorithm: the Grove description states that no change is required to routing algorithms and that the architecture is fully compatible with top-$k$7 routing and established gating strategies. They also do not imply that each activated expert entails a distinct auxiliary forward pass, because adjugate computations are explicitly shared at the group level. Nor are they confined to homogeneous expert banks, since heterogeneous intermediate dimensions are a defining part of the formulation (Wu et al., 11 Aug 2025).

The term “adjugate” in this architectural setting should also be distinguished from the classical adjugate matrix used elsewhere in mathematics and physics. In other arXiv work, “adjugate” denotes the distributional adjugate of $k$8 for bi-$k$9 homeomorphisms (Hencl et al., 2019), the adjugate of $$\vec{y}=\sum_{i\in \arg \mathrm{top}k(\vec{\rho})}\rho_i \vec{e}_i, \qquad \vec{e}_i=E_i(\vec{x}).$$0 for extracting neutrino-matter eigenvectors (Abdullahi et al., 2022), derivatives of $$\vec{y}=\sum_{i\in \arg \mathrm{top}k(\vec{\rho})}\rho_i \vec{e}_i, \qquad \vec{e}_i=E_i(\vec{x}).$$1 in relation to Jordan decomposition (Rubiano-Murcia et al., 2023), generalized Thompson–McEnteggert formulas for $$\vec{y}=\sum_{i\in \arg \mathrm{top}k(\vec{\rho})}\rho_i \vec{e}_i, \qquad \vec{e}_i=E_i(\vec{x}).$$2 (Castillo et al., 2021), and the adjugate of a characteristic matrix in quaternion pose extraction (Hanson et al., 2022). In Grove MoE, by contrast, “adjugate experts” names a group-shared expert mechanism in sparse LLM architecture rather than an operation on cofactors or determinants.

This distinction matters for interpretation. The Grove paper uses “adjugate” as a model-design term attached to shared experts and dynamic activation, whereas the earlier literature uses it in the linear-algebraic sense. The shared name does not imply a common mathematical object.

7. Significance for MoE design

Adjugate experts occupy a specific place in the evolution of sparse LLMs: they are presented as a means of combining dynamic computation, heterogeneous capacity, and upcycling compatibility within the MoE paradigm. The architecture’s stated benefits are expanded capacity at modest compute cost, reduced redundant computation through shared group-level evaluation, specialization through small group experts plus a larger shared expert, and straightforward integration with existing top-$$\vec{y}=\sum_{i\in \arg \mathrm{top}k(\vec{\rho})}\rho_i \vec{e}_i, \qquad \vec{e}_i=E_i(\vec{x}).$$3 gating (Wu et al., 11 Aug 2025).

From the reported evidence, the principal significance of adjugate experts is architectural rather than purely algorithmic. They alter the granularity at which sparse computation is shared, moving from independent expert activation toward a grouped structure in which extra capacity is amortized across co-selected experts. This suggests a broader design direction for MoE systems: parameter growth need not be tied to uniform expert blocks or to fixed per-token activation patterns. In Grove MoE, adjugate experts are the mechanism through which that direction is concretely realized.