---
title: 'MEML-GRPO: Multi-Expert Mutual Learning'
url: https://www.emergentmind.com/topics/multi-expert-mutual-learning-grpo-meml-grpo
type: topic
---

# MEML-GRPO: Multi-Expert Mutual Learning

Searching arXiv for the cited MEML-GRPO paper and related mutual-RL/GRPO work to ground the article.
Multi-Expert Mutual Learning GRPO (MEML-GRPO) is a framework for reinforcement learning with verifiable rewards (RLVR) that addresses reward sparsity in reasoning-oriented large language model (LLM) post-training by coupling diverse expert prompting with inter-expert knowledge transfer. In the original formulation, diverse expert prompts are used as system prompts to generate a broader range of responses, thereby increasing the likelihood of identifying correct solutions, while an inter-expert mutual learning mechanism enables knowledge sharing and transfer among experts during RLVR [2508.09670]. Reported benchmark results indicate an average performance gain of 4.89% with Qwen and 11.33% with Llama across multiple reasoning benchmarks [2508.09670]. A later self-contained summary presents a formalized MEML-GRPO-style mutual RL substrate for heterogeneous language models, instantiated on top of GRPO through three controlled sharing regimes—Peer Rollout Pooling (PRP), Cross-Policy GRPO Advantage Sharing (XGRPO), and Success-Gated Transfer (SGT)—and characterizes their stability-support trade-offs in a contextual-bandit setting [2605.07244].

## 1. Origin, problem setting, and stated objective

MEML-GRPO is motivated by a specific failure mode of standard RLVR: reward sparsity when candidate answers are consistently incorrect and therefore receive zero reward. In that regime, RL updates provide little or no useful learning signal, especially on challenging reasoning tasks [2508.09670]. The stated objective of MEML-GRPO is to overcome this limitation by increasing response diversity through heterogeneous expert prompts and by transferring useful information across experts during RLVR [2508.09670].

The framework is explicitly positioned within reasoning enhancement for LLMs. The original description states that recent advances show RLVR significantly enhances LLM reasoning capabilities, but that standard RLVR remains constrained by the absence of informative gradients on all-failure groups [2508.09670]. MEML-GRPO responds by broadening the response distribution via multiple experts and by adding mutual learning among them, with the intended effect of raising the probability that at least one candidate in a rollout group is correct and thus produces a nonzero verifiable reward [2508.09670].

A plausible implication is that MEML-GRPO should be understood less as a single optimizer than as a family of RLVR procedures in which diversity generation and cross-expert transfer are co-designed. That interpretation is supported by the later formalization, which decomposes mutual learning into data-level, value-level, and outcome-level sharing mechanisms on top of a common GRPO base [2605.07244].

## 2. GRPO base and the mutual-learning substrate

The later formalization defines the GRPO base in standard critic-free, group-relative terms. A policy $\pi_\theta$ maps a prompt $x$ to a response $y=(y_1 \ldots y_T)$, while rollouts are drawn from a fixed behavior snapshot $\pi_{\theta_{\text{old}}}$ during training [2605.07244]. Token-level log-probability vectors are
$$
\ell_\theta(x,y)=[\log \pi_\theta(y_t|x,y_{<t})]_{t=1\ldots T},
$$
and
$$
\ell_{\text{old}}(x,y)=[\log \pi_{\theta_{\text{old}}}(y_t|x,y_{<t})]_{t=1\ldots T}.
$$
Per-token importance weights are then
$$
w_{\theta,t}(x,y)=\exp[\ell_{\theta,t}(x,y)-\ell_{\text{old},t}(x,y)].
$$
For each prompt, $K$ responses are sampled with scalar rewards $r_i$, the group mean reward is $\bar r=(1/K)\sum_j r_j$, and group-relative advantages are
$$
A_i=\frac{r_i-\bar r}{\operatorname{std}_j\, r_j+\epsilon}.
$$
The GRPO loss is given as
$$
\mathcal{L}_{\text{GRPO}}(\theta)= -\mathbb{E}_x \frac{1}{K}\sum_{i=1}^K \mathbb{E}_{t=1\ldots |y_i|}
\left[\min\left(w_{\theta,t}(x,y_i)A_i,\ \operatorname{clip}(w_{\theta,t}(x,y_i),1-\epsilon,1+\epsilon)A_i\right)\right]
+\beta R_{\text{KL}}(\theta),
$$
with a corresponding policy-gradient form using $\rho(x,y)=\prod_t w_{\theta,t}(x,y)$ [2605.07244].

On top of this base, the mutual-learning substrate consists of three modules: Shared Experience Exchange (SEE), Multi-Worker Resource Allocation (MWRA), and a Tokenizer Heterogeneity Layer (THL) [2605.07244]. SEE is a central store of records
$$
D=(id, x_{\text{text}}, y_{\text{text}}, r, A, \ell, meta),
$$
through which each policy can publish and subscribe to selected fields while preserving provenance [2605.07244]. MWRA assigns GPU/TPU devices and worker pools across policies, with even splitting of $N$ devices across $M$ policies by default and user-configurable pinning, thereby ensuring matched compute budgets across policies [2605.07244]. THL performs text adaptation by decoding peer text and retokenizing it with the local tokenizer, then aligns token-level traces across incompatible vocabularies by constructing word-span partitions via whitespace or per-character spans for CJK and redistributing per-word source log mass across target tokens [2605.07244].

The THL alignment procedure defines, for each span $w$, a source mass
$$
Z_w=\sum_{t\in S_w}\ell^{\text{src}}_t,
$$
which is evenly assigned to each target token $u\in T_w$ as
$$
\tilde{\ell}^{\text{tgt}}_u=\frac{Z_w}{|T_w|}.
$$
Uncovered or boundary-mismatched spans induce a residual $R_{\text{THL}}$, with the bound
$$
|R_{\text{THL}}| \le B \cdot C_{\text{mis}},
$$
where $B$ bounds $|\ell|$ and $C_{\text{mis}}$ counts misaligned tokens [2605.07244]. This later substrate makes explicit how heterogeneous experts can exchange typed experience while keeping separate parameters, objectives, and tokenizers [2605.07244].

## 3. Heterogeneous experts, prompt diversity, and knowledge transfer

The defining claim of the original MEML-GRPO formulation is that it “utilizes diverse expert prompts as system prompts to generate a broader range of responses” and introduces “an inter-expert mutual learning mechanism that facilitates knowledge sharing and transfer among experts” [2508.09670]. In that description, expert heterogeneity is prompt-induced rather than necessarily architecture-induced: experts differ through system-prompt specialization, and the expanded response manifold increases the chance that verifiable rewards will be observed on difficult instances [2508.09670].

The later formalization broadens heterogeneity beyond prompting to include heterogeneous LLM policies with separate tokenizers, objectives, and parameters [2605.07244]. That is an extension in representational scope: the sharing substrate is designed to function across model families rather than only across prompt variants of a single base model [2605.07244]. This suggests that “multi-expert” in MEML-GRPO can be instantiated at several levels of heterogeneity, including prompt, model family, tokenizer, and training objective, provided the exchange interface is sufficiently typed and the alignment layer handles token incompatibilities.

Knowledge transfer in this framework is not monolithic. Instead, the later summary decomposes it into three operational regimes that import different peer fields from SEE [2605.07244]. This decomposition is significant because it isolates distinct mechanisms of mutual learning: importing peer trajectories, importing peer-derived normalization statistics, and importing successful outcomes only under a gate. In effect, the formalization replaces a general notion of “inter-expert mutual learning” with a controlled design space whose stability and support properties can be analyzed separately [2605.07244].

## 4. Controlled sharing regimes: PRP, XGRPO, and SGT

The first sharing regime is Peer Rollout Pooling (PRP), described as data-level sharing [2605.07244]. Learner $n$ augments its own group $Y_n=\{y^{(n)}_{1\ldots K}\}$ with peer rollouts $Y_m$ for $m\neq n$ to form a pooled set $Y_{\text{pool}}^{(n)}$, and group-relative advantages are computed over the pooled responses [2605.07244]. Two denominator choices are defined for the importance ratio: a learner-snapshot variant that treats peer rollouts as if they came from the learner’s own behavior snapshot, and a THL-aligned peer variant that uses the aligned peer trace $\tilde{\ell}^{(m\rightarrow n)}$ in the denominator [2605.07244]. Theoretical analysis states that this regime is unbiased only if $\pi^{(n)}\ll \mu^{(m)}$, that its variance is proportional to $\chi^2(\pi\|\mu)$, and that THL residuals multiply the importance ratio [2605.07244].

The second regime is Cross-Policy GRPO Advantage Sharing (XGRPO), described as value-level sharing [2605.07244]. Each learner samples only its own rollouts, but pooled reward statistics are computed across policies for each prompt. If $\{r^{(m)}_j\}_{m=1\ldots M,\,j=1\ldots K}$ denotes the reward multiset, then pooled mean $\mu_{\text{pool}}(x)$ and standard deviation $\sigma_{\text{pool}}(x)$ define the effective advantage
$$
\hat A_i=\frac{r_i^{(n)}-\mu_{\text{pool}}(x)}{\sigma_{\text{pool}}(x)+\epsilon}.
$$
This $\hat A_i$ replaces the local $A_i$ in the GRPO loss [2605.07244]. The actor remains strictly on-policy, no off-policy $\rho$ is introduced, and a stated variance analysis gives
$$
\text{Var change}=H_n[\Delta^2-2D\Delta],
$$
which can reduce variance when $\Delta$ aligns with local baseline error [2605.07244].

The third regime is Success-Gated Transfer (SGT), described as outcome-level sharing [2605.07244]. A gate
$$
G_n(x)=(1-p_n(x))^K\cdot \left[1-\prod_{m\neq n}(1-p_m(x))^K\right]
$$
fires when learner $n$ has no successes but some peer has at least one success [2605.07244]. On that rescue subset, a peer success $y^*$ is selected uniformly from all peer successes, retokenized via THL, and an auxiliary negative log-likelihood term
$$
\mathcal{L}_{\text{SGT}}(\theta)=-\log \pi_\theta^{(n)}(y^*|x)
$$
is added to the total loss
$$
\mathcal{L}=\mathcal{L}_{\text{GRPO}}+\lambda \cdot I_{\text{gate}(x)}\cdot \mathcal{L}_{\text{SGT}}.
$$
The stated guarantee is that, on a rescue event, SGT supplies a positive score-function direction for $y^*$, increasing $\log \pi_\theta(y^*|x)$ and decreasing the peer-success KL in expectation; the perturbation is weighted by $G_n(x)$ [2605.07244].

These three probes provide a structured interpretation of mutual learning. PRP shares support, XGRPO shares baselines, and SGT shares verified successes only when local learning signal is absent. That structure is not stated explicitly in the original abstract, but it is a plausible synthesis of the later decomposition with the original claim that MEML-GRPO addresses sparse reward through expert diversity and mutual learning [2508.09670; 2605.07244].

## 5. Theoretical characterization and the stability-support trade-off

The later analysis treats each prompt $x$ as a single-step contextual bandit and defines $p_n(x)=\Pr_{y\sim \pi^{(n)}}[y\text{ is successful}]$ [2605.07244]. Within that reduction, the three sharing regimes occupy different positions on a stated stability-support trade-off [2605.07244].

For PRP, the main issue is off-policy support mismatch. The variance bound is reported as
$$
\operatorname{Var}[\rho A \nabla \log \pi] \le A_{\max}^2 G^2 \cdot (1+\chi^2(\pi\|\mu)),
$$
and THL residuals enter multiplicatively via $\tilde{\rho}\in [e^{-\delta}, e^{+\delta}]\,\rho$ [2605.07244]. This formalizes the cost of importing peer rollouts when the learner’s policy and the peer’s behavior distribution are insufficiently aligned.

For XGRPO, the central claim is that pooled baselines introduce no bias in the expectation of the score term, while variance may improve depending on how the pooled baseline corrects local baseline error [2605.07244]. The regime leaves $\nabla \log \pi(y^*|x)$ attached only to learner-sampled trajectories and therefore does not introduce a direct term for peer-only successes [2605.07244]. In other words, it improves normalization rather than support.

For SGT, the rescue gate both enables and limits transfer. On the event $G_n(x)$, the learner receives an explicit score-function direction toward a verified peer success; as the learner’s own success probability rises, the gate self-limits [2605.07244]. The expected perturbation norm is bounded by
$$
\|\Delta_\theta\| \le \eta \lambda G_S \mathbb{E}_x[G_n(x)],
$$
which makes the transfer strength explicitly proportional to rescue-event frequency [2605.07244].

A common misconception would be to treat all mutual-learning schemes as equivalent forms of cross-policy distillation. The formalization does not support that equivalence. PRP changes the sampled support, XGRPO changes only the scalar baseline structure, and SGT injects supervised pressure toward verified peer successes on a gated subset [2605.07244]. Their statistical roles are therefore materially different.

## 6. Empirical findings, implementation, and scope

The original MEML-GRPO abstract reports “extensive experiments across multiple reasoning benchmarks” and states that the method achieves “an average performance gain of 4.89% with Qwen and 11.33% with Llama,” thereby “effectively overcoming the core limitations of traditional RLVR methods” [2508.09670]. No additional experimental breakdown is provided in the supplied source for that paper beyond these benchmark-level summary claims [2508.09670].

The later summary supplies a more granular evaluated regime. It uses pools of heterogeneous open-weight LLMs, including examples such as Qwen2.5-Math-1.5B and Qwen3-1.7B, plus Phi-4-mini or Llama-3B, and evaluates on verifiable reasoning on MATH, extended to multiple-choice and scientific QA tasks including ARC, OpenBookQA, BoolQ, PIQA, HellaSwag, and Social IQa under a unified “reasoning + boxed answer” prompt template [2605.07244]. All regimes share matched-compute conditions: the same GRPO schedules, rollouts with $K=5$, GRPO hyperparameters, and deterministic evaluation decoding [2605.07244].

The reported outcomes distinguish sharply among the sharing regimes [2605.07244]:

| Regime | Reported empirical behavior | Reported overhead |
|---|---|---|
| PRP | Unstable; improves the stronger model early but collapses on the weaker | $\approx 1\times$ full rollout |
| XGRPO | Stable; often matches or slightly improves standalone GRPO, but cannot recover peer-only successes | $\approx 0$ |
| SGT | Best stability-support trade-off; stays $\geq$ GRPO curve, yields faster early gains, adds up to +3–5 pp final accuracy on MATH | $\approx 0.0065\times$ rollout |

Diagnostic observations reinforce this separation. PRP lowers entropy and raises KL, indicating strong coupling; XGRPO traces overlap almost exactly with GRPO, indicating low coupling; SGT preserves or slightly raises entropy, keeps KL low, and fires only on approximately 2–8% of “rescue” prompts where peer success rescues learner failure [2605.07244]. These findings are cited in support of the conclusion that, in the standard on-policy GRPO regime, outcome-level sharing occupies the favorable point of the stability-support trade-off [2605.07244].

Implementation is reported as being built on VERL with minimal changes, with SEE and THL plugging into the existing dataflow [2605.07244]. All source code, pseudocode, and detailed hyperparameters are described as archived for reproducibility [2605.07244]. Since the supplied source for the 2025 MEML-GRPO paper lacks methodology sections, the most explicit operational account available in the present record comes from this later formalization [2508.09670; 2605.07244].

## 7. Interpretation, limitations, and relation to RLVR

Within the available record, MEML-GRPO occupies a specific niche in RLVR: it is designed for settings where verifiable rewards are available but sparse, and where multiple experts can be arranged so that one expert’s success supplies useful information when another expert fails [2508.09670; 2605.07244]. Its central innovation is therefore not merely multi-policy training, but selective information transfer under verifiability constraints.

The main limitation of the public description is asymmetry between claim-level and mechanism-level detail. The original publication establishes the motivating problem, the use of diverse expert prompts, the existence of inter-expert mutual learning, and aggregate gains on Qwen and Llama, but the supplied source contains no technical description, definitions, equations, algorithms, or experimental details beyond the abstract [2508.09670]. By contrast, the later summary provides a full mutual-RL substrate with exact losses, alignment rules, theoretical claims, and benchmark diagnostics [2605.07244]. This suggests that current understanding of MEML-GRPO in the literature may be partly mediated by subsequent formalization.

A further point of interpretation concerns what “expert” denotes. In the 2025 abstract, experts are explicitly linked to “diverse expert prompts as system prompts” [2508.09670]. In the 2026 mutual-RL formalization, heterogeneity extends to separate tokenizers, objectives, and model families [2605.07244]. A plausible implication is that MEML-GRPO should be read as a general design principle for heterogeneous RLVR post-training rather than as a single narrow architecture.

Finally, the empirical and theoretical record given here does not support the view that more sharing is uniformly better. The later results instead indicate a structured hierarchy: data-level sharing can be unstable under misalignment, value-level sharing is stable but limited in support, and outcome-level sharing can exploit verified peer successes with low overhead and low coupling [2605.07244]. In that sense, MEML-GRPO is best understood as a selective mutual-learning framework for sparse-reward reasoning, grounded in GRPO and tailored to heterogeneous experts rather than a generic pooled-training recipe.

Source: https://www.emergentmind.com/topics/multi-expert-mutual-learning-grpo-meml-grpo