CME-CAD: Heterogeneous Multi-Expert RL

• CME-CAD is a reinforcement learning paradigm that integrates diverse expert agents with different architectures and training data to jointly optimize policy performance.
• It employs explicit knowledge sharing, cross-expert distillation, and diversity regularization to overcome challenges like reward sparsity, exploration deficits, and mode collapse.
• The framework demonstrates state-of-the-art results in CAD code generation, multi-agent coordination, and sparse-reward exploration.

Heterogeneous Collaborative Multi-Expert Reinforcement Learning (CME-CAD) is a paradigm in reinforcement learning that systematically integrates a portfolio of diverse expert agents—differing in architecture, training data, reasoning style, and/or input modalities—into a unified collaborative framework. CME-CAD leverages the complementary strengths of each expert through explicit knowledge-sharing, cross-expert regularization, and joint optimization. This approach addresses limitations in conventional RL, such as poor sample efficiency, reward sparsity, exploration deficits, mode collapse, and limited inductive bias coverage, by orchestrating expert diversity and distillation under robust RL objectives. CME-CAD systems have established new state-of-the-art results in complex domains such as CAD code generation, multi-agent coordination, and sparse-reward exploration.

1. Architectural Principles and Collaboration Mechanisms

CME-CAD frameworks maintain a pool of $M$ heterogeneous experts $\{\pi_1, \ldots, \pi_M\}$, where each expert $\pi_i$ is parameterized as a distinct policy network that specializes in a unique subdomain, input-processing style, or architectural bias (Jia et al., 13 Aug 2025, Niu et al., 29 Dec 2025). Heterogeneity arises from:

• Distinct system prompts (in LLM-based settings), unique pretraining/fine-tuning data, or curated demonstration subsets per expert.
• Divergent neural architectures (e.g., transformer, RNN, or hybrid topologies).
• Model capacity or inductive bias variations.

During training and inference, each expert operates independently to generate trajectories or candidate solutions (e.g., policy rollouts or Chain-of-Thought (CoT) traces), often under its personal conditioning template or prompt. A collaborative “topology”—typically a fully connected peer-to-peer, but also plausible as ring/star structures—facilitates mutual learning where all experts serve as both teachers and students. Information is aggregated either through selector networks, value-based routing, or ensemble-style voting. In advanced instantiations, shared memory or communication channels enable cross-expert gradient or state-action posting (Jia et al., 13 Aug 2025).

2. Formal Optimization Objectives

CME-CAD systems optimize a composite loss that encompasses independent reward maximization, cross-expert alignment, and diversity preservation. The core per-expert objective is a KL-regularized policy gradient—sometimes referred to as Generalized Reverse-KL Policy Optimization (GRPO) (Jia et al., 13 Aug 2025, Niu et al., 29 Dec 2025): $$L_{\mathrm{RL}}(\pi_i) = -\mathbb{E}_{\tau \sim \pi_i} \Big[ r(\tau) - \beta \cdot \mathrm{KL}\big(\pi_i(\cdot|s)\,\|\;\pi_{\text{ref}}(\cdot|s)\big) \Big]$$ where $r(\tau)$ is the (possibly sparse or structured) environment reward, $\pi_{\text{ref}}$ is a fixed pretrained reference, and $\beta$ tunes policy conservatism.

Inter-expert mutual learning employs pairwise distillation terms: $$L_{\text{mutual}} = \sum_{i \neq j} \lambda_{ij}\;\mathbb{E}_{s \sim D}\big[ \mathrm{KL}( \pi_i(\cdot|s)\;\|\;\pi_j(\cdot|s) ) \big]$$ with $\lambda_{ij}$ modulating the direction and strength of knowledge transfer. To enforce expert diversity and avoid mode collapse, diversity regularizers such as negative Jensen-Shannon divergence or collision-avoidance penalties are introduced: $$L_{\mathrm{CAD}} = \gamma \sum_{i < j}\; \mathbb{E}_{s} \left[ \exp(-\mathrm{JS}(\pi_i(\cdot|s)\,\|\;\pi_j(\cdot|s))) \right]$$ (Jia et al., 13 Aug 2025). The total CME-CAD loss integrates these terms and can further include centralized value estimation losses when actor–critic coordination is enabled.

3. Training Algorithms and Implementation Workflow

CME-CAD training typically proceeds in two major phases (Niu et al., 29 Dec 2025):

1. Multi-Expert Fine-Tuning (MEFT): Each expert undergoes supervised learning to refine its unique reasoning style and solution patterns based on high-quality demonstration data. This step establishes baseline competence and maximizes intra-expert diversity.
2. Multi-Expert Reinforcement Learning (MERL): Experts are trained in parallel under RL objectives (e.g., GRPO), with per-expert advantage estimation (advantage truncation, mean-normalized rewards), inter-expert KL distillation, and hard-negative instance buffering. Collaboration is enhanced by periodically fine-tuning on hard cases where all experts underperform, and explicitly shaping rewards to encourage software executability, geometric metric attainment (e.g., CAD IoU), and constraint satisfaction.

The primary update cycle alternates between collecting expert-specific trajectories, computing per-expert RL and mutual distillation losses, and synchronously updating expert parameters. When a centralized critic is employed, an auxiliary regression loss is added for state-value or Q-function consistency across experts (Jia et al., 13 Aug 2025). Key hyperparameters—including the strength of KL and diversity regularization, learning rates, and curriculum design—are tuned through validation sweeps or grid/Bayesian optimization.

4. Diversity Induction, Knowledge Transfer, and Critic Integration

Preserving meaningful heterogeneity among experts is critical to avoid “mode collapse” where experts converge to indistinguishable policies. Mechanisms for maintaining expert diversity include:

• Diversity regularization via negative JS losses or explicit collision-avoidance penalties.
• Distinct data, architecture choices, or random initializations for each expert.
• Hard-negatives buffer and adversarial state selection to challenge all experts on difficult instances.

Mutual distillation (pairwise or all-to-all) is central for collaborative improvement: experts transfer high-probability action distributions to peers, allowing lagging experts to adopt strategies observed to be successful in other “views.” When equipped, a centralized critic aggregates experience from all experts, providing shared value estimation and improving sample efficiency and variance reduction. Such critic backbones can use parameter sharing and agent-ID conditioning (Andres et al., 2022) to allow varied expert specializations without sacrificing batch-induced generality.

5. Practical Applications and Representative Benchmarks

CME-CAD has demonstrated empirical advantage in multiple domains:

• CAD Code Generation: CME-CAD achieves 80.7% Intersection-over-Union (IoU) and 98.25% executability rate on the CADExpert benchmark by combining heterogeneous LLMs, expert-internal advantage estimation, hard-negative replay, and multi-expert KL (Niu et al., 29 Dec 2025).
• Reasoning with Verifiable Rewards: The MEML-GRPO framework, a forerunner to CME-CAD, overcomes reward sparsity in RLVR by diverse expert prompting and mutual distillation, yielding 4.89%–11.33% performance gains over single-expert RLVR (Jia et al., 13 Aug 2025).
• Multi-Agent and MARL: Modular implementations use personalized expert demonstration buffers and dual-discriminator (behavior/dynamics) shaping for efficient collaborative learning among heterogeneous agents (Yu et al., 2024).
• Sparse-Reward Exploration: Centralized critics with intrinsic curiosity modules have shown 30–40% improved convergence rates and robust negative transfer prevention in ViZDooM-like environments (Andres et al., 2022).
• Continuous Control: Hybrid ensembles of off-policy, on-policy, and evolutionary experts (e.g., SAC/PPO/CEM) improve MuJoCo returns and stability by integrating policy transfer, mixed replay, and hierarchical memory relay (Zheng et al., 2020).

Framework	Core Collaboration	Diversity Mechanism	Notable Domain
CME-CAD (Niu et al., 29 Dec 2025)	KL distillation	Trunc. adv., HSB, KL	CAD code generation
MEML-GRPO (Jia et al., 13 Aug 2025)	Mutual KL	Diversity regularizer	RLVR, LLMs
PegMARL (Yu et al., 2024)	Demonstration distil	Occupancy filtering	Multi-agent RL
CHDRL (Zheng et al., 2020)	Policy transfer	Agent specialization	MuJoCo control
CDRL (Lin et al., 2017)	Deep knowledge distil	Alignment net	Atari, A3C

6. Design Guidelines and Implementation Considerations

Best practices for CME-CAD development include:

• Initialize from independent, high-performing expert fine-tuning to maximize behavioral diversity.
• Gradually introduce mutual distillation (increasing $\lambda_{ij}$) to propagate expertise without inducing premature consensus.
• Enforce diversity regularization ($L_{\mathrm{CAD}}$, negative JS, hard-negatives) at all collaboration stages.
• Incorporate centralized critics or value estimators after stabilization for variance reduction.
• Carefully tune reward gating, format enforcement, and metric weighting to balance executability with geometric or task-specific constraints (Niu et al., 29 Dec 2025).
• Use buffer-based or prioritized replay to focus training on difficult or underexplored regions.

CME-CAD provides extensible blueprints for scaling to larger expert pools, hierarchical ensembles, and adaptive communication via selectors or meta-controllers. A plausible implication is that CME-CAD frameworks can subsume classical ensemble and multi-agent RL paradigms by explicitly modeling both diversity and collaboration at the policy-update level.

7. Limitations, Empirical Insights, and Future Directions

Limitations of current CME-CAD implementations include elevated system complexity, increased computational cost due to parallel expert rollouts and distillation overhead, and challenges in hyperparameter scheduling (e.g., KL/diversity strengths, advantage mixture rates). Empirical ablations indicate that expert-internal advantage estimation and hard-negative replay offer the most significant singular performance gains in practical systems (Niu et al., 29 Dec 2025).

Common misconceptions include the belief that naive ensembling or inference-time voting among experts can match the gains of joint optimization—CME-CAD demonstrates that collaborative, regularized joint training is essential for maximal performance. Future research is likely to explore meta-learned communication topologies, deeper integration of model-based experts, domain-adaptive curricula, and stronger theoretical convergence guarantees.

CME-CAD unifies and generalizes collaborative reinforcement learning by integrating portfolio diversity, structured knowledge sharing, and reward-driven policy correction into a scalable, principled framework, with demonstrated advances across reasoning, code generation, multi-agent interaction, and sparse-reward exploration (Jia et al., 13 Aug 2025, Niu et al., 29 Dec 2025, Yu et al., 2024, Andres et al., 2022, Zheng et al., 2020).