Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search

Abstract: We present an automated large-scale search pipeline for heterogeneous 4-Expert Mixture-of-Experts (MoE4) architectures within the LEMUR neural network dataset ecosystem. Building on a hand-crafted heterogeneous MoE reference model, we replace manual design with a deterministic code-assembly generator that systematically combines base architecture families drawn from the LEMUR database into MoE4 ensembles, each governed by a convolutional gating network with temperature scaling, mixup augmentation, and cosine-annealed learning rate scheduling. Over a 28-day campaign on an NVIDIA RTX 4090, the pipeline generated 4,463 candidate models across 197 batches, of which 1,021 were evaluated successfully. A critical finding emerged from the campaign: due to alphabetical enumeration via itertools.combinations, the entire explored search space (4.8% of the theoretical 23,751 possible 4-family combinations) is anchored to a single family, AirNet. We characterise this coverage bias precisely, identify the root cause in the generator, and propose a stratified random sampling fix. Within the AirNet anchored scope, ShuffleNet and MobileNetV3 consistently co-produce the highest-accuracy ensembles (mean accuracy up to 0.632), while FractalNet and MNASNet are identified as low-yield families warranting exclusion in future campaigns. The pipeline, analysis artefacts, and corrected generator are released as part of the open-source NNGPT project at https://github.com/ABrain-One/nn-gpt

• The paper demonstrates that an automated pipeline can systematically construct and evaluate 4-expert heterogeneous MoE models.
• It employs deterministic assembly with multi-stage syntax and forward-pass validation to overcome combinatorial design complexities, reaching up to 68% accuracy in a single epoch.
• The study identifies a search space bias in expert combinations and proposes stratified random shuffling to enhance unbiased model exploration and scalability.

Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search

Introduction

This paper presents an automated pipeline for the systematic search and evaluation of 4-expert heterogeneous Mixture-of-Experts (MoE) architectures in the LEMUR neural network dataset. The pipeline replaces manual expert combination design with deterministic code assembly, aiming to address the combinatorial complexity imposed by 29 distinct base architecture families. The search protocol leverages programmatic synthesis, multi-stage validation, and campaign automation to efficiently generate, verify, and register large numbers of unique heterogeneous MoE4 candidates.

Pipeline Architecture and Methodology

The design centers around four stages: (1) expert pool construction from rigorously filtered base architectures, (2) deterministic assembly of PyTorch-compliant MoE4 models, (3) multi-stage validation for syntax, forward-pass compatibility, and database uniqueness, and (4) campaign automation with persistent recovery mechanisms. Each candidate model integrates a convolutional HeterogeneousGate for expert routing, temperature scaling, mixup augmentation, and cosine-annealed learning rate scheduling. The wrapper ensures forward-pass robustness and hyperparameter consistency. GPU training and evaluation are orchestrated for maximal screening throughput, targeting 1,000 models over a 28-day campaign, with extensive logging and error tracking.

Experimental Setup

The evaluation leverages CIFAR-10, with each candidate MoE4 ensemble trained for one epoch at batch size 32 and learning rate 0.01 on an NVIDIA GeForce RTX 4090. Models undergo norm_256_flip preprocessing and mixup augmentation ($\alpha=0.2$). Campaign execution proceeds in 25-model batches; parameter thresholds and blacklisting procedures proactively minimize recurring out-of-memory (OOM) failures and duplicate entries. The campaign protocol does not seek full convergence but rather high-potential expert quartets for subsequent fine-tuning and downstream integration.

Numerical Results and Statistical Analysis

The campaign generated 4,463 models and successfully evaluated 1,021 unique MoE4 ensembles, exceeding the initial screening target. Early success rates diminished as the generator progressed into combinations with larger architectures, with OOM failures comprising 99.7% of all unsuccessful evaluations. The single-epoch accuracy distribution is left-skewed, with a mean of 0.5221, median of 0.5388, and a maximum of 0.6801.

Figure 1: Accuracy distribution across 1,021 successful MoE4 models (1 training epoch, batch=32, lr=0.01). Red line: mean (0.5221), dashed orange: median (0.5388), dotted green: maximum (0.6801).

The highest-performing model (AirNet + AlexNet + DPN68 + ResNet) attains 68.0% Top-1 accuracy in a single epoch, confirming that automated heterogeneous expert assembly is capable of producing competitive candidates without manual architecture tuning.

Family-level analysis identifies ShuffleNet and MobileNetV3 as the top-performing expert families (mean accuracies of 0.632 and 0.621, respectively), further supported by high success rates ($>$96%). FractalNet is a structural dead end, with a 0.7% success rate and frequent OOM due to recursive expansion at high input resolution. MNASNet consistently degrades ensemble accuracy, indicating its architecture is suboptimal for inclusion in multi-expert settings.

Figure 2: Mean and median accuracy per expert family in successful MoE4 models (families with $\geq$40 appearances; bars = mean, diamonds = median, whiskers = $\pm$1 SD).

Critical Search Space Coverage Bias

A pivotal discovery is the search space coverage bias rooted in the deterministic enumeration protocol. The use of lexicographical ordering (itertools.combinations(sorted(model_names), 4)) systematically anchors all attempted combinations to AirNet, the alphabetically first family, resulting in exploration of only 4.8% of the theoretical 23,751 possible combinations, all containing AirNet. This structural bias is quantitatively diagnosed and addressed by proposing stratified random shuffling, which would yield unbiased sampling and drastically increase search diversity.

Implications and Future Directions

The findings demonstrate the necessity for unbiased sampling strategies in large-scale architecture search, particularly when leveraging throughput-limited, deterministic enumeration. The identification of high-yield (ShuffleNet, MobileNetV3, ResNet) and low-yield (FractalNet, MNASNet) families provides actionable guidelines for expert pool construction. The pipeline's engineering framework, including robust validation and campaign state recovery, establishes a scalable template for subsequent exploration across larger combinatorial spaces and finer screening protocols.

The practical implications involve accelerated automated discovery of heterogeneous ensembles for vision tasks, resource-efficient screening, and database expansion. Theoretically, the pipeline operationalizes the synthesis and evaluation of structurally diverse expert combinations at scale, supporting ongoing research into conditional computation and architectural diversity in sparse MoE regimes. The future direction includes full search space coverage with unbiased sampling, memory-aware filtering to mitigate OOM failures, and multi-epoch evaluation for convergence assessment.

Conclusion

The paper delivers a comprehensive automated pipeline for heterogeneous MoE4 exploration, incorporating deterministic assembly, rigorous validation, and campaign automation. Statistical results reveal strong candidates capable of near state-of-the-art accuracy after minimal training. The decisive identification of search space coverage bias and actionable countermeasures epitomize methodological rigor in architecture search. The released pipeline and dataset artifacts supply a foundation for scalable, unbiased exploration of heterogeneous expert ensembles in vision systems (2606.23739).