AxMoE: Characterizing the Impact of Approximate Multipliers on Mixture-of-Experts DNN Architectures

Abstract: Deep neural network (DNN) inference at the edge demands simultaneous improvements in accuracy, computational efficiency, and energy consumption. Approximate computing and Mixture-of-Experts (MoE) architectures have each been studied as independent routes towards efficient inference, the former by replacing exact arithmetic with low-power approximate multipliers, the latter by routing inputs through specialized expert sub-networks to enable conditional computation. However, their interaction remains entirely unexplored. This paper presents AxMoE, the first study of the impact of approximate multiplication on MoE DNN architectures. We evaluate three MoE variants: Hard MoE, Soft MoE, and Cluster MoE against dense baselines across three CNN architectures (ResNet-20, VGG11_bn, VGG19_bn) on CIFAR-100 and a Vision Transformer (ViT-Small) on Tiny ImageNet-200 dataset, using eight 8-bit signed multipliers (including one exact baseline) from the EvoApproxLib library. Results show that, without retraining, the Dense baseline is the most resilient topology across all CNN architectures, whereas on ViT-Small, all topologies degrade at comparable rates regardless of routing strategy. After approximate-aware retraining, recovery varies substantially across architectures, topologies, and multipliers. ResNet-20 achieves full recovery across the entire multiplier range, whereas VGG architectures recover at moderate multipliers but fail irreversibly at aggressive ones for all topologies except Cluster MoE on VGG11_bn; on ViT-Small, Hard MoE outperforms Dense under aggressive approximation at equal normalized inference cost. These results pave the way for future approximate MoE hardware-software co-design strategies.

• The paper shows that integrating approximate multipliers in MoE DNN architectures leads to topologically dependent trade-offs in accuracy and power consumption.
• Methodologically, it assesses various MoE routing strategies on CNNs and ViTs using approximation-aware retraining to understand error resilience.
• Results indicate that while dense configurations perform best under moderate conditions, Hard MoE in ViTs can outperform dense setups under aggressive approximation.

AxMoE: Assessing Approximate Multipliers in Mixture-of-Experts Deep Architectures

Introduction and Motivation

AxMoE addresses a previously unexplored intersection of two efficiency paradigms in DNN inference: Approximate Computing (AxC) and Mixture-of-Experts (MoE) architectures. AxC exploits DNNs' inherent error resilience by leveraging reduced-accuracy arithmetic units such as approximate multipliers to minimize energy and silicon cost. In contrast, MoE strategies partition models into expert subnetworks and introduce dynamic sparsity through input-dependent routing. This combination is especially appealing for deployment on energy-constrained edge devices, yet prior investigations have treated AxC and MoE orthogonally. AxMoE systematically examines the algorithmic and architectural interplay between these domains and probes how error propagation, retraining recovery, and efficiency trade-offs behave for different MoE topologies under approximate arithmetic.

Experimental Design and Methodology

The study evaluates three principal MoE routing contexts—Hard MoE (Top-1 expert selection), Soft MoE (weighted fusion of all experts), and Cluster MoE (input-level gateway routing)—implemented within ResNet-20, VGG11_bn, VGG19_bn on CIFAR-100, and ViT-Small on Tiny ImageNet. Eight 8-bit signed multipliers, ranging from precise (mul8s_1KV6) to the most aggressive in the EvoApproxLib library (mul8s_1L2L), serve as the approximation backbone. Approximate computations are isolated to dominant MAC-consuming layers (Conv2D for CNNs, Linear for ViT blocks), with all routing gates retaining full precision.

Power consumption is normalized relative to a Dense exact-multiplier baseline, factoring both dynamic MAC activation and per-operator energy profiles. Approximate operations are emulated via GPU-efficient LUTs, and for retraining, an approximation-aware fine-tuning protocol retrains experts while freezing router weights, ensuring routing is not co-adapted to hardware artifacts.

Numerical Results and Analysis

Baseline Accuracy and MAC Cost

Soft MoE generally yields the highest baseline Top-1 accuracy in shallow CNNs, but with more substantial effective MACs and memory requirements. For deeper architectures (VGG19_bn), all-expert Soft routing yields diminishing or negative returns on accuracy while incurring substantial cost. Hard MoE consistently lags Dense and Soft in accuracy, with its deficit growing in deeper networks. Cluster MoE introduces the highest overhead with the lowest test accuracy, especially pronounced in ViT-Small, confirming that gateway networks struggle to rival per-layer routers in both selection granularity and efficiency.

Impact of Approximate Multipliers Without Retraining

Dense CNNs outperform all MoE topologies under intrinsic approximation resilience: for example, at an aggressive L2J setting, ResNet-20 Dense maintains ~59% accuracy compared to ~13% for Hard and ~9% for Cluster, with VGG MoE variants universally collapsing below 20% by moderate approximation. An exception is observed at specific multipliers (e.g., KVM), where Soft MoE briefly maintains parity with exact baselines. ViT-Small exhibits synchronized degradation for all topologies due to the inescapable serialization of approximation in critical shared QKV projections, giving routing strategy negligible influence before retraining.

Figure 1: Accuracy degradation versus approximation aggressiveness, comparing absolute Top-1 accuracy without and with retraining for 5 epochs.

Retraining and Recovery Dynamics

Approximate-aware retraining delivers disparate benefits depending on architecture and multiplier severity.

• ResNet-20: All variants fully recover baseline accuracy even under the harshest approximations (e.g., L2L).
• VGG11_bn/VGG19_bn: Dense and most MoEs recover accuracy for moderate multipliers; only Cluster MoE in VGG11_bn demonstrates partial resilience at maximal approximation levels (52.0% at KVA, 42.5% at L2L). Notably, in VGG19_bn Hard MoE, accuracy collapse at KVA is not monotonic with error probability, indicating qualitative differences in error sensitivity or retraining efficacy between hardware designs.
• ViT-Small: Post-retraining, Hard MoE (especially r=0.5) demonstrates the flattest accuracy-power degradation. Dense degrades by 6.46 points (L2L), whereas Hard MoE (r=0.5) loses only 1.74 points. This owes to patch-level routing granularity in ViT, where each expert adapts to a consistent subset during retraining, facilitating more stable updates than the more spatially entangled CNN setting.

Power-Accuracy Trade-off and Pareto Analysis

Pareto-optimal trade-offs are strongly model- and topology-specific. For CNNs, Dense with the L2J multiplier always lies on the Pareto front (29.2% per-op savings, <0.7 percentage points loss), and none of the MoE topologies can challenge this dominance at equal or lower power. Soft MoE can gain modest accuracy but only at a prohibitive cost in energy, exceeding the Dense baseline's exact power even with maximal approximation. For ViT-Small, the Pareto surface features both Dense and Hard MoE (r=0.25), with Hard MoE (L2L) achieving 75.97% versus Dense’s 75.00% at strictly lower normalized power (0.485×). Thus, Hard MoE is uniquely able to outperform Dense in both accuracy and power under aggressive approximation, a claim not matched by any other configuration in the study.

Figure 2: Top-1 accuracy after approximate-aware retraining against normalized power consumption for all MoE variants.

Cluster MoE is always Pareto-dominated in all regimes, being both less accurate and more power-hungry.

Theoretical and Practical Implications

These results establish that error propagation behavior in approximate-aware MoEs is intricately topology- and architecture-dependent. Adaptive retraining is effective across a wide approximation spectrum in configurations with robust gradient pathways (i.e., ResNet-20 with skip links), but shows abrupt phase transitions in recovery in deeper VGGs or for particular multiplier error profiles. ViT’s architectural separation of attention and FFN routing layers means global approximation error is less mitigable by MoE routing, yet MoE's adaptation mechanism still offers power-normalized accuracy advantages that are unreachable by Dense alone.

For practitioners, this suggests:

• On edge devices, deploying approximate multipliers in Dense configurations of robust CNNs is Pareto-optimal when aggressive but not maximal approximation is applied.
• On vision transformers, MoE (Hard, low routing ratio) outperforms Dense at equivalent or lower power under highest-grade approximation, especially post retraining.
• Cluster MoE, despite being compelling in supervised partitioning, lacks competitive efficiency or resilience in approximate settings.

Future Research Directions

These observations highlight several promising research trajectories:

• Hardware-software co-design targeting architectural sweet spots (e.g., CNNs with extensive skip connections, transformers with patch-level MoE granularity).
• Dynamic incremental retraining strategies, possibly leveraging model uncertainty metrics to optimize approximation-specific adaptation.
• Extending analysis to mixed-precision and heterogeneous approximation schedules that adjust error-criticality at layer or expert granularity.
• Investigating router co-adaptation, as current retraining leaves route weights fixed—adapting routing logic to hardware faults may further improve resilience.

Conclusion

AxMoE provides the first fine-grained empirical characterization of how approximate multipliers interact with Mixture-of-Experts DNN architectures. The results demonstrate that the power-accuracy landscape is highly topology-dependent, invalidating extrapolations from dense-only studies. Specifically, while dense CNNs remain optimal under moderate approximation and retraining, in vision transformer models, selective dynamic MoE routing can surpass dense baselines in both accuracy and normalized power under aggressive approximate arithmetic. These findings pave the way for principled, architecture-aware approximate hardware-software codesign in next-generation efficient DNN inference.