Zero-Computation Experts

• Zero-computation experts are specialized function blocks that perform conditional, parameter-light operations with negligible floating-point computation.
• They integrate into Mixture-of-Experts systems by routing easy or less influential tokens to fixed-function experts, reducing computation and communication overhead.
• Their deployment in models can yield 20–40% reduction in FFN FLOPs and improved throughput while maintaining performance.

A zero-computation expert is an operator or function block within a larger machine learning or optimization framework that contributes a token-, input-, or region-specific action while incurring negligible or zero floating-point computation and containing little or no learned parameters. Their principal role is to enable conditional skipping or affine adjustment of computation, realize extreme model sparsity, or enable amortization of costly decision-making by relegating “easy” or uninfluential cases to parameter-free or fixed-function logic. Zero-computation experts have arisen independently in distributed Mixture-of-Experts (MoE) architectures, input-partitioned piecewise-constant regression, post-hoc sparsity transformations in transformers, Bayesian optimization with human-in-the-loop, and other modalities.

1. Formal Definitions and Principal Classes

Zero-computation experts were systematically defined in MoE++ as operators $E(x)$ satisfying at least one of the following: (i) output is a deterministic function requiring $O(1)$ FLOPs; (ii) parameters are negligible or absent; (iii) inclusion in the expert mixture does not induce extra communication or load balancing demands (Jin et al., 2024).

The canonical types are:

• Zero expert (discard): $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$.
• Copy expert (skip): $E_{\rm copy}(x) = x$.
• Constant expert (replace/adjust):

$$E_{\rm const}(x) = \alpha_1 x + \alpha_2 v, \quad \left[\alpha_1, \alpha_2\right] = \mathrm{Softmax}(W_c x)$$

with $W_c \in \mathbb{R}^{2 \times D}$, $v \in \mathbb{R}^D$.

In pure piecewise-constant regression MoE (termed “ZC‐1SMoE” (Dar, 3 Oct 2025)), each region is assigned a constant predictor, and at inference, the only operation is an input-region lookup. For dynamic routing contexts, “zero experts” generalize to any function acting as the null element or unchanged pass-through (Jin et al., 2024, Lv et al., 18 May 2026).

2. Integration in Mixture-of-Experts Architectures

In advanced MoE layers, input tokens $x \in \mathbb{R}^D$ are routed via a learned gating function $G(x)$ to a top-$K$ subset of $O(1)$0 experts. Each expert is either a compute-intensive feed-forward network (FFN) or a zero-computation expert. The output is

$O(1)$1

MoE++ incorporates the zero, copy, and constant experts into this mixture, enabling per-token reduction of FFN computation. Routing weights for zero experts are computed identically to others and, if selected, their (zero or trivial) output enters the sum (Jin et al., 2024).

Post-training MoE adaptation, as in ZEDA (Lv et al., 18 May 2026), achieves a similar effect by adding parameter-free zero-output experts to each MoE layer, allowing the routing mechanism to select zero experts and thus skip costly computation at inference. In these approaches, the router’s logits for zero experts are initialized to match the empirical distribution of real experts, ensuring a stable architectural transition.

3. Routing, Dynamic Adaptation, and Token-wise Computation

In MoE++ and its variants, the token-level routing mechanism is extended to consider both standard FFNs and zero-computation experts. Gating residuals, implemented as

$O(1)$2

where $O(1)$3 and $O(1)$4 are gating matrices, enhance routing stability by permitting each token to use pathway information from previous layers (Jin et al., 2024).

Empirically, linguistic analysis shows that tokens corresponding to “easy” constructs (e.g., punctuation, fragments) are routed to fewer or exclusively zero-computation experts, whereas content words (verbs, rare nouns) activate full FFN compute (Jin et al., 2024). In ZEDA, group-level balancing losses

$O(1)$5

control the real-to-zero expert load ratio, enabling direct trade-off between speed and accuracy (Lv et al., 18 May 2026).

4. FLOP, Latency, and Memory Effects

Zero-computation experts deliver order-of-magnitude FLOP savings within MoE or similar sparse architectures. Suppose a fraction $O(1)$6 of tokens utilize FFN experts, then

$O(1)$7

FLOPs/tok, compared to $O(1)$8 for vanilla MoE. In practice (with, e.g., $O(1)$9), this translates to 20–40% reduced FFN FLOPs and empirical 1.1–2.1$E_{\rm zero}(x) = 0 \in \mathbb{R}^D$0 throughput improvement, with token-level compute skippable for simple inputs (Jin et al., 2024, Lv et al., 18 May 2026).

Zero-computation experts eliminate cross-GPU communication, as their parameters are trivial and replicated per device, and help to ameliorate expert-load imbalance by reducing straggler risk. Practical deployments (e.g., MoE++ 7B on 1T tokens) sustain such improvements without model scaling loss and outperform MoEs of significantly higher FLOP cost (Jin et al., 2024).

5. Theoretical Perspectives and Non-Parametric Zero-Compute Experts

In quantization-inspired regression MoEs (“ZC-1SMoE”), a large number $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$1 of non-overlapping regions $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$2 are assigned constant-valued zero-compute experts. The only inference operation is input-region mapping; a computation-free “expert” returns the regional mean or assigned value. The approximation error exhibits optimal $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$3 scaling in $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$4 dimensions, with a classic bias-variance tradeoff between the number of experts and sample size. The exact decomposition is

$E_{\rm zero}(x) = 0 \in \mathbb{R}^D$5

with minimum test error for $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$6, achieving principled sample-complexity trade-offs (Dar, 3 Oct 2025).

The segmentations $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$7 concentrate zero-compute experts (i.e., region density) where the signal varies most or where sampling density is highest, paralleling the “easy vs. hard” routing in MoE++ (Jin et al., 2024).

6. Post-hoc and Zero-Shot MoE Transformations

MLPMoE introduces a distinct, static zero-computation expert setting by partitioning the weights of standard dense MLP layers into $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$8 functionally independent “experts,” summing their outputs to recover the original dense output. By overlaying structured sparsity (Fractal Fade) and pruning (Compensated Pruning), up to 20% of parameters and corresponding computation can be “zeroed” or skipped, with proxy perplexity staying within 2% of the dense baseline for LLMs up to 8B parameters (Novikov, 26 Nov 2025).

No further loss computation is needed for pruned branches if the hardware kernel supports explicit skipping; otherwise, bypassed branches still incur dummy compute, limiting realized speedups. The method is entirely function-preserving and operates without gradient updates, calibration data, or router training.

7. Human-in-the-Loop Expert Optimization

Zero-computation expert concepts extend to Bayesian optimization with human feedback. COBOL, in expert-augmented Bayesian optimization, introduces a handover guarantee demonstrating that the cumulative count $E_{\rm zero}(x) = 0 \in \mathbb{R}^D$9 of expert queries grows sublinearly: $E_{\rm copy}(x) = x$0, so $E_{\rm copy}(x) = x$1 over time. Thus, the expert functions asymptotically as a “zero-computation” advisor, queried vanishingly often, while optimizing convergence rate is never worse than pure Bayesian optimization (no-harm guarantee). Empirical results confirm that in all tested regimes, expert label budgeting drops sharply after an initial learning phase (Xu et al., 2024).

8. Practical Impact, Limitations, and Extensions

Zero-computation experts have demonstrated several engineering and modeling advantages:

• Substantial reduction in forward FLOPs and communication overhead in MoE models (Jin et al., 2024, Lv et al., 18 May 2026).
• Dynamic, per-token or per-input routing and expert selection enables compute to be focused on hard cases, increasing statistical and training efficiency (Jin et al., 2024, Lv et al., 18 May 2026).
• Zero-shot architectural conversion of dense blocks to static MoE with simple slicing and structured sparsity, preserving function and reducing memory/compute (Novikov, 26 Nov 2025).
• Provable sample-complexity optimality in regression by exploiting nonparametric partitioning with constant (zero-compute) experts (Dar, 3 Oct 2025).
• In expert-in-the-loop optimization or human label allocation, formal asymptotic guarantees of zero expert utilization without harming performance (Xu et al., 2024).

Not all generic sparsification is beneficial: e.g., copy or bypass experts can disrupt output scale and direction in certain MoE post-hoc settings, motivating a strong preference for strict zero-output experts in practice (Lv et al., 18 May 2026).

A plausible implication is that future hardware and model design should optimize for explicit expert skipping, as realized speedup otherwise lags theoretical FLOP reduction. Open directions include extension to >100B scale LLMs, retrieval-augmented routing, direct hardware support for fine-grained expert skipping, and application to non-NLP modalities.

---

References:

(Jin et al., 2024, Dar, 3 Oct 2025, Novikov, 26 Nov 2025, Lv et al., 18 May 2026, Xu et al., 2024)