Runtime Mixture-of-Models (MoM) Overview

• Runtime Mixture-of-Models (MoM) is a family of inference-time architectures that dynamically select whole pretrained models, model layouts, or composite agents based on query requirements.
• It enables adaptive computation by routing queries using capability, difficulty, and cost metrics, thereby improving quality and reducing latency.
• MoM systems are applied in cost-sensitive serving, bursty traffic management, and ensemble reasoning, as demonstrated in frameworks like Brick, Moebius, and NSED.

Runtime Mixture-of-Models (MoM) denotes a class of inference-time architectures in which model-scale computation is selected, switched, or composed during execution rather than fixed as a static deployment choice or as a pretrained token-level routing pattern. In recent arXiv usage, the term covers at least three distinct but related formulations: deploy-time routing of each query to one whole backend model, seamless switching between tensor-parallel and expert-parallel layouts of one Mixture-of-Experts model, and recurrent deliberation among heterogeneous agents assembled for a session (Massa et al., 11 Jun 2026, Wang et al., 25 Jun 2026, Pecerskis et al., 23 Jan 2026). Related work also uses the acronym “MoM” for “Mixture-of-Memories” and “Mixture-of-Modules”; those are not the same formulation, but they share the emphasis on runtime sparse selection and dynamic computation graphs (Du et al., 19 Feb 2025, Gong et al., 2024).

1. Conceptual scope and relation to Mixture-of-Experts

A recurring distinction in the literature is between Mixture-of-Experts (MoE) and Runtime Mixture-of-Models. In MoE, sparse sub-networks inside a single model are activated token-by-token; in MoM, the routed or switched unit is larger. Brick states this explicitly as “the unit of routing is a model, not a layer,” while NSED presents itself as a runtime alternative to traditional static Mixture-of-Experts, and Moebius frames the serving problem as a runtime Mixture-of-Models-style adaptation problem in which the “current model layout” changes live (Massa et al., 11 Jun 2026, Pecerskis et al., 23 Jan 2026, Wang et al., 25 Jun 2026).

Formulation	Routed or switched unit	Runtime action
Brick	one whole model per query	choose one model from a heterogeneous pool
Moebius / AsymEP	TP or EP layout of the same model	switch layouts between decode iterations
NSED	subset of heterogeneous agents	assemble, evaluate, and refine over rounds
Mixture-of-Memories / Mixture-of-Modules	memory states or modules	top-$k$ routing or dynamic assembly

This comparison shows that Runtime MoM is not a single architecture. In one line of work, the system performs deploy-time selection over a fixed pool of pretrained models. In another, the system switches between two serving modes of one shared model state. In a third, the system constructs a temporary composite model from interacting agents. A plausible implication is that Runtime MoM is better understood as a family of runtime control schemes over model-scale computational units than as one canonical topology.

2. Query-level routing over heterogeneous pretrained models

Brick presents MoM as a deployment architecture in which the router chooses one whole model per query from a heterogeneous pool, and its stated objective is to choose the cheapest model that can answer the query correctly (Massa et al., 11 Jun 2026). The router maps both queries and models into the same six-dimensional capability space: coding, creative synthesis, instruction following, math reasoning, planning/agentic, and world knowledge. Its runtime pipeline is deterministic: trim and truncate the query for routing-stage classification only; apply high-precision keyword rules to bias capability inference; run a fine-tuned ModernBERT classifier to get a six-dimensional capability distribution $p(x)$; run a separate complexity classifier to estimate query difficulty; compute a score $J_m$ for each model $m$; and choose the model with minimal score,

$m^\star = \arg\min_m J_m .$

The capability-and-difficulty formulation is expressed in a shared log-odds space. For each query, the router estimates a blended difficulty $\tau_q \in (0,1)$ and transforms it as

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$

The required capability for dimension $c$ is

$r_{q,c} = p_c\, z_q,$

and the model capacity is

$v_{m,c} = p_c\, \mathrm{logit}(s_{m,c}),$

where $p(x)$0 is the model’s calibrated success rate on that capability. Under-capacity and over-capacity are then separated: $p(x)$1 and combined into

$p(x)$2

The paper exposes a continuous preference knob

$p(x)$3

with named settings $p(x)$4 for min / eco, $p(x)$5 for neutral / balanced, and $p(x)$6 for max / pro.

Brick is explicitly pure routing rather than a cascade. It does not probe a cheap model and then escalate after inspecting its answer. The full query is forwarded unchanged to the selected backend model. This architectural choice matters for agents because the paper argues that cascades add sequential probe-and-escalate overhead on every step of an agent loop, whereas Brick pays one routing decision per step and makes one model call per step.

The benchmark is Brick2 Dataset A, a stratified routing benchmark of 5,504 queries, evaluated on a fixed three-model pool: qwen3.5-9b, deepseek-v4-flash, and kimi2.6. The reported single-model accuracies are 63.17% for always-qwen, 73.69% for always-ds4, and 75.02% for always-kimi, with a three-model oracle ceiling of 83.25%. At the named operating points, Brick reports 63.17% accuracy at $p(x)$7 average cost for neutral, and 76.98% accuracy at $p(x)$8 average cost for max-quality. The neutral operating point is reported as about 4.71× cheaper than always-kimi for only 0.91 pp lower accuracy, and the max-quality setting beats always-kimi’s 75.02% while being about 28% cheaper per call. Median end-to-end latency drops from 51.246 s for always-kimi to 22.802 s for Brick at max quality, while the paper reports a production median router decision time of 494 ms.

The paper also defines the limits of this formulation. Its skill matrix and knob settings are calibrated on Dataset A itself using held-in folds; strict out-of-distribution generalization is still unproven. The results are shown on a fixed three-model pool, the cost scalars are tied to the pricing landscape at evaluation time, and routing quality depends on the ModernBERT capability classifier and the separate complexity classifier.

3. Runtime layout switching as a MoM-style serving system

Moebius addresses a different problem: a single MoE model may require multiple GPUs operating in parallel, but the optimal parallelism layout changes with concurrency (Wang et al., 25 Jun 2026). The paper states that tensor parallelism (TP) is faster at low concurrency, whereas expert parallelism (EP) wins at high concurrency. Production workloads cross this boundary continually: online serving sees bursty arrivals that subside into quiet periods, and reinforcement-learning rollouts begin as a high-concurrency burst that decays into a long tail of stragglers. The paper therefore frames the problem as a runtime Mixture-of-Models-style adaptation problem in which the serving engine must switch layouts on the fly as the “current model layout” changes, without stopping the system.

The key conceptual claim is that TP and EP are not treated as two different models, but as two layouts of the same model. The paper states that EP and TP compute the same function over byte-identical expert weights and KV cache; what changes is only ownership and sharding. Under TP, each rank holds a shard of every expert, and attention is tensor-parallel. Under EP, each rank owns whole experts plus a disjoint slice of requests, and attention is data-parallel. The switch is therefore not semantic recomputation; it is a redistribution of state.

The layout distinction is given explicitly. Let $p(x)$9 be total experts, $J_m$0 the switching group size, $J_m$1 hidden size, and $J_m$2 intermediate size. Under EP, each rank owns $J_m$3 whole experts, with

$J_m$4

Under TP, each rank owns one shard of every expert, with

$J_m$5

On the request side, under EP each rank owns a disjoint subset of requests and stores their full KV cache; under TP, every rank serves every request but only stores the KV head shard it owns.

A switch happens between decode iterations, not mid-kernel, and it does not restart the engine or drop in-flight requests. Before the switch, the coordinator chooses the target layout using the current global in-flight request count. All ranks then transition together: the execution engine selects the already-prepared runtime for the new mode, request ownership is remapped, and the memory manager reshards expert weights and KV cache into the destination layout. The paper emphasizes that request metadata, sampling state, and KV values migrate together, so decoding resumes at the same position after the switch.

A central systems choice is to keep both layouts’ control-plane runtime resident while only one data-plane copy is live. The control plane includes CUDA graphs, communication groups, and attention metadata. To keep captured CUDA graphs valid, AsymEP uses a unified memory manager that pre-allocates one large contiguous GPU buffer and exposes stable tensor views into fixed slots. For expert weights, it reserves $J_m$6 slots for $J_m$7 layers. TP maps layer $J_m$8 to slot $J_m$9, while EP maps layer $m$0 to slot $m$1; this one-slot offset ensures source and destination do not collide during a switch.

Resharding is implemented with fused GPU-to-GPU direct-transfer kernels rather than a conventional staged collective path. For expert weights, the paper decomposes resharding into a local permute, an All-to-All, and a local scatter, but reports that only two stages are needed in each direction. In EP$m$2TP, each rank packs complete experts into per-peer chunks and transfers them so each destination rank receives its shard of every expert already in place. In TP$m$3EP, transfer happens first and local repacking reconstructs complete experts afterward. KV transfer uses page-table-guided gather/exchange/scatter logic, because pages are scattered and irregular.

The switch signal uses asymmetric hysteresis: $m$4 where TP$m$5EP happens when the count exceeds high threshold $m$6, and EP$m$7TP happens only when the moving average over the last $m$8 iterations falls below low threshold $m$9. A cooldown $m^\star = \arg\min_m J_m .$0 limits switch frequency. For rollout workloads, the paper sets $m^\star = \arg\min_m J_m .$1 and $m^\star = \arg\min_m J_m .$2, since the batch decays monotonically.

The reported results are on 8×H200 GPUs serving Qwen3-235B-A22B. In the static sweep, TP/TP wins by about $m^\star = \arg\min_m J_m .$3 at small batch sizes, while EP/EP wins by about $m^\star = \arg\min_m J_m .$4 at large batches, with the crossover between batch sizes 128 and 256. AsymEP tracks the better side at every point. On dynamic online serving, it switches four times over a 375 s trace, improving both burst-time TTFT and quiet-period TPOT. On RL rollouts, it is fastest on every step and beats the better static layout by $m^\star = \arg\min_m J_m .$5–$m^\star = \arg\min_m J_m .$6 per step, with up to $m^\star = \arg\min_m J_m .$7 over the worse baseline. The switch latency is reported as 215–434 ms end-to-end, and the direct-transfer kernel reshards expert weights in 152 ms, about $m^\star = \arg\min_m J_m .$8 faster than NCCL. Restart-level switching takes 93–133 s, while host-memory reload or CUDA-graph recapture still costs 13–20 s and require draining in-flight work. Holding both layouts resident adds only about 2.4% memory overhead, roughly 0.2 GB above static EP.

4. Deliberative Runtime MoM and emergent composite models

The paper “Mixture-of-Models: Unifying Heterogeneous Agents via N-Way Self-Evaluating Deliberation” defines Runtime MoM as an inference-time ensemble architecture in which multiple heterogeneous checkpoints are dynamically assembled, coordinated, and iteratively refined into a temporary composite system (Pecerskis et al., 23 Jan 2026). Its central protocol, N-Way Self-Evaluating Deliberation (NSED), is described as a white-box, recurrent, self-evaluating protocol and repeatedly framed as a Semantic Recurrent Neural Network or “Macro-Scale RNN.”

NSED is cyclic rather than one-shot. Each round ingests the task plus the current consensus state, has each selected agent generate a proposal, has agents evaluate one another’s proposals, aggregates scores through quadratic voting, updates a shared consensus state, and optionally halts when marginal improvement is small. The execution layer is formalized with a consensus-state recurrence. The input $m^\star = \arg\min_m J_m .$9 and prior state $\tau_q \in (0,1)$0 are embedded as

$\tau_q \in (0,1)$1

each agent $\tau_q \in (0,1)$2 produces

$\tau_q \in (0,1)$3

and the consensus update is

$\tau_q \in (0,1)$4

Here $\tau_q \in (0,1)$5 is the semantic forget gate, $\tau_q \in (0,1)$6 is the vote matrix, and $\tau_q \in (0,1)$7 is proposal content. The halting rule monitors the update magnitude $\tau_q \in (0,1)$8 and can stop when

$\tau_q \in (0,1)$9

Model selection is delegated to a Dynamic Expertise Broker, which the paper treats as a runtime optimization engine and a variation of the Multi-Dimensional Knapsack Problem. Its objective is

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$0

subject to

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$1

with

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$2

The broker chooses a subset of agents and a deliberation budget while binding heterogeneous checkpoints to functional roles such as generator, verifier, critic, or noise source, based on static profile information, task/domain fit, cost and latency limits, and runtime telemetry.

The orchestration fabric is designed as a trustless N-to-N peer-review topology. It uses identity masking, diagonal masking of self-votes,

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$3

and a quadratic-voting activation function intended to saturate excessive voting weight and prevent “shouting.” The paper also defines an influence score,

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$4

to infer which models are strong proposers and which are strong discriminators.

Empirically, the paper reports that ensembles of small (less than 20B) consumer-grade models can match or exceed the performance of state-of-the-art 100B+ parameter models. On AIME 2025, the consumer-grade heterogeneous ensemble reaches 84.0% Pass@1, and the high-performance ensemble reaches 90.0% Pass@1. On LiveCodeBench v5 Hard, the NSED ensemble improves from about 51.5% to 60.2% Pass@1, while naive majority voting is about 33.1%. On the DarkBench safety suite, NSED-R2 achieves the lowest overall RMS score of 0.285, and sycophancy drops to 0.040 at R2, which the paper describes as about a 40% reduction over the best single-agent baseline. The same section also identifies the failure modes: performance is not monotonic in deliberation depth, too much deliberation causes fatigue, weak agents can become noise, and the ensemble must be composed carefully.

5. Adjacent meanings of “MoM”: memories and modules

The acronym “MoM” is overloaded in adjacent literatures. In “MoM: Linear Sequence Modeling with Mixture-of-Memories,” MoM stands for Mixture-of-Memories rather than Mixture-of-Models, but the paper describes it as behaving like a runtime mixture-of-models in the sense that, for each token, only a sparse subset of memory modules is active and the model combines those active memories to produce the output (Du et al., 19 Feb 2025). The architecture replaces the single shared recurrent memory state of linear attention, state-space models, or linear RNNs with multiple independent memory states,

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$5

and routes each token using

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$6

Selected memories are updated recurrently and then mixed: $z_q = b + \mu\,\mathrm{logit}(\tau_q).$7 The main experimental setup uses 4 memory states, 2 activated memories per token, and 1 shared memory. Training remains linear in sequence length, $z_q = b + \mu\,\mathrm{logit}(\tau_q).$8, and the paper reports an average score of 27.59 at 340M parameters and 36.04 at 1.3B on its recall-intensive suite. It also reports that 4 memories / 2 activated is best in the tested setting, while 8 memories / 2 activated is worse than 4.

“Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules” uses MoM for a different runtime-selection idea: a Transformer is treated as a dynamic assembly of modules rather than a fixed shallow-to-deep stack (Gong et al., 2024). The model is formalized as

$z_q = b + \mu\,\mathrm{logit}(\tau_q).$9

with a module set $c$0, router $c$1, assembling function $c$2, $c$3 selected modules per assembly step, and $c$4 assembly steps. Two routers iteratively select attention modules and feed-forward modules, including a SKIP module $c$5, and the forward computation graph is assembled on the fly during the token’s forward pass. The strongest reported configuration, MoM-P, yields GLUE / XSUM averages of 78.22 / 15.19 at 122M, 81.93 / 19.30 at 346M, and 85.90 / 22.36 at 774M. The paper also reports that MoM-large enables an over 38% increase in depth for computation graphs compared to GPT-2-large, as well as an over 60% reduction in depth with a 16% reduction in TFLOPs and a 43% decrease in memory usage while maintaining comparable performance.

These papers do not redefine Runtime Mixture-of-Models in the deployment sense used by Brick, Moebius, or NSED. They do, however, show that the central intuition behind runtime selection can be pushed inward from whole models to memories or modules. This suggests that “MoM” functions as a broader design motif centered on runtime conditional composition, even when the routed entity is not literally a full pretrained model.

6. Misconceptions, trade-offs, and practical significance

A common misconception is to treat Runtime MoM as synonymous with either static ensembling or MoE. The cited work rejects both simplifications. Brick is pure routing, not a cascade, and it chooses one model call per query rather than probing multiple models sequentially (Massa et al., 11 Jun 2026). Moebius does not replace one model with another; it switches between EP and TP as two layouts of the same model, preserving byte-identical expert weights and KV cache and changing only ownership and sharding (Wang et al., 25 Jun 2026). NSED is not a one-pass vote over a fixed committee; it is a recurrent self-evaluating protocol with a broker, a consensus state, a forget gate, and a halting condition (Pecerskis et al., 23 Jan 2026).

Another misconception is that runtime adaptation is free once a router exists. Each formulation exposes a different systems constraint. Brick depends on capability and difficulty estimation, on calibration quality, and on a pricing landscape that can change. Moebius depends on the equivalence of TP and EP over shared model state, on fixed-address CUDA graph validity, and on high-bandwidth GPU interconnects that make ownership-only data movement fast enough between decode steps. NSED depends on session-level optimization, trustless peer review, and the empirical fact that deliberation exhibits a utility optimum rather than monotonic improvement. In related architectures, Mixture-of-Memories can deteriorate when the number of memories becomes too large, and Mixture-of-Modules faces exponential growth in routing search space and reports that load-balancing regularization hurt performance by increasing validation perplexity by 1.8 points (Du et al., 19 Feb 2025, Gong et al., 2024).

The practical significance of Runtime MoM lies in the deployment regimes it targets. Brick addresses cost-sensitive serving over a heterogeneous pool of pretrained models and shows that routing in capability space can improve the quality-cost-latency frontier without retraining the underlying LLMs (Massa et al., 11 Jun 2026). Moebius addresses bursty online traffic and RL rollout engines whose concurrency evolves within a single episode, demonstrating that adaptive serving can occur without draining or dropping in-flight requests (Wang et al., 25 Jun 2026). NSED addresses cases where heterogeneous small models can be assembled into a deliberative system that improves reasoning and mitigates some safety pathologies such as sycophancy (Pecerskis et al., 23 Jan 2026).

Taken together, these works indicate that Runtime Mixture-of-Models is not defined by one specific routed unit or one routing algorithm. Its defining property is the relocation of adaptation from pretraining-time parameterization to inference-time control: the runtime system decides which whole model, model layout, or model team should own the current computation, under explicit constraints of cost, latency, memory, or quality.