Mixture of Autoencoder Experts

• Mixture of Autoencoder Experts is a framework that integrates multiple autoencoder models to specialize reconstruction and perform selective inference.
• It employs diverse routing strategies—from soft mixtures to hard winner-take-all—that balance expert contributions and enhance model performance.
• Challenges include preventing expert collapse and data fragmentation while offering benefits like efficient continual learning and task customization.

to=arxiv_search.search ＿影音先锋 json {"^^^^^^^^23query23 OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^^^^^"," OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^,"^^^^^^^^23sort_by23 to=arxiv_search.search 经彩票 json {"^^^^^^^^23query23 OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^^^^^"," to=arxiv_search.search 大发pkjson {"^^^^^^^^23query23 OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23sort_order23^^^^^^^^"," OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^,"^^^^^^^^23sort_by23 OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^^^^^"," to=arxiv_search.search 天天中彩票中奖了ոնը 天天爱彩票app 天天中彩票开奖json {"^^^^^^^^23query23 OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23sort_order23^^^^^^^^\""," OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^,"^^^^^^^^23sort_by23 OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^^^^^"," to=arxiv_search.search qq上json {"^^^^^^^^23query23 Mixture-of-Experts with autoencoder routing for continual RANS turbulence modelling","^^^^^^^^23max_results23 OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^^^^^"," ^^^^^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23sort_order23^^^^^^^^ denotes a family of mixture-of-experts constructions in which autoencoders participate either as the experts themselves, as the routing mechanism, or as the parameterization of a fixed expert mixture decoder. Across the literature, the common objective is to exploit heterogeneity in data or tasks by decomposing a problem into specialized submodels, while retaining sparse or selective inference. In some systems, each expert is a full autoencoder or variational autoencoder trained on a subset or mode of the data; in others, a bank of autoencoders provides unsupervised regime recognition for downstream experts; and in masked-autoencoding variants, experts are embedded inside selected feed-forward sublayers rather than instantiated as standalone encoder–decoder pairs (&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^&&&, &&&^^^^23sort_by23^^^^&&&, &&&^^^^23submittedDate23^^^^&&&, &&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^&&&, &&&^^^^23query23^^^^&&&, &&&^^^^23sort_order23^^^^&&&, &&&^^^^23query23^^^^&&&, &&&^^^^23(Ji et al., 14 Jan 2026)23^^^^&&&, &&&^^^^23max_results23^^^^&&&, &&&^^^^23descending23^^^^&&&).

^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^. Conceptual scope and taxonomy

The most general mixture form appearing across these systems is

PRESERVED_PLACEHOLDER_^^^^23query23^^^^

where PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ denotes an expert and PRESERVED_PLACEHOLDER_^^^^23max_results23^^^^ a routing or gating weight. What changes across formulations is the role played by the autoencoder. In MIXAE, the experts are PRESERVED_PLACEHOLDER_^^^^23sort_by23^^^^ full autoencoders and a mixture assignment network receives the concatenated latent codes to infer soft cluster responsibilities (&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^&&&). In L-MVAE, each expert is a VAE, the mixture coefficients are sampled from a Dirichlet distribution, and lifelong learning is implemented through selective freezing and expansion (&&&^^^^23sort_by23^^^^&&&). In MoE-Sim-VAE and SMoE-VAE, a shared encoder is followed by multiple decoder experts, so specialization occurs in the generative path while a single latent representation is maintained (&&&^^^^23submittedDate23^^^^&&&, &&&^^^^23(Ji et al., 14 Jan 2026)23^^^^&&&).

A different interpretation appears in continual turbulence modelling. The progressive mixture-of-experts framework uses a modular bank of autoencoders as the router, while the experts themselves are heterogeneous turbulence closures rather than autoencoders. The autoencoder bank performs unsupervised regime recognition through reconstruction error and confidence thresholds, and hard top-^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ gating selects a single specialized expert ^^^^^^^^23(Ji et al., 14 Jan 2026)23 In reward-guided reinforcement learning, MoAE-GUIDE uses a ^^^^^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23sort_order23^^^^^^^^ in the stricter sense: multiple autoencoders specialize on different modes or features of expert states, and the weighted reconstruction error defines how expert-like a state is (&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^&&&).

A further branch embeds expertization into masked autoencoding. MoCE replaces selected MAE MLP layers by cluster-conditional experts trained only on semantically relevant images, while a metadata-aware MoE-MAE for Earth observation places sparse experts in transformer FFNs and conditions the encoder on geo-temporal metadata tokens (&&&^^^^23sort_order23^^^^&&&, &&&^^^^23query23^^^^&&&). By contrast, SMoE image-modelling work uses a single deep autoencoder encoder to predict the parameters of a parameter-free steered mixture-of-experts decoder, so the autoencoder is not the expert set but the mechanism that amortizes parameter estimation (&&&^^^^23max_results23^^^^&&&, &&&^^^^23descending23^^^^&&&).

Construction	Role of the autoencoder	Representative paper
Expert autoencoders	Each expert reconstructs one mode or manifold	(&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^&&&, &&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^&&&)
Expert VAEs	Each expert is a VAE in a lifelong mixture	(&&&^^^^23sort_by23^^^^&&&)
Shared encoder, expert decoders	Specialization occurs only in the decoding path	(&&&^^^^23submittedDate23^^^^&&&, &&&^^^^23(Ji et al., 14 Jan 2026)23^^^^&&&)
Autoencoder router	Autoencoders perform regime recognition for non-AE experts	^^^^^^^^23(Ji et al., 14 Jan 2026)23
Expertized masked autoencoding	Experts live inside MAE sublayers or FFNs	(&&&^^^^23sort_order23^^^^&&&, &&&^^^^23query23^^^^&&&)
AE-parametrized fixed MoE decoder	AE predicts parameters for a non-trainable mixture decoder	(&&&^^^^23max_results23^^^^&&&, &&&^^^^23descending23^^^^&&&)

^^^^23max_results23^^^^. Routing mechanisms and expert specialization

Routing spans a continuum from fully soft mixtures to hard winner-takes-all selection. MIXAE trains with soft cluster responsibilities

PRESERVED_PLACEHOLDER_^^^^23submittedDate23^^^^

where PRESERVED_PLACEHOLDER_^^^^23sort_order23^^^^ is the concatenation of all expert latent codes. The per-sample reconstruction error is then weighted by these responsibilities, so the gating network and all autoencoders are jointly optimized (&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^&&&). MoE-Sim-VAE similarly computes gating probabilities from the latent code, but reconstruction and generation use hard routing with PRESERVED_PLACEHOLDER_^^^^23descending23^^^^, yielding a hard-assignment approximation to a decoder-side MoE (&&&^^^^23submittedDate23^^^^&&&).

In SMoE-VAE, the shared encoder maps a sketch to a latent PRESERVED_PLACEHOLDER_^^^^23query23^^^^, a three-layer MLP computes logits, and training uses a differentiable soft mixture of decoder outputs,

PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23^^^^

while inference switches to hard routing with a single decoder PRESERVED_PLACEHOLDER_^^^^23max_results23^^^^ and PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^. Entropy regularization makes the routing distribution low-entropy, narrowing the train–test mismatch and inducing practical sparsity at inference (&&&^^^^23(Ji et al., 14 Jan 2026)23^^^^&&&).

MoAE-GUIDE uses masked reconstruction to handle incomplete demonstrations. For a state vector PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ with mask PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23^^^^, expert PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23sort_by23^^^^ computes

PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23submittedDate23^^^^

the gating network produces PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23sort_order23^^^^, and the mixture error is

PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23descending23^^^^

This reconstruction energy is not only a routing signal but also the basis for intrinsic reward shaping in SAC (&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^&&&).

The PMoE router replaces a conventional Softmax gate by a modular bank of autoencoders, each trained on the feature distribution of a specific flow regime. For a new case, a point PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^ is recognized by component PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23(Ji et al., 14 Jan 2026)23^^^^ if PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23^^^^, with PRESERVED_PLACEHOLDER_^^^^23max_results23query23^^^^ chosen as the ^^^^23max_results23max_results23^^^^.^^^^23max_results23 percentile of training reconstruction errors. The confidence is

PRESERVED_PLACEHOLDER_^^^^23max_results23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^

and the selected component is PRESERVED_PLACEHOLDER_^^^^23max_results23max_results23^^^^. If PRESERVED_PLACEHOLDER_^^^^23max_results23sort_by23^^^^ with PRESERVED_PLACEHOLDER_^^^^23max_results23submittedDate23^^^^, the system uses expert PRESERVED_PLACEHOLDER_^^^^23max_results23sort_order23^^^^; otherwise it expands by adding a new autoencoder component and a new expert ^^^^^^^^23(Ji et al., 14 Jan 2026)23

Masked-autoencoding variants use yet another routing locus. MoCE routes all tokens of an image using the embedding of its assigned cluster,

PRESERVED_PLACEHOLDER_^^^^23max_results23descending23^^^^

rather than routing each token independently by its own embedding. This is designed to enforce semantic routing rather than token-level grouping by low-level statistics (&&&^^^^23sort_order23^^^^&&&). The metadata-aware MoE-MAE for Earth observation uses NoisyTop-PRESERVED_PLACEHOLDER_^^^^23max_results23query23^^^^ routing in transformer FFNs,

PRESERVED_PLACEHOLDER_^^^^23max_results23(Ji et al., 14 Jan 2026)23^^^^

followed by a softmax over the selected indices and a weighted sum of the activated experts (&&&^^^^23query23^^^^&&&).

^^^^23sort_by23^^^^. Learning objectives

Reconstruction remains the central organizing principle, but the objective is almost always augmented by regularizers that enforce confident routing, balanced expert use, similarity preservation, or lifelong retention. In MIXAE, the batch objective combines a weighted reconstruction term, sample-wise entropy

PRESERVED_PLACEHOLDER_^^^^23max_results23max_results23^^^^

and batch-wise entropy over the average assignment vector. The sample-wise term encourages peaky assignments, while the batch-wise term discourages collapse to a single expert (&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^&&&).

L-MVAE replaces deterministic autoencoders by VAEs and jointly trains all experts by maximizing a mixture of individual component ELBOs,

PRESERVED_PLACEHOLDER_^^^^23sort_by23query23^^^^

with PRESERVED_PLACEHOLDER_^^^^23sort_by23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^. The paper further relates this MELBO objective to a lower-bound proxy for the mixture log-likelihood via PRESERVED_PLACEHOLDER_^^^^23sort_by23max_results23^^^^ (&&&^^^^23sort_by23^^^^&&&). This shifts the mixture-of-autoencoders idea from purely reconstructive clustering to generative lifelong representation learning.

MoE-Sim-VAE augments VAE training with a similarity-preserving loss and a DEPICT term. The model encourages the latent representation to follow a Gaussian mixture, and aligns soft cluster assignments with a user-defined similarity matrix through

PRESERVED_PLACEHOLDER_^^^^23sort_by23sort_by23^^^^

where PRESERVED_PLACEHOLDER_^^^^23sort_by23submittedDate23^^^^. The full objective is

PRESERVED_PLACEHOLDER_^^^^23sort_by23sort_order23^^^^

with hard routing used in the decoder path (&&&^^^^23submittedDate23^^^^&&&).

SMoE-VAE uses a PRESERVED_PLACEHOLDER_^^^^23sort_by23descending23^^^^-VAE-style loss together with explicit balance and entropy penalties,

PRESERVED_PLACEHOLDER_^^^^23sort_by23query23^^^^

where PRESERVED_PLACEHOLDER_^^^^23sort_by23(Ji et al., 14 Jan 2026)23^^^^ matches average expert usage to the uniform distribution and PRESERVED_PLACEHOLDER_^^^^23sort_by23max_results23^^^^ penalizes diffuse per-sample routing (&&&^^^^23(Ji et al., 14 Jan 2026)23^^^^&&&). The paper reports PRESERVED_PLACEHOLDER_^^^^23submittedDate23query23^^^^, PRESERVED_PLACEHOLDER_^^^^23submittedDate23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^, and PRESERVED_PLACEHOLDER_^^^^23submittedDate23max_results23^^^^.

Masked-autoencoding systems preserve the MAE reconstruction objective but apply it selectively to routed experts. MoCE defines a hard-gated objective

PRESERVED_PLACEHOLDER_^^^^23submittedDate23sort_by23^^^^

and supplements it with a distillation loss and an imbalance loss, with PRESERVED_PLACEHOLDER_^^^^23submittedDate23submittedDate23^^^^ in practice (&&&^^^^23sort_order23^^^^&&&). The EO model uses

PRESERVED_PLACEHOLDER_^^^^23submittedDate23sort_order23^^^^

with PRESERVED_PLACEHOLDER_^^^^23submittedDate23descending23^^^^ and PRESERVED_PLACEHOLDER_^^^^23submittedDate23query23^^^^, where PRESERVED_PLACEHOLDER_^^^^23submittedDate23(Ji et al., 14 Jan 2026)23^^^^ sums coefficient-of-variation penalties on gate importance and load across layers (&&&^^^^23query23^^^^&&&).

In PMoE, autoencoder training is itself explicit. The encoder–decoder pair PRESERVED_PLACEHOLDER_^^^^23submittedDate23max_results23^^^^ is trained with

PRESERVED_PLACEHOLDER_^^^^23sort_order23query23^^^^

using Adam with learning rate PRESERVED_PLACEHOLDER_^^^^23sort_order23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^, step decay by PRESERVED_PLACEHOLDER_^^^^23sort_order23max_results23^^^^ every PRESERVED_PLACEHOLDER_^^^^23sort_order23sort_by23^^^^ epochs, up to PRESERVED_PLACEHOLDER_^^^^23sort_order23submittedDate23^^^^ epochs with early stopping. The paper additionally presents an Information Bottleneck view,

PRESERVED_PLACEHOLDER_^^^^23sort_order23sort_order23^^^^

to formalize why regime-specific autoencoders isolate different minimal informative feature subsets ^^^^^^^^23(Ji et al., 14 Jan 2026)23

^^^^23submittedDate23^^^^. Continual learning, expansion, and customization

A major line of work uses mixtures of autoencoder experts to avoid catastrophic forgetting or negative transfer by expanding or specializing only where needed. PMoE provides the clearest progressive mechanism. It begins with a baseline expert for a ^^^^23max_results23^^^^D airfoil near-wake case and adds modules only when router confidence falls below the acceptance threshold. The reported stage-wise curriculum is S^^^^23query23^^^^ for ^^^^23max_results23^^^^DANW, S^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ for channel flows, S^^^^23max_results23^^^^ for a periodic hill, and S^^^^23sort_by23^^^^ for square ducts. Existing router confidence is approximately PRESERVED_PLACEHOLDER_^^^^23sort_order23descending23^^^^ for the channel data, PRESERVED_PLACEHOLDER_^^^^23sort_order23query23^^^^ for PH^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^p^^^^23query23^^^^, and PRESERVED_PLACEHOLDER_^^^^23sort_order23(Ji et al., 14 Jan 2026)23^^^^ for square-duct flows, triggering the addition of PRESERVED_PLACEHOLDER_^^^^23sort_order23max_results23^^^^, PRESERVED_PLACEHOLDER_^^^^23descending23query23^^^^, and PRESERVED_PLACEHOLDER_^^^^23descending23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^, respectively ^^^^^^^^23(Ji et al., 14 Jan 2026)23 Because new components are trained without modifying old ones, the framework is explicitly designed to avoid catastrophic forgetting.

L-MVAE implements lifelong learning through a different mechanism. Mixing coefficients are sampled from a Dirichlet distribution whose parameters are updated non-parametrically, experts associated with earlier tasks are effectively frozen, and a new expert is added when a task-similarity criterion exceeds a threshold PRESERVED_PLACEHOLDER_^^^^23descending23max_results23^^^^ (&&&^^^^23sort_by23^^^^&&&). The result is a mixture that can learn new tasks fast when these are similar to those previously learnt, and that expands its architecture when learning a completely new task.

Task customization rather than task accumulation is central in MoCE. The pre-training data are first clustered using dense MAE features and balanced Sinkhorn assignments, then the last two MLP layers with the largest gradient magnitudes are replaced by MoCE layers. Each downstream dataset is assigned to the closest cluster, and fine-tuning uses the corresponding expertized sub-model. This implements “train once, customize everywhere” for downstream tasks with semantically different data distributions (&&&^^^^23sort_order23^^^^&&&).

The metadata-aware EO model also exemplifies structured customization, though through conditioning rather than explicit downstream expert search. It concatenates four metadata tokens—week-of-year, hour-of-day, latitude, and longitude—with the class token and patch tokens, and uses staged expert counts across encoder depth: PRESERVED_PLACEHOLDER_^^^^23descending23sort_by23^^^^ for layers ^^^^23query23^^^^–^^^^23submittedDate23 PRESERVED_PLACEHOLDER_^^^^23descending23submittedDate23^^^^ for layers ^^^^23sort_order23^^^^–^^^^23max_results23 and PRESERVED_PLACEHOLDER_^^^^23descending23sort_order23^^^^ for layers ^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^–^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23submittedDate23^^^^, all with top-PRESERVED_PLACEHOLDER_^^^^23descending23descending23^^^^ routing and PRESERVED_PLACEHOLDER_^^^^23descending23query23^^^^ (&&&^^^^23query23^^^^&&&). This suggests a design in which specialization capacity increases with representational depth while shared attention and embeddings remain fixed.

A common misconception is that continual or customized mixtures require retraining the full system. The surveyed architectures do not support that claim uniformly. PMoE and L-MVAE are explicitly modular and isolate new components from old ones, whereas MoCE customizes transfer by selecting among already pretrained experts rather than expanding during deployment (&&&^^^^23query23^^^^&&&, &&&^^^^23sort_by23^^^^&&&, &&&^^^^23sort_order23^^^^&&&).

^^^^23sort_order23^^^^. Application domains and empirical behavior

The versatility of the paradigm is visible in the diversity of target domains: unsupervised clustering, generative modelling, reinforcement learning, turbulence closure, image compression, denoising, masked-image pretraining, and Earth observation. The empirical record is correspondingly heterogeneous, but several recurrent outcomes appear: improved specialization, robust routing, and sparse inference.

Domain	System	Reported outcome
Continual RANS turbulence modelling	PMoE	Inter-/intra-cluster distance ratios PRESERVED_PLACEHOLDER_^^^^23descending23(Ji et al., 14 Jan 2026)23^^^^; router time PRESERVED_PLACEHOLDER_^^^^23descending23max_results23^^^^ s per case; overhead PRESERVED_PLACEHOLDER_^^^^23query23query23^^^^–PRESERVED_PLACEHOLDER_^^^^23query23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^
RL exploration from incomplete demos	MoAE-GUIDE	Swimmer: PRESERVED_PLACEHOLDER_^^^^23query23max_results23^^^^ to PRESERVED_PLACEHOLDER_^^^^23query23sort_by23^^^^; Ant: PRESERVED_PLACEHOLDER_^^^^23query23submittedDate23^^^^ to PRESERVED_PLACEHOLDER_^^^^23query23sort_order23^^^^
Unsupervised clustering	MIXAE / MoE-Sim-VAE	MIXAE: MNIST PRESERVED_PLACEHOLDER_^^^^23query23descending23^^^^ ACC; MoE-Sim-VAE: MNIST NMI PRESERVED_PLACEHOLDER_^^^^23query23query23^^^^, ACC PRESERVED_PLACEHOLDER_^^^^23query23(Ji et al., 14 Jan 2026)23^^^^
Sparse MoE-VAE interpretation	SMoE-VAE	Best unsupervised test MSE PRESERVED_PLACEHOLDER_^^^^23query23max_results23^^^^ versus supervised PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23query23^^^^
Real-time SMoE image modelling	SMoE-AE	Encode-time reductions of PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ to PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23max_results23^^^^ on ^^^^23sort_order23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23^^^^×^^^^23sort_order23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23^^^^ images
MAE task customization	MoCE	Outperforms vanilla MAE by PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23sort_by23^^^^ on average across ^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ downstream tasks
Compact EO foundation modelling	MoE-MAE	Approximately PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23submittedDate23^^^^M parameters; BEN-LS mAP micro PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23sort_order23^^^^; EuroSAT-LS OA PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23descending23^^^^

In turbulence modelling, PMoE-S^^^^23sort_by23^^^^ retains performance on earlier regimes while improving on later ones. The paper reports that in the ^^^^23max_results23^^^^DANW wake PMoE-S^^^^23sort_by23^^^^ matches baseline SA and experiments; in channels it improves wall-normal profiles; in PH^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^p^^^^23query23^^^^ it predicts reattachment near PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23query23^^^^ versus a baseline delayed reattachment at PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23(Ji et al., 14 Jan 2026)23^^^^; and in square ducts it captures secondary motions that SA cannot ^^^^^^^^23(Ji et al., 14 Jan 2026)23 These results are tied to sparse activation with PRESERVED_PLACEHOLDER_^^^^23(Ji et al., 14 Jan 2026)23max_results23^^^^, so model expansion does not incur additional computational cost during inference.

In RL, MoAE-GUIDE is designed for unlabeled and incomplete demonstrations, including state-only trajectories recorded every five steps and settings in which the PRESERVED_PLACEHOLDER_^^^^23max_results23query23^^^^-coordinate is hidden. The paper reports improvements over ER-only in Swimmer, Walker^^^^23max_results23^^^^d, and Ant, and notes that intrinsic-only reaches expert-level in Hopper. It also reports strong sparse-reward results, including Ant sparse with IR+pretraining PRESERVED_PLACEHOLDER_^^^^23max_results23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^, MoE-GUIDE PRESERVED_PLACEHOLDER_^^^^23max_results23max_results23^^^^, and ER-only PRESERVED_PLACEHOLDER_^^^^23max_results23sort_by23^^^^ (&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^&&&).

In unsupervised clustering, MIXAE achieves MNIST PRESERVED_PLACEHOLDER_^^^^23max_results23submittedDate23^^^^, Reuters PRESERVED_PLACEHOLDER_^^^^23max_results23sort_order23^^^^, and HHAR PRESERVED_PLACEHOLDER_^^^^23max_results23descending23^^^^ clustering accuracy without pretraining, outperforming DEC on all three datasets and VaDE on HHAR (&&&^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^&&&). MoE-Sim-VAE reports MNIST NMI PRESERVED_PLACEHOLDER_^^^^23max_results23query23^^^^ and ACC PRESERVED_PLACEHOLDER_^^^^23max_results23(Ji et al., 14 Jan 2026)23^^^^, mouse-organ scRNA-seq F-measure PRESERVED_PLACEHOLDER_^^^^23max_results23max_results23^^^^ and NMI PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23query23^^^^, and strong CyTOF results across Weber, Robinson, and Bodenmiller datasets (&&&^^^^23submittedDate23^^^^&&&). SMoE-VAE adds an interpretability result: unsupervised routing produces cleaner latent clusters than class labels, with expert-ID linear probe accuracy PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^, class-label probe accuracy PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23max_results23^^^^, and correlation between expert IDs and class labels PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23sort_by23^^^^ (&&&^^^^23(Ji et al., 14 Jan 2026)23^^^^&&&).

For image modelling and compression, the steered SMoE autoencoder reports runtime reductions from hundreds of seconds to sub-second encoding on ^^^^23sort_order23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23^^^^×^^^^23sort_order23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23^^^^ grayscale images. One configuration reports SMoE-GD encode PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23submittedDate23^^^^ s versus SMoE-AE encode PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23sort_order23^^^^ s for ^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23descending23^^^^×^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23descending23^^^^ radial-kernel blocks, with speedup PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23descending23^^^^ and decoding PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23query23^^^^ s (&&&^^^^23max_results23^^^^&&&). The earlier edge-aware compression work reports a speedup of roughly PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23(Ji et al., 14 Jan 2026)23^^^^ to PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23max_results23^^^^, with quantitative examples such as Peppers at PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^ bpp: JPEG PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ dB / SSIM PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23^^^^, SMoE-GD PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23sort_by23^^^^ dB / PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23submittedDate23^^^^, and SMoE-AE PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23sort_order23^^^^ dB / PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23descending23^^^^ (&&&^^^^23descending23^^^^&&&).

In masked autoencoding, MoCE improves average top-^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ accuracy by PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23^^^^ over a matched-training-time MAE baseline across eleven tasks, with representative gains from PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23(Ji et al., 14 Jan 2026)23^^^^ to PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23^^^^ on Aircraft, PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23query23^^^^ to PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23^^^^ on Cars, and PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23max_results23^^^^ to PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23sort_by23^^^^ on CIFAR-^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23query23query23^^^^; it also reports ADE^^^^23max_results23query23^^^^K mIoU PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23submittedDate23^^^^ versus MAE PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23sort_order23^^^^ and COCO Cascade Mask R-CNN PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23descending23^^^^ APPRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23query23^^^^ and PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23(Ji et al., 14 Jan 2026)23^^^^ APPRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Malomgré et al., 21 Jul 2025) OR id:(Fleig et al., 2023) OR id:(Ye et al., 2021) OR id:(Kopf et al., 2019) OR id:(Liu et al., 2024) OR id:(Fleig et al., 2022) OR id:(Albughdadi, 13 Sep 2025) OR id:(Nikolic et al., 12 Sep 2025)23max_results23max_results23^^^^ (&&&^^^^23sort_order23^^^^&&&). The EO model extends masked autoencoding to a compact geo-temporal setting: approximately PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 2026) OR id:(Ji et al., 14 Jan 20260) OR id:(Ji et al., 14 Jan 20261) OR id:(Ji et al., 14 Jan 20262) OR id:(Ji et al., 14 Jan 20263) OR id:(Ji et al., 14 Jan 20264) OR id:(Ji et al., 14 Jan 20265) OR id:(Ji et al., 14 Jan 20266) OR id:(Ji et al., 14 Jan 20267)23sort_by23query23^^^^M parameters overall, BEN-LS all-token mAP micro PRESERVED_PLACEHOLDER_^^^^23id:(Ji et al., 14 Jan 20268) OR id:(Ji et al., 14 Jan 20269) OR id:(Malomgré et al., 21 Jul 20250) OR id:(Malomgré et al., 21 Jul 20251) OR id:(Malomgré et al., 21 Jul 20252) OR id:(Malomgré et al., 21 Jul 20253) OR id:(Malomgré et al., 21 Jul 20254) OR id:(Malomgré et al., 21 Jul 20255) OR id:(Malomgré et al., 21 Jul 20256)23sort_by23id:(Malomgré et al., 21 Jul 20257) OR id:(Malomgré et al., 21 Jul 20258) OR id:(Malomgré et al., 21 Jul 20259) OR id:(Fleig et al., 20230) OR id:(Fleig et al., 20231) OR id:(Fleig et al., 20232) OR id:(Fleig et al., 20233) OR id:(Fleig et al., 20234) OR id:(Fleig et al., 20235)23^^^^ and F^^^^23id:(Fleig et al., 20236) OR id:(Fleig et al., 20237) OR id:(Fleig et al., 20238) OR id:(Fleig et al., 20239) OR id:(Ye et al., 20210) OR id:(Ye et al., 20211) OR id:(Ye et al., 20212) OR id:(Ye et al., 20213) OR id:(Ye et al., 20214)23^^^^ micro PRESERVED_PLACEHOLDER_^^^^23id:(Ye et al., 20215) OR id:(Ye et al., 20216) OR id:(Ye et al., 20217) OR id:(Ye et al., 20218) OR id:(Ye et al., 20219) OR id:(Kopf et al., 20190) OR id:(Kopf et al., 20191) OR id:(Kopf et al., 20192) OR id:(Kopf et al., 20193)23sort_by23max_results23^^^^, and EuroSAT-LS all-token OA PRESERVED_PLACEHOLDER_^^^^23id:(Kopf et al., 20194) OR id:(Kopf et al., 20195) OR id:(Kopf et al., 20196) OR id:(Kopf et al., 20197) OR id:(Kopf et al., 20198) OR id:(Kopf et al., 20199) OR id:(Liu et al., 20240) OR id:(Liu et al., 20241) OR id:(Liu et al., 20242)23sort_by23sort_by23^^^^ despite the absence of explicit metadata at transfer time (&&&^^^^23query23^^^^&&&).

^^^^23descending23^^^^. Limitations, misconceptions, and open directions

Several failure modes recur across the literature. Expert collapse remains a central issue. MIXAE addresses it with batch-wise entropy, but the same regularizer can bias learning on unbalanced data, as observed on Reuters where actual batch entropy approaches PRESERVED_PLACEHOLDER_^^^^23id:(Liu et al., 20243) OR id:(Liu et al., 20244) OR id:(Liu et al., 20245) OR id:(Liu et al., 20246) OR id:(Liu et al., 20247) OR id:(Liu et al., 20248) OR id:(Liu et al., 20249) OR id:(Fleig et al., 20220) OR id:(Fleig et al., 20221)23sort_by23submittedDate23^^^^ despite nonuniform class proportions (&&&^^^^23id:(Fleig et al., 20222) OR id:(Fleig et al., 20223) OR id:(Fleig et al., 20224) OR id:(Fleig et al., 20225) OR id:(Fleig et al., 20226) OR id:(Fleig et al., 20227) OR id:(Fleig et al., 20228) OR id:(Fleig et al., 20229) OR id:(Albughdadi, 13 Sep 20250)23query23^^^^&&&). SMoE-VAE reports that roughly half of experts can remain inactive, and performance degrades when the number of experts is too large because of data starvation and over-fragmentation (&&&^^^^23(Albughdadi, 13 Sep 20251)23^^^^&&&). MoAE-GUIDE notes that too many experts or poorly tuned gating can inflate false positives by labeling non-expert regions as expert-like, and that mapping thresholds and decay schedules are sensitive hyperparameters (&&&^^^^23id:(Albughdadi, 13 Sep 20252) OR id:(Albughdadi, 13 Sep 20253) OR id:(Albughdadi, 13 Sep 20254) OR id:(Albughdadi, 13 Sep 20255) OR id:(Albughdadi, 13 Sep 20256) OR id:(Albughdadi, 13 Sep 20257) OR id:(Albughdadi, 13 Sep 20258) OR id:(Albughdadi, 13 Sep 20259) OR id:(Nikolic et al., 12 Sep 20250)23^^^^&&&).

Another common misunderstanding is that “^^^^^^^^23id:(Nikolic et al., 12 Sep 20251) OR id:(Nikolic et al., 12 Sep 20252) OR id:(Nikolic et al., 12 Sep 20253) OR id:(Nikolic et al., 12 Sep 20254) OR id:(Nikolic et al., 12 Sep 20255) OR id:(Nikolic et al., 12 Sep 20256) OR id:(Nikolic et al., 12 Sep 20257) OR id:(Nikolic et al., 12 Sep 20258) OR id:(Nikolic et al., 12 Sep 20259)23sort_order23^^^^^^^^” always means that each expert is a complete autoencoder. That description is accurate for MIXAE and the behavior-modelling component of MoAE-GUIDE, but not for PMoE, where autoencoders are routers and the experts are symbolic-regression, neural-network, or constitutive-modification turbulence models, nor for MoCE and MoE-MAE, where experts are inserted into selected MAE layers or transformer FFNs rather than instantiated as full encoder–decoder modules (&&&^^^^23query23^^^^&&&, &&&^^^^23sort_order23^^^^&&&, &&&^^^^23query23^^^^&&&).

A further distinction concerns soft versus sparse inference. Several training procedures use soft mixtures for differentiability and then switch to hard routing at deployment. This is explicit in MoE-Sim-VAE, SMoE-VAE, MIXAE evaluation, and PMoE’s top-^^^^23id:(Ji et al., 14 Jan 20260) OR id:(Ji et al., 14 Jan 20261) OR id:(Ji et al., 14 Jan 20262) OR id:(Ji et al., 14 Jan 20263) OR id:(Ji et al., 14 Jan 20264) OR id:(Ji et al., 14 Jan 20265) OR id:(Ji et al., 14 Jan 20266) OR id:(Ji et al., 14 Jan 20267) OR id:(Ji et al., 14 Jan 20268)23^^^^ gating (&&&^^^^23submittedDate23^^^^&&&, &&&^^^^23(Ji et al., 14 Jan 20269)23^^^^&&&, &&&^^^^23id:(Malomgré et al., 21 Jul 20250) OR id:(Malomgré et al., 21 Jul 20251) OR id:(Malomgré et al., 21 Jul 20252) OR id:(Malomgré et al., 21 Jul 20253) OR id:(Malomgré et al., 21 Jul 20254) OR id:(Malomgré et al., 21 Jul 20255) OR id:(Malomgré et al., 21 Jul 20256) OR id:(Malomgré et al., 21 Jul 20257) OR id:(Malomgré et al., 21 Jul 20258)23query23^^^^&&&, &&&^^^^23query23^^^^&&&). Sparse activation is therefore not a universal training property, but it is a recurrent inference property.

The limitations specific to masked and image-centric formulations differ from those in lifelong or control settings. MoCE is sensitive to clustering quality and can fragment training data when the number of clusters is too high (&&&^^^^23sort_order23^^^^&&&). The EO model is exposed to domain shifts across sensors or regions and to missing or noisy metadata (&&&^^^^23query23^^^^&&&). The SMoE image-modelling line reports oversmoothing on highly textured images such as Baboon, fixed block sizes, no rate term in training, and absent quantization-aware optimization (&&&^^^^23max_results23^^^^&&&, &&&^^^^23descending23^^^^&&&).

Open directions are explicit in the surveyed papers. PMoE notes that its modularity would allow replacement of symbolic-regression and NN experts by autoencoder-based surrogates that reconstruct closure fields or outputs under physics-informed losses, although this would require careful physics constraints such as invariance and realizability ^^^^^^^^23(Malomgré et al., 21 Jul 20259)23 MoAE-GUIDE points to VAE-based experts, density models, multimodal demonstrations, hierarchical mixtures, episodic novelty bonuses, and automated threshold selection (&&&^^^^23id:(Fleig et al., 20230) OR id:(Fleig et al., 20231) OR id:(Fleig et al., 20232) OR id:(Fleig et al., 20233) OR id:(Fleig et al., 20234) OR id:(Fleig et al., 20235) OR id:(Fleig et al., 20236) OR id:(Fleig et al., 20237) OR id:(Fleig et al., 20238)23^^^^&&&). The SMoE compression line points toward true steerable kernels, variable PRESERVED_PLACEHOLDER_^^^^23id:(Fleig et al., 20239) OR id:(Ye et al., 20210) OR id:(Ye et al., 20211) OR id:(Ye et al., 20212) OR id:(Ye et al., 20213) OR id:(Ye et al., 20214) OR id:(Ye et al., 20215) OR id:(Ye et al., 20216) OR id:(Ye et al., 20217)23sort_by23sort_order23^^^^, learned hyperpriors, end-to-end rate–distortion training, and multi-expert AE gating (&&&^^^^23max_results23^^^^&&&, &&&^^^^23descending23^^^^&&&). MoCE generalizes the cluster-conditional paradigm to other masked modelling tasks and modalities, while L-MVAE and SMoE-VAE suggest continual expert growth and deeper specialization hierarchies (&&&^^^^23sort_order23^^^^&&&, &&&^^^^23sort_by23^^^^&&&, &&&^^^^23(Ye et al., 20218)23^^^^&&&).

Taken together, these results indicate that the term does not identify a single architecture but a design principle: use reconstruction-driven representation learning to define expert boundaries, route inputs selectively, and preserve either efficiency, specialization, or adaptability. The exact locus of the autoencoder—expert, router, or amortized parameterizer—determines the mathematical form of the mixture and the failure modes it must address.