Astro-MoE: Sparse Expert Astro Architectures

• Astro-MoE is a Mixture-of-Experts framework that employs sparse activation to efficiently process astronomical data in multiband time series and distributed satellite networks.
• It dynamically selects a subset of experts via gating and Top-K routing, optimizing transformer performance while managing constraints like battery and bandwidth.
• Simulation results report up to 45% latency reduction and 30% energy savings, demonstrating scaling potential to models over 50B parameters.

Astro-MoE is a name used in two recent arXiv contexts for Mixture-of-Experts (MoE) architectures adapted to astro-domain workloads. In one usage, it denotes a transformer for multiband astronomical time series that introduces sparsely gated experts at the input embedding stage and within feed-forward sublayers (Cádiz-Leyton et al., 16 Jul 2025). In another, the term is explicitly used as an alternative label for SpaceMoE, a paradigm for distributed MoE inference in low-Earth-orbit satellite networks, where expert modules are placed across satellites and invoked on demand during LLM inference (Chen et al., 16 May 2026). Across both usages, the unifying principle is sparse expert activation: only a subset of experts is executed for a given token or input vector, allowing model capacity to grow without requiring dense execution of all expert modules.

1. Scope and shared MoE formalism

The two usages of Astro-MoE address different problem classes, but both inherit the standard MoE design in which a gating function selects a small number of experts from a larger pool. In the satellite formulation, a standard MoE layer replaces the single feed-forward network in a transformer with $E$ parallel experts and a gating network that computes a relevance score vector $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$, followed by Top-$K$ routing and weighted aggregation of expert outputs (Chen et al., 16 May 2026). In the multiband time-series formulation, the gate is written as raw scores $g(x)=W_g x+b_g$, after which $\mathrm{TopK}(g(x),k)$ masks all but the $k$ largest entries and a softmax yields normalized routing weights $G(x)$, with the layer output given by $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$ (Cádiz-Leyton et al., 16 Jul 2025).

Usage of “Astro-MoE”	Domain	Core specialization unit
Astro-MoE / SpaceMoE	LEO satellite networks	Experts distributed across satellites
Astro-MoE	Multiband astronomical time series	Experts specialized by band and FFN routing

This terminological overlap is important because the two systems are not variants of a single implementation. One is a distributed-systems architecture for on-orbit LLM inference; the other is a domain model for light-curve representation learning and classification. The shared MoE formalism nevertheless makes the comparison technically meaningful: both exploit sparse activation, both use explicit routing mechanisms, and both introduce auxiliary structure to prevent pathological concentration of traffic or expert usage.

2. Astro-MoE as SpaceMoE: distributed MoE inference in orbit

In the satellite literature, Astro-MoE refers to SpaceMoE, described as “a new paradigm for running large Mixture-of-Experts (MoE) LLMs across a network of low-Earth-orbit (LEO) satellites” (Chen et al., 16 May 2026). The central architectural move is to distribute expert modules themselves among satellites, while dynamically selecting, routing, and executing only the needed experts on demand. The paper positions this as an evolution beyond three alternatives: ground-centralized inference, which has high backbone latency and no on-orbit intelligence; on-orbit centralized inference, which requires hosting an entire LLM on a single satellite and is described as impossible for large models; and split layer-wise inference, which chains transformer layers across satellites and incurs high per-token communication while remaining strictly sequential.

SpaceMoE instead decomposes a transformer layer at the expert level. The gate and embedding remain on a source satellite, while different experts may reside on different satellites. Because only a few experts are activated per token, hidden states and expert outputs are transmitted only when needed, and the selected experts compute in parallel. This architecture is explicitly orbit-aware: inter-satellite link bandwidth, time-varying connectivity, limited onboard memory, energy harvesting, battery health, and thermal dissipation all enter the inference design space (Chen et al., 16 May 2026).

The paper organizes the resulting design problem into three fundamental components: expert placement, expert selection, and hidden-state transmission and routing. This decomposition is significant because it makes MoE routing a joint systems-and-model problem rather than a purely neural-architecture problem. In terrestrial MoE deployments, collocation and stable interconnects often dominate the design. In SpaceMoE, mobility, orbital geometry, and resource aging reshape the feasible routing policy itself.

3. Placement, routing, constraints, and reported scalability in SpaceMoE

Expert placement is formulated with binary variables $a_{e,s}\in\{0,1\}$, where $a_{e,s}=1$ indicates that expert $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$0 is hosted or cached on satellite $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$1, subject to per-satellite memory constraints $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$2. Over a time horizon $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$3, the latency between satellites is represented by $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$4, and a placement objective minimizes expected access latency under skewed expert activation probabilities $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$5 plus a replication penalty $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$6. The paper further notes that co-location constraints can encode frequently co-activated expert pairs, yielding mobility- and co-activation-aware placement rather than static allocation (Chen et al., 16 May 2026).

Expert selection extends conventional gating with an energy-aware term. If $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$7 denotes the incremental battery degradation cost of running expert $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$8 on satellite $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$9 at time $K$0, the gating score is modified by an $K$1 penalty, explicitly trading off immediate utility against long-term battery degradation. Load balancing is also retained through a regularizer

$K$2

and the joint training objective is written as

$K$3

The paper states that, in this setting, the balancing term can be weighted more heavily to avoid over-draining any single satellite’s battery (Chen et al., 16 May 2026).

Hidden-state transmission is modeled using time-varying inter-satellite bandwidth $K$4 and packet error rate $K$5. For a hidden state of $K$6 bytes, the one-way delay is the sum over path hops of serialization delay $K$7 and fixed hop delay $K$8, while the end-to-end packet success probability is the product of $K$9 over the path. To improve reliability without excessive ISL load, the paper proposes rateless network coding such as fountain codes. Path selection then trades off low transmission delay, high packet success probability, and minimal extra hops against the thermal and energy budgets of intermediate relays (Chen et al., 16 May 2026).

These mechanisms operate under explicitly satellite-specific constraints. The network graph $g(x)=W_g x+b_g$0 changes on the order of minutes or even seconds, so placement, selection, and routing are computed on sliding windows of predicted topology, typically using orbital ephemeris to forecast $g(x)=W_g x+b_g$1 over $g(x)=W_g x+b_g$2–$g(x)=W_g x+b_g$3 and re-optimizing periodically. Compute energy is approximated by $g(x)=W_g x+b_g$4, communication energy by $g(x)=W_g x+b_g$5, and battery lifetime loss by a first-order model $g(x)=W_g x+b_g$6. Thermal feasibility is constrained by

$g(x)=W_g x+b_g$7

with radiative cooling dominating heat rejection in orbit (Chen et al., 16 May 2026).

The reported simulation results are tied to a Starlink-scale setting with $g(x)=W_g x+b_g$8 LEO satellites and link parameters from existing ISL testbeds. Mobility- and co-activation-aware expert placement reduces average per-token latency from approximately $g(x)=W_g x+b_g$9 under static random placement to approximately $\mathrm{TopK}(g(x),k)$0, described as a 45% save. Similarity-aware, energy-penalized expert selection achieves up to 30% lower battery discharge for the same Rouge-2 accuracy. End-to-end throughput can exceed 150 tokens/sec per constellation with peak ISL utilization under 60%, whereas layer-split inference saturates links at only 30 tokens/sec. The paper also states that SpaceMoE scales to models with more than 50B parameters by exploiting sparsity and parallelism across approximately 10–20 satellites per token, compared with on-orbit centralized inference limited to fewer than 7B parameters (Chen et al., 16 May 2026).

4. Astro-MoE for multiband astronomical time series

In the astronomical time-series literature, Astro-MoE is introduced by Cádiz-Leyton et al. as “a foundational transformer architecture that enables dynamic processing via a Mixture of Experts module” for multiband time series (Cádiz-Leyton et al., 16 Jul 2025). The base model is an encoder-only transformer with 3 stacked self-attention blocks. Each block contains multi-head self-attention, a feed-forward sublayer replaced by MoE, and pre- and post-LayerNorm with residual connections. The internal model dimension $\mathrm{TopK}(g(x),k)$1 and number of attention heads $\mathrm{TopK}(g(x),k)$2 are inherited from the Astromer backbone of Donoso-Oliva et al. (2023), which the summary states typically uses $\mathrm{TopK}(g(x),k)$3–$\mathrm{TopK}(g(x),k)$4 and $\mathrm{TopK}(g(x),k)$5–$\mathrm{TopK}(g(x),k)$6.

Astro-MoE inserts sparsely gated MoE modules in two places. The input embedding stage uses $\mathrm{TopK}(g(x),k)$7, with one expert per photometric band $\mathrm{TopK}(g(x),k)$8, Top-$\mathrm{TopK}(g(x),k)$9 routing with $k$0, and each expert implemented as a linear map $k$1. The feed-forward sublayers inside each attention block use $k$2, again with Top-$k$3 experts per token, and each expert is a two-layer feed-forward network with hidden size $k$4. Because only $k$5 experts are active for each token, the compute remains comparable to a standard transformer while total parameter count grows by approximately $k$6 (Cádiz-Leyton et al., 16 Jul 2025).

The routing formulation is explicit. For an input vector $k$7 with $k$8 for brightness and uncertainty, the gate computes

$k$9

with $G(x)$0 and $G(x)$1. After Top-$G(x)$2 masking and softmax normalization, the mixture output is

$G(x)$3

At the embedding layer, $G(x)$4; in the FFN-MoE blocks, each $G(x)$5 is a two-layer MLP with GeLU activations. To prevent expert collapse, the model adds the Shazeer et al. (2017) auxiliary loss

$G(x)$6

where $G(x)$7 is the mean selection probability and $G(x)$8 the fraction of tokens routed to expert $G(x)$9, giving a total loss $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$0 with $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$1 (Cádiz-Leyton et al., 16 Jul 2025).

The paper’s interpretation of specialization is band-aware. By assigning one input expert per photometric band, the model can learn band-dependent linear projections, while shared attention layers capture cross-band dependencies. A plausible implication is that the architecture combines explicit per-band inductive bias at the embedding interface with shared sequence modeling at the transformer core.

5. Temporal encoding, training protocol, and empirical results

The input representation consists of per-band observations $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$2 for each object and band $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$3. Each band is zero-padded to a fixed length, with the summary stating a maximum of approximately 65 per band, and then concatenated in time order into a single token sequence (Cádiz-Leyton et al., 16 Jul 2025).

Two temporal encoding strategies are compared. The first is standard sine-cosine positional encoding, with

$\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$4

and $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$5 as given in the summary. The second is Learnable Time Modulation (TM) from Cabrera-Vives et al. (2024),

$\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$6

where $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$7 and $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$8 are band-specific Fourier series. The reported empirical conclusion is that TM outperforms fixed PE by 3–4% in reconstruction and downstream tasks (Cádiz-Leyton et al., 16 Jul 2025).

Training is split into self-supervised pretraining and downstream fine-tuning. Pretraining uses masked reconstruction with 30% tokens masked, 30% random, and 40% unchanged; 90% of curves are sampled per batch; dropout is 0.1; and the optimizer is Adam with learning rate $\mathrm{MoE}(x)=\sum_e G_e(x)\,E^{(e)}(x)$9 and batch size 256. Fine-tuning for classification uses cross-entropy on class labels. For ELAsTiCC, light-curve embeddings are concatenated with metadata embeddings through a small tabular transformer in ATAT style; for PS1, only the light-curve embedding and a linear head are used. The principal hyperparameters are fixed across the model family: three transformer layers, dropout 0.1, Adam $a_{e,s}\in\{0,1\}$0, batch 256, 6 input experts, 8 FFN experts, and Top-$a_{e,s}\in\{0,1\}$1 in both MoE components (Cádiz-Leyton et al., 16 Jul 2025).

The reported experimental setup includes both simulated and real data. ELAsTiCC-1 contains 1.8 M light curves in six bands, with 32 raw classes collapsed into 20 groups, an 80/20 train/validation split stratified across 5 folds, a test set of 1,000 per class, and 64 associated metadata features. Pan-STARRS1 contains approximately 20,000 objects across 6 periodic variable classes, in 5 bands $a_{e,s}\in\{0,1\}$2, with at least 4 observations per band and at most 10,000 per class, and a 70/10/20 train/validation/test split over 7 folds. Pretraining is evaluated with $a_{e,s}\in\{0,1\}$3 and RMSE on held-out reconstruction; classification uses macro $a_{e,s}\in\{0,1\}$4; and the baselines are Multiband-Astromer and ATAT (Cádiz-Leyton et al., 16 Jul 2025).

The quantitative results are explicit. On ELAsTiCC-1 reconstruction, Multiband-Astromer (PE) obtains $a_{e,s}\in\{0,1\}$5 and $a_{e,s}\in\{0,1\}$6, Astro-MoE (PE) obtains $a_{e,s}\in\{0,1\}$7 and $a_{e,s}\in\{0,1\}$8, and Astro-MoE (TM) obtains $a_{e,s}\in\{0,1\}$9 and $a_{e,s}=1$0. On ELAsTiCC-1 classification, macro $a_{e,s}=1$1 is $a_{e,s}=1$2 for Multiband-Astromer without pretraining, $a_{e,s}=1$3 for Multiband-Astromer with pretraining, $a_{e,s}=1$4 for ATAT without pretraining, $a_{e,s}=1$5 for Astro-MoE (PE, pretrain), $a_{e,s}=1$6 for Astro-MoE (TM, pretrain), and $a_{e,s}=1$7 for Astro-MoE (TM, no pretrain), identified in the summary as the best overall result. On transfer to PS1, Astro-MoE (TM, no pretrain) reaches $a_{e,s}=1$8 macro $a_{e,s}=1$9, while Astro-MoE (TM, pretrain on ELAsTiCC-1) reaches $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$00 (Cádiz-Leyton et al., 16 Jul 2025).

The accompanying ablation summary attributes the gains to three factors: the contribution of input embedding MoE alone, further gains from adding FFN-MoE, and the consistent advantage of Time Modulation over fixed positional embeddings. The summary also states that the load-balancing loss yields more uniform expert utilization, verified through router-usage histograms, and that sparse activation keeps inference cost low by activating only 2 of 8 experts per token while increasing parameter count by approximately $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$01 (Cádiz-Leyton et al., 16 Jul 2025).

6. Comparative interpretation, nomenclature, and open questions

A recurring source of confusion is that “Astro-MoE” does not denote a single canonical architecture across the cited literature. One paper explicitly uses Astro-MoE as an alternative label for SpaceMoE in orbital LLM inference, while another uses Astro-MoE as the title of a model for multiband astronomical time series (Chen et al., 16 May 2026, Cádiz-Leyton et al., 16 Jul 2025). The overlap is terminological rather than application-level.

At the architectural level, the two systems specialize along different axes. SpaceMoE externalizes experts across satellites and turns routing into a joint optimization over memory placement, dynamic topology, battery degradation, and thermal budgets. The astronomical Astro-MoE internalizes experts within a single transformer, where experts specialize over photometric bands and feed-forward transformations. This suggests that the common MoE abstraction is functioning in two different regimes: as a distributed execution substrate in one case, and as a structured representational prior in the other.

Several misconceptions are directly addressed by the reported results. One is that MoE necessarily implies dense or prohibitive inference; both papers instead rely on sparse activation, with $g(x)=\mathrm{softmax}(W_g x)\in\mathbb{R}^E$02 in the time-series model and only a few experts firing per token in SpaceMoE (Cádiz-Leyton et al., 16 Jul 2025, Chen et al., 16 May 2026). Another is that pretraining uniformly improves downstream performance in the astronomical setting. The ELAsTiCC classification table shows the best overall score for Astro-MoE (TM, no pretrain), whereas the PS1 transfer result shows a substantial gain from ELAsTiCC pretraining, indicating that the effect of pretraining is task- and domain-transfer-dependent (Cádiz-Leyton et al., 16 Jul 2025). A further misconception is that distributed orbital inference is equivalent to layer-split inference; SpaceMoE is explicitly distinguished from layer-wise splitting by expert-level decomposition, on-demand hidden-state exchange, and parallel execution of activated experts (Chen et al., 16 May 2026).

The open research agenda is stated most fully in the SpaceMoE paper. The listed directions are security and trustworthiness, including lightweight encryption, watermarking, and secure multi-party inference protocols; adaptive quantization under time-varying thermal budgets; on-orbit fine-tuning via federated or mobility-aided “store-carry-forward” updates; multi-modal and cross-domain MoE integrating vision, radar, and communications signals; and joint constellation scheduling that co-optimizes expert placement with orbital slot assignments such as orbital plane phasing (Chen et al., 16 May 2026). A plausible implication is that future work will increasingly treat MoE not only as a parameter-scaling mechanism but also as a coordination layer across heterogeneous sensing, communication, and compute substrates.

Taken together, the two Astro-MoE usages show how sparse expert routing can be adapted either to the statistical heterogeneity of multiband light curves or to the systems heterogeneity of LEO constellations. The common mathematical template is MoE; the substantive differences lie in what counts as an expert, where the expert is placed, and which constraints dominate routing.