Spatial Heterogeneous MoE

• Spatial Heterogeneous Mixture of Experts (SHMoE) is a framework that conditions expert routing on spatial structures to enable region-specific computation.
• It leverages heterogeneous experts with varied architectures, receptive fields, and semantic roles to enhance tasks like image restoration, scene parsing, and mobility modeling.
• SHMoE employs diverse routing mechanisms—from hard top-k gating to softmax and hierarchical strategies—that improve computational efficiency and model interpretability.

Spatial Heterogeneous Mixture of Experts (SHMoE) denotes a class of mixture-of-experts architectures in which expert selection, weighting, or composition is conditioned on spatial structure, and in which experts may differ in architecture, receptive field, semantic role, or representational basis. The canonical neural precursor is the Spatial Mixture-of-Experts (SMoE) layer, introduced as a sparsely gated layer that learns spatial structure in the input domain and routes experts at a fine-grained level (Dryden et al., 2022). Subsequent work extends the same organizing principle to per-pixel scene parsing (Fu et al., 2018), image restoration under spatially heterogeneous distortions (Kim et al., 2020), cross-city mobility modeling (Han et al., 24 May 2025), remote-sensing super-resolution (Chen et al., 12 Feb 2025), hyperspectral classification (Xu et al., 29 Apr 2025), deformable image registration (Zheng et al., 24 Sep 2025), two-view geometry (Le et al., 1 Aug 2025), and geometry-aware multimodal reasoning from sparse RGB video (Feng et al., 7 Jul 2026). Across these formulations, SHMoE replaces homogeneous processing with spatially or structurally specialized computation.

1. Core formulation

In the SMoE layer, the input is a 2D feature map $x\in\mathbb R^{C\times H\times W}$ and the model maintains a pool of $|E|$ experts $\{e_1,\dots,e_{|E|}\}$. At each spatial coordinate $(h,w)$, exactly $E\ll |E|$ experts are selected. The binary gating tensor $G(x)\in\{0,1\}^{|E|\times H\times W}$ has exactly $E$ ones in each column $G(\cdot,h,w)$, and the selected experts process the corresponding location (Dryden et al., 2022). In the concatenative form,

$$y(h,w)=\bigl[f_{i_1}(x;h,w);\dots;f_{i_E}(x;h,w)\bigr], \quad \{i_1,\dots,i_E\}=\mathrm{TopE}\bigl(g(x)\bigr).$$

A weighted form instead writes

$y(h,w)=\sum_{e=1}^{|E|} G(x)_{e,h,w}\;e_e(x;h,w).$

The defining SHMoE extension is that each expert $|E|$0 may have its own architecture $|E|$1, rather than all experts sharing one template (Dryden et al., 2022). The same source gives three design rules for heterogeneous experts: larger experts may be assigned to regions with more structure by incorporating a capacity prior $|E|$2 into the gate, outputs with different dimensionalities $|E|$3 may be unified by expert-specific projections $|E|$4, and balanced utilization may be encouraged by matching average expert utilization to a target proportional to capacity.

A broader MoE formulation appears in later work. TrajMoE defines expert outputs $|E|$5, gating weights $|E|$6 that sum to one, and final output

$|E|$7

with specialization induced by spatial semantics rather than purely by coordinate location (Han et al., 24 May 2025). Statistical work on the Varying-Coefficient Mixture of Experts model gives an analogous spatial formulation in which mixing proportions $|E|$8 and expert coefficients vary smoothly over spatial location $|E|$9 (Zhao et al., 5 Jan 2026). This suggests that SHMoE is not a single layer type but a family of models in which heterogeneity is indexed by space, spatial semantics, or a spatially meaningful latent decomposition.

2. Routing granularity and gating mechanisms

The most distinctive property of SHMoE systems is the spatial scale at which routing occurs. Some models gate at every pixel or voxel; others route at the level of sub-fields, semantic streams, or multimodal tokens. The mechanism may be hard top-$\{e_1,\dots,e_{|E|}\}$0, softmax-weighted, hierarchical, or structurally fixed.

System	Routed unit	Gating mechanism
SMoE	spatial coordinate $\{e_1,\dots,e_{|E|}\}$1	Top-$\{e_1,\dots,e_{|E|}\}$2 over $\{e_1,\dots,e_{|E|}\}$3; in tensor routing, $\{e_1,\dots,e_{|E|}\}$4 (Dryden et al., 2022)
MoE-SPNet	pixel $\{e_1,\dots,e_{|E|}\}$5	per-pixel softmax weight map $\{e_1,\dots,e_{|E|}\}$6 over expert branches (Fu et al., 2018)
DR-HMoE	pixel $\{e_1,\dots,e_{|E|}\}$7	coarse group softmax $\{e_1,\dots,e_{|E|}\}$8 plus fine expert softmax $\{e_1,\dots,e_{|E|}\}$9, then top-$(h,w)$0 (Chen et al., 12 Feb 2025)
GeoMoE	sub-field	softmax over $(h,w)$1 experts, sparsified to top-$(h,w)$2 (Le et al., 1 Aug 2025)
SpaR3D-MoE	token	instruction-pose aware Top-$(h,w)$3 gating probabilities $(h,w)$4 (Feng et al., 7 Jul 2026)

In SMoE, dense scores are first computed as $(h,w)$5, then a top-$(h,w)$6 operator produces sparse routing. The authors report that a purely location-dependent gate $(h,w)$7, called tensor routing, works best in practice, so no input-dependent computation is needed (Dryden et al., 2022). Because the gate is static, experts can be pre-scheduled to regions, improving cache locality.

MoE-SPNet uses a fully spatial gate. Each expert branch produces a prediction map $(h,w)$8, the gating network concatenates $(h,w)$9, applies two convolutions, and normalizes by a per-pixel softmax to obtain $E\ll |E|$0 (Fu et al., 2018). The final aggregated prediction is

$E\ll |E|$1

This is a direct realization of spatially heterogeneous weighting over multiple receptive fields.

TrajMoE introduces a Spatial-Temporal-Adapted Router (STAR). It combines a trajectory gate, a time gate, and a hard adaptation selector. The final semantic weights are

$E\ll |E|$2

and these weights modulate three semantic experts while a shared fuse expert captures city-invariant patterns (Han et al., 24 May 2025). Here routing depends on both historical trajectory context and time-of-day.

Remote-sensing super-resolution uses a dual-routing mechanism. For each pixel feature vector, a coarse router selects one expert group and a fine router selects the top-$E\ll |E|$3 experts within that group. The combined gating weights factorize as $E\ll |E|$4 (Chen et al., 12 Feb 2025). This introduces hierarchy into the routing problem: spatial features are first assigned a coarse processing scale and then a specialist within that scale.

Other SHMoE systems move beyond literal pixel routing. GeoMoE decomposes a motion field into $E\ll |E|$5 sub-fields, then routes each enhanced sub-field to $E\ll |E|$6 out of $E\ll |E|$7 lightweight expert MLPs (Le et al., 1 Aug 2025). SpaR3D-MoE computes routing logits from instruction, visual, geometric, and pose features, then performs Top-$E\ll |E|$8 gating over four geometry-inductive experts (Feng et al., 7 Jul 2026). By contrast, VidPrism explicitly states that there is no learned routing per token; each temporal sampling rate is statically bound to one expert (Lin et al., 27 May 2026). LD-H-MoE similarly uses a fixed task mask rather than a learned gating network: temporal scheduling and spatial beamforming are always handled by separate experts, and the mixture occurs only in the loss through Lyapunov weights (Fan et al., 29 Apr 2026). A common misconception is therefore that SHMoE requires learned sparse dispatch at every location. The literature shows a wider design space.

3. What “heterogeneous” means in practice

Heterogeneity in SHMoE is realized in several non-equivalent ways. The simplest is variation in receptive field. MoE-SPNet uses four expert branches with dilation rates $E\ll |E|$9, so that each branch sees a different context size (Fu et al., 2018). Remote-sensing super-resolution organizes experts into groups with kernel sizes $G(x)\in\{0,1\}^{|E|\times H\times W}$0, homogeneous within each group and heterogeneous across groups (Chen et al., 12 Feb 2025). SHMoAReg uses three decoder experts with $G(x)\in\{0,1\}^{|E|\times H\times W}$1, $G(x)\in\{0,1\}^{|E|\times H\times W}$2, and $G(x)\in\{0,1\}^{|E|\times H\times W}$3 convolutions, enabling per-direction, per-voxel heterogeneity in deformation prediction (Zheng et al., 24 Sep 2025). In all three cases, heterogeneity is an explicit control over spatial support.

A second form of heterogeneity is semantic specialization. TrajMoE uses four experts: a shared Transformer “fuse expert,” a POI expert, a pos expert, and a pop expert. Each expert is fed a different subset of trajectory embeddings, so specialization is induced by architectural priors rather than by an auxiliary KL or entropy penalty (Han et al., 24 May 2025). SpaR3D-MoE defines four heterogeneous experts with distinct fusion rules: holistic additive fusion, visual-to-geometry cross-attention, a pose-conditioned dynamic adapter, and a gravity-aligned structural expert (Feng et al., 7 Jul 2026). These models make the experts different not by width or depth alone but by the semantics of the information path they implement.

A third form is heterogeneity of representational basis. AdaptIR introduces three orthogonal branches that capture local spatial, global spatial, and channel representations. The Local Interaction Module uses a low-rank depth-separable $G(x)\in\{0,1\}^{|E|\times H\times W}$4 convolution, the Frequency Affine Module applies FFT-domain affine transforms to magnitude and phase, and the Channel Shift Module collapses spatial dimensions to focus on channel interactions (Guo et al., 2023). The published version does not learn gating over these branches; the outputs are simply summed. Nonetheless, the paper explicitly presents the module as a heterogeneous MoE because the branches have non-overlapping parameter pools and distinct computation patterns.

A fourth form is heterogeneity induced by stream construction. VidPrism binds one expert to each temporal sampling rate: high-rate experts focus on fine-grained motion, while low-rate experts specialize in spatial semantics (Lin et al., 27 May 2026). MambaMoE uses four spatial experts corresponding to four 2D scanning directions and two shared spectral experts corresponding to forward and backward scans along the band dimension (Xu et al., 29 Apr 2025). GeoMoE routes motion sub-fields to lightweight expert MLPs after a bi-path enhancement stage that separates spatial-context and channel-semantic paths (Le et al., 1 Aug 2025). A plausible implication is that SHMoE is best understood as a division-of-labor principle: experts differ because the data contain multiple spatial regimes, and the model architecture makes those regimes computationally separable.

4. Optimization, auxiliary objectives, and regularization

The base SMoE training objective combines a supervised loss with a routing classification loss,

$G(x)\in\{0,1\}^{|E|\times H\times W}$5

together with expert-error damping (Dryden et al., 2022). The routing classification loss is built from the local error signal entering the SMoE layer, $G(x)\in\{0,1\}^{|E|\times H\times W}$6, thresholded by a quantile $G(x)\in\{0,1\}^{|E|\times H\times W}$7. Selected experts with $G(x)\in\{0,1\}^{|E|\times H\times W}$8 are treated as correct, while incorrectly routed experts are down-weighted by scaling their gradients with a factor $G(x)\in\{0,1\}^{|E|\times H\times W}$9. This directly couples routing supervision to local prediction errors.

SHMoAReg adopts a closely related design in deformable image registration. Its total loss is

$E$0

where $E$1 is MSE image similarity, $E$2 is a deformation regularizer, and $E$3 is a routing-classification loss defined from per-voxel gradient magnitudes (Zheng et al., 24 Sep 2025). The recurrence of routing-classification losses in both SMoE and SHMoAReg indicates that local error-driven self-supervision is a natural fit when expert assignment is spatially explicit.

Other systems avoid routing regularization entirely. TrajMoE states that no additional gating regularizers such as entropy or load-balancing are used; mixture specialization is structural (Han et al., 24 May 2025). AdaptIR likewise introduces no auxiliary MoE or gating regularizers, and MEPSNet uses only a reconstruction loss $E$4 with no explicit distort-classification or gating regularization (Guo et al., 2023, Kim et al., 2020). Remote-sensing super-resolution uses only a standard pixel-wise $E$5 reconstruction loss and reports no explicit regularization on gating weights beyond the softmax constraints (Chen et al., 12 Feb 2025). These choices reflect a second design philosophy in SHMoE: expert roles can be encoded architecturally rather than enforced by balancing penalties.

Several recent models add explicit anti-collapse terms. GeoMoE combines inlier classification, geometry regression, and an expert load-balancing loss

$E$6

with total objective

$E$7

(Le et al., 1 Aug 2025). SpaR3D-MoE adds a load-balancing loss $E$8 to the language generation loss (Feng et al., 7 Jul 2026). VidPrism adds classification, ranking, diversity, and gate-balancing losses, while MambaMoE adds uncertainty-guided corrective losses on ambiguous regions in addition to the final full-image loss (Lin et al., 27 May 2026, Xu et al., 29 Apr 2025). The field therefore spans both minimally regularized structural specialization and explicitly balanced sparse routing.

5. Empirical behavior across domains

The empirical record for SHMoE is broad rather than confined to a single modality. The table summarizes representative reported outcomes.

Domain	Representative result	Paper
Heat diffusion / weather	100% within 1% after 8 epochs; WeatherBench Z500 RMSE 316 $E$9 278	(Dryden et al., 2022)
Scene parsing	DeepLab-ASPP 79.7% $G(\cdot,h,w)$0 82.5% mIoU on PASCAL VOC 2012 test	(Fu et al., 2018)
Spatially heterogeneous distortion restoration	moderate SHDD: 33.47 / 0.9331 PSNR / SSIM	(Kim et al., 2020)
Cross-city human mobility	up to 27% relative improvement after one epoch fine-tuning	(Han et al., 24 May 2025)
Remote-sensing super-resolution	UCMerced $G(\cdot,h,w)$1: 29.29 PSNR, 0.7954 SSIM	(Chen et al., 12 Feb 2025)
Hyperspectral classification	OA 95.20% (Pavia), 90.94% (Houston), 92.67% (Whu-HanChuan)	(Xu et al., 29 Apr 2025)
Two-view geometry	AUC@5° 34.21% vs. 32.57% on YFCC100M	(Le et al., 1 Aug 2025)
3D spatial reasoning	VSI-Bench average 63.5; Route Plan 44.0; Relative Direction 70.1	(Feng et al., 7 Jul 2026)

In the original SMoE study, the controlled heat-diffusion experiment is especially diagnostic. Baselines including CNN, CoordConv, CondConv, LRLCN, and vision-MoE learned only an “average” diffusivity, with approximately 92% of points within 1%, whereas SMoE reached 100% within 1% after 8 epochs. Removing routing classification loss or damping reduced performance to approximately 91%, adding routing classification increased it to approximately 97%, and adding damping recovered 100% with faster convergence (Dryden et al., 2022). On WeatherBench, replacing all $G(\cdot,h,w)$2 convolutions in ResNet-19 with SMoE reduced 3-day Z500 RMSE from 316 to 278 and T850 RMSE from 1.80 K to 1.69 K; adding a land-sea prior on the tensor-routing gate further reduced Z500 to 270 and T850 to 1.66 K (Dryden et al., 2022). On ENS-10, a U-Net with SMoE improved Z500 CRPS from approximately 76.5 to 68.9 and T850 CRPS from approximately 0.685 to 0.612. On ImageNet, replacing all $G(\cdot,h,w)$3 convolutions in ResNet-50 increased Top-1 from 80.83% to 81.33% (Dryden et al., 2022).

Per-pixel spatial heterogeneity also improves semantic parsing. MoE-SPNet raises DeepLab-ASPP with ResNet-101 from 79.7% to 82.5% mIoU on PASCAL VOC 2012 test and from 62.5% to 63.9% on SceneParse150; the lighter AHFA variant raises FCN-8s with VGG16 from 62.2% to 70.6% on PASCAL VOC 2012 and FCN-16s with ResNet-50 from 59.8% to 60.7% on SceneParse150 (Fu et al., 2018). Qualitative examples reportedly show better handling of small objects and region homogeneity.

In image restoration, the SHDD benchmark and MEPSNet show that multi-distortion images benefit from expert specialization. On moderate SHDD, MEPSNet achieves 33.47 / 0.9331 PSNR / SSIM, and on difficult SHDD it achieves 32.71 / 0.9284, outperforming DnCNN, VDSR, OWAN, and RIDNet in the reported comparison (Kim et al., 2020). Feature-map visualization indicates that different experts specialize in blur regions, high-frequency noise regions, and contrast shifts. In parameter-efficient restoration, AdaptIR reports that only approximately 0.6% of parameters are tunable in a typical setting, yet on Rain100H it reaches 31.28 dB, on LOLv1 21.56 dB, and on LR4 + Noise30 26.46 dB while using at most about 1% trainable parameters (Guo et al., 2023).

Mobility and geospatial imaging provide strong evidence that the same design transfers beyond dense vision. TrajMoE reports up to 27% relative improvement over competitive mobility foundation models after only one epoch of fine-tuning and states that using only 5% of target-city data can match or exceed the full-data non-pretrained baseline; in Atlanta, Acc@1 is 0.297 with 5% fine-tuning versus 0.285 for the full-data baseline without pretraining (Han et al., 24 May 2025). In remote-sensing super-resolution, MFG-HMoE achieves 29.29 PSNR and 0.7954 SSIM on UCMerced $G(\cdot,h,w)$4, and 30.62 PSNR and 0.8110 SSIM on AID $G(\cdot,h,w)$5 (Chen et al., 12 Feb 2025). Expert-selection visualization links small-kernel experts to vegetation and small objects, and larger-kernel experts to buildings and large structures.

Recent work extends SHMoE to geometry-intensive and multimodal tasks. GeoMoE reaches AUC@5° = 34.21% versus 32.57% for DeMo on YFCC100M, and ablating probability injection, the spatial path, the channel path, or F-MoE each degrades performance (Le et al., 1 Aug 2025). SpaR3D-MoE reaches the highest average score of 63.5 on VSI-Bench, outperforming the strongest baseline by 7.8 absolute points, with Route Plan 44.0 and Relative Direction 70.1 (Feng et al., 7 Jul 2026). These results indicate that SHMoE is effective not only when outputs are spatial maps, but also when the model must reason over latent geometric structure derived from sparse views or correspondence fields.

6. Interpretation, misconceptions, and open directions

One recurrent advantage of SHMoE is interpretability. SMoE’s tensor-routing gate is location-dependent and can be initialized from external masks such as land versus sea (Dryden et al., 2022). MoE-SPNet yields full spatial weight maps over experts (Fu et al., 2018). MEPSNet visualizes expert-specific feature maps for different distortions (Kim et al., 2020). SHMoAReg defines expert load percentages $G(\cdot,h,w)$6 at each resolution and reports that different directions can select different experts at the same spatial location (Zheng et al., 24 Sep 2025). VidPrism reports class-dependent expert activation patterns in expert-usage heatmaps, and SpaR3D-MoE uses expert masking ablations to show that removing the pose-conditioned dynamic adapter causes the largest drop on VSI-Bench (Lin et al., 27 May 2026, Feng et al., 7 Jul 2026). In this sense, SHMoE often makes specialization directly observable.

A second important point is terminological. SHMoE is not synonymous with sparse top-$G(\cdot,h,w)$7 dispatch, and it is not restricted to experts that differ only in kernel size. The literature includes per-pixel softmax fusion over dilated branches (Fu et al., 2018), image-level self-attention over expert paths (Kim et al., 2020), structurally specialized semantic experts with no gating regularizer (Han et al., 24 May 2025), equally weighted heterogeneous branches without learned routing (Guo et al., 2023), fixed task masks with gradient isolation (Fan et al., 29 Apr 2026), and instruction-pose aware token routing over multimodal experts (Feng et al., 7 Jul 2026). The common denominator is spatially meaningful specialization, not a single canonical router.

Limitations also recur. MEPSNet notes that current gating mixes experts’ outputs uniformly across $G(\cdot,h,w)$8 and identifies spatial gating as a promising next step (Kim et al., 2020). Remote-sensing HMoE reports no explicit regularization on gating weights beyond softmax constraints (Chen et al., 12 Feb 2025). TrajMoE lists the lack of explicit regularization for load balancing as a limitation and suggests entropy or KL penalties as future work (Han et al., 24 May 2025). SMoE notes that a naive PyTorch implementation still pays scatter/gather overhead even when it saves approximately $G(\cdot,h,w)$9 of compute, where $$y(h,w)=\bigl[f_{i_1}(x;h,w);\dots;f_{i_E}(x;h,w)\bigr], \quad \{i_1,\dots,i_E\}=\mathrm{TopE}\bigl(g(x)\bigr).$$0 is the fraction of experts used per location (Dryden et al., 2022). GeoMoE and SpaR3D-MoE address related concerns with explicit load-balancing losses, but this remains an architectural choice rather than a settled standard (Le et al., 1 Aug 2025, Feng et al., 7 Jul 2026).

Finally, there is a useful connection to statistical mixture modeling. The spatial adaptation of the Varying-Coefficient Mixture of Experts model defines softmax gating functions $$y(h,w)=\bigl[f_{i_1}(x;h,w);\dots;f_{i_E}(x;h,w)\bigr], \quad \{i_1,\dots,i_E\}=\mathrm{TopE}\bigl(g(x)\bigr).$$1 over spatial location and spatially varying expert coefficients $$y(h,w)=\bigl[f_{i_1}(x;h,w);\dots;f_{i_E}(x;h,w)\bigr], \quad \{i_1,\dots,i_E\}=\mathrm{TopE}\bigl(g(x)\bigr).$$2, together with identifiability, consistency, asymptotic normality, simultaneous confidence bands, and a generalized likelihood-ratio test for whether a coefficient function is genuinely varying across the index variable (Zhao et al., 5 Jan 2026). This suggests a broader interpretation of SHMoE: neural architectures such as SMoE, GeoMoE, or SpaR3D-MoE can be viewed as high-capacity descendants of a more general idea—mixtures whose latent subpopulations, covariate effects, or routing weights vary across space.

In aggregate, SHMoE has evolved from a sparsely gated spatial layer into a general methodology for modeling spatial inhomogeneity. Its implementations differ substantially, but they converge on the same premise: when data exhibit region-specific, scale-specific, direction-specific, or geometry-specific structure, expert specialization coupled to spatially meaningful routing is often preferable to a single homogeneous computation path.