Gaussian Splatting-guided Mixture of Experts

• GS-MoE is a design family that leverages Gaussian splatting to guide expert specialization via differentiable rendering and routing signals.
• It integrates modular pipelines—from a two-expert pose-free 3DGS to a formal dynamic MoE and temporal anomaly detection—to address heterogeneous tasks.
• Key mechanisms include pose-aware conditioning, volume-aware pixel routing, and temporal splatting that transform latent geometric and temporal cues into effective supervisory signals.

Gaussian Splatting-guided Mixture of Experts (GS-MoE) denotes a family of architectures in which Gaussian-splatting-derived signals structure expert specialization, supervision, or routing. In current literature, the designation spans at least three closely related but technically distinct usages: a pose-free feed-forward 3D Gaussian Splatting pipeline in which a geometry expert and an appearance expert are coupled through differentiable 3DGS rendering; a formal Mixture-of-Experts framework for dynamic Gaussian splatting in which a router projects volumetric Gaussian-level weights into pixel space; and a weakly supervised video anomaly detection framework in which temporal Gaussian splatting constructs pseudo-labels for expert learning and gate fusion (Jeong et al., 22 Mar 2026, Jin et al., 22 Oct 2025, D'Amicantonio et al., 8 Aug 2025). A plausible implication is that GS-MoE is best understood as a design family rather than a single canonical model.

1. Terminology and scope

The literature uses the term in two explicit naming conventions. The dynamic-scene reconstruction paper introduces “MoE-GS: Mixture of Experts for Dynamic Gaussian Splatting” and states that “Gaussian Splatting-guided Mixture of Experts (GS-MoE)” is synonymous in intent when emphasizing guidance by Gaussian splatting. The pose-free 3DGS paper does not implement a formal MoE, but describes its two-expert design as a modular mixture in which 3DGS is both the output representation and the supervisory signal. The video anomaly detection paper uses “GS-MoE” directly for a framework in which temporal Gaussian splatting guides weak supervision (Jin et al., 22 Oct 2025, Jeong et al., 22 Mar 2026, D'Amicantonio et al., 8 Aug 2025).

Setting	Role of Gaussian splatting	Expert mechanism
Pose-free feed-forward 3DGS	Output representation and rendering-based supervision	Geometry expert + appearance expert
Dynamic Gaussian splatting	Volumetric-to-pixel routing via differentiable weight splatting	Multiple frozen dynamic experts + Volume-aware Pixel Router
Weakly supervised VAD	Temporal pseudo-label construction around abnormal peaks	Class-experts or cluster-experts + gate

Across these settings, Gaussian splatting does not merely serve as a rendering backend. In the pose-free and dynamic 3DGS formulations it transports geometric or volumetric information into optimization or routing signals, while in weak supervision it becomes a temporal label-construction mechanism. This suggests a common abstraction: splatting converts structured latent evidence into a field on which experts can be trained, fused, or regularized.

2. Two-expert decoupling in pose-free feed-forward 3D Gaussian Splatting

In “Two Experts Are Better Than One Generalist: Decoupling Geometry and Appearance for Feed-Forward 3D Gaussian Splatting,” GS-guided modularity appears as a two-expert pipeline called 2Xplat. The geometry expert predicts camera parameters per view, $p_i = [K_i, R_i, t_i]$, from uncalibrated multi-view RGB images. The appearance expert then synthesizes a 3D Gaussian Splatting representation conditioned explicitly on the predicted poses and the context images. The geometry backbone is Depth Anything 3 as the primary geometry foundation model, with alternatives including $\pi^3$ and DA3-L/G variants. The appearance backbone is MVP (Multi-view Pyramid Transformer), with pose conditioning via PRoPE, Alternating Attention, dual hierarchical token pyramids, and register tokens (Jeong et al., 22 Mar 2026).

The end-to-end data flow is explicit. Uncalibrated multi-view images $\{I_i\}$ are processed by the geometry expert $F_{\text{pose}}$ to predict $\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$ for all views. The appearance expert $F_{\text{3dgs}}$ takes context images and predicted poses and produces pixel-aligned Gaussians $\{G_i\}$. A differentiable 3DGS renderer projects and composites these Gaussians to synthesize target views $\hat I_i$. Supervision is split between rendering loss and camera loss. The rendering loss is

$$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$

while the geometry expert is trained with relative rotation, translation, and intrinsics losses through

$$L_{\text{cam}} = \frac{1}{N(N-1)} \sum_{i \ne j} \left( \lambda_R L_R(i,j) + \lambda_t L_t(i,j) \right) + \frac{\lambda_K}{N} \sum_{j=1}^N L_K(j).$$

The paper gives $\pi^3$0, $\pi^3$1, $\pi^3$2, and $\pi^3$3.

This system is explicitly described as not using a formal gating module. There is no explicit soft or hard gating; routing is deterministic, because predicted camera poses are passed to the appearance expert and injected via pose-aware conditioning. The GS-guided interaction arises because differentiable 3DGS rendering generates gradients that flow back through the appearance expert to the geometry expert. Appearance fidelity therefore guides geometric predictions, while geometric accuracy enables pose-aware appearance conditioning. The paper characterizes this as a “GS-guided expert interaction.”

The geometric and rendering machinery is standard but central. A 3D point $\pi^3$4 is projected by

$\pi^3$5

and each Gaussian primitive has mean $\pi^3$6, covariance $\pi^3$7, opacity $\pi^3$8, and color parameters $\pi^3$9. In a view $\{I_i\}$0, the projected mean is $\{I_i\}$1 and the 2D footprint covariance is approximated by $\{I_i\}$2. Front-to-back compositing uses

$\{I_i\}$3

The training protocol emphasizes efficiency. End-to-end fine-tuning converges in $\{I_i\}$4–$\{I_i\}$5 iterations on 8 H200 GPUs. AdamW is used with $\{I_i\}$6, $\{I_i\}$7, $\{I_i\}$8, weight decay $\{I_i\}$9, and gradient norm clipping. Evaluation-time pose alignment optionally refines poses for 100 Adam iterations at $F_{\text{pose}}$0. On DL3DV at $F_{\text{pose}}$1, pose-free inference without ground-truth intrinsics and without EPA yields PSNR/SSIM/LPIPS of 26.007/0.839/0.126 for 6 views, 26.015/0.826/0.129 for 12 views, and 25.894/0.832/0.125 for 24 views; with EPA these become 26.670/0.855/0.122, 26.963/0.854/0.121, and 27.083/0.865/0.116. On RE10K at $F_{\text{pose}}$2 with 6 context views, pose-free results are 26.161 PSNR, 0.859 SSIM, and 0.132 LPIPS; with EPA, 27.239, 0.881, and 0.126. Pose estimation AUC on RE10K is 0.718 at $F_{\text{pose}}$3, 0.843 at $F_{\text{pose}}$4, and 0.912 at $F_{\text{pose}}$5. The paper reports that the two-expert design substantially outperforms prior pose-free feed-forward 3DGS approaches in fewer than 5K training iterations and challenges the prevailing unified paradigm.

3. Formal MoE-GS in dynamic Gaussian splatting

“MoE-GS: Mixture of Experts for Dynamic Gaussian Splatting” introduces a formal MoE architecture for dynamic scene reconstruction. Its premise is that no single dynamic 3DGS formulation consistently dominates across scenes, regions, or frames. The framework therefore integrates multiple specialized dynamic Gaussian experts, each trained and frozen, and blends their rendered outputs per pixel through a Volume-aware Pixel Router. The paper integrates, for example, 4DGaussians, E-D3DGS, Ex4DGS, and STG as complete dynamic 3DGS renderers that take camera pose, intrinsics, and time index $F_{\text{pose}}$6 as inputs and produce rendered RGB images $F_{\text{pose}}$7 (Jin et al., 22 Oct 2025).

The router begins from per-expert, per-Gaussian learnable weights in 3D: $F_{\text{pose}}$8, $F_{\text{pose}}$9, and $\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$0. For routing, each expert duplicates its Gaussians and replaces color with

$\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$1

These quantities are splatted to pixel space with the same alpha-compositing visibility used by Gaussian splatting, yielding $\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$2, $\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$3, and $\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$4. A lightweight shared MLP $\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$5 then computes router logits

$\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$6

followed by softmax gating

$\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$7

and final image blending

$\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$8

The router is therefore “volume-aware” because depth ordering, alpha compositing, and occlusion structure are inherited from the underlying 3DGS rasterization.

The underlying Gaussian formulation is standard. A Gaussian $\hat p_i = [\hat K_i, \hat R_i, \hat t_i]$9 has mean $F_{\text{3dgs}}$0, covariance $F_{\text{3dgs}}$1, color $F_{\text{3dgs}}$2, and opacity scalar $F_{\text{3dgs}}$3. Projection to screen space gives $F_{\text{3dgs}}$4 and $F_{\text{3dgs}}$5, inducing a screen-space Gaussian density. The per-pixel opacity contribution is

$F_{\text{3dgs}}$6

and front-to-back compositing uses

$F_{\text{3dgs}}$7

Training is two-stage. In Stage 1, each expert is trained independently using its original strategy and hyperparameters. In Stage 2, experts are frozen and the router parameters $F_{\text{3dgs}}$8 and $F_{\text{3dgs}}$9 are optimized with photometric losses,

$\{G_i\}$0

The paper explicitly notes that it does not add explicit router regularization such as entropy, KL, or temporal smoothness terms.

Because MoE increases capacity and rendering overhead, the method adds two efficiency mechanisms. Single-pass multi-expert rendering merges all experts’ Gaussians into one batch and accumulates expert-specific colors during a single alpha-compositing pass:

$\{G_i\}$1

Gate-aware Gaussian pruning measures importance by

$\{G_i\}$2

and prunes low-sensitivity Gaussians. On N3V with $\{G_i\}$3, the MoE-GS baseline reports 32.82 dB PSNR, 44 FPS, and 878.7 MB; about 55% pruning gives 32.80 dB, 83 FPS, and 351.2 MB; about 75% pruning gives 32.45 dB, 101 FPS, and 281.3 MB. For $\{G_i\}$4, with single-pass rendering and pruning enabled, the system achieves 33.23 dB PSNR, 68 FPS, and 270 MB. The distillation stage further transfers MoE performance to individual experts through gate-weighted supervision.

Experimentally, the method reports average 33.27 dB on N3V with $\{G_i\}$5, average 34.55 dB on Technicolor with $\{G_i\}$6, and average 24.81 dB on HyperNeRF with $\{G_i\}$7. Router ablations on N3V show PSNR/SSIM/LPIPS of 31.12/0.952/0.022 for a Pixel Router, 32.05/0.951 for a Volume Router, and 33.23/0.954/0.021 for the Volume-aware Pixel Router. The paper identifies compute and memory overhead, gating sensitivity, and pruning trade-offs as principal limitations.

4. Temporal GS-MoE in weakly supervised video anomaly detection

In weakly supervised video anomaly detection, GS-MoE denotes a different but structurally related use of splatting. “Mixture of Experts Guided by Gaussian Splatters Matters: A new Approach to Weakly-Supervised Video Anomaly Detection” trains only with video-level labels but requires snippet- or frame-level anomaly scores at inference. The backbone encoder is I3D (ResNet-50 variant), pretrained on Kinetics-400. Videos are divided into sliding windows of 16 frames, producing 1024-dimensional snippet features. For batching, variable-length feature sequences are linearly resampled to fixed temporal length $\{G_i\}$8. A task-aware encoder, UR-DMU, is first trained with standard MIL loss and then fine-tuned with Temporal Gaussian Splatting (TGS) loss to produce refined anomaly-aware features or logits of dimension 1024 (D'Amicantonio et al., 8 Aug 2025).

The expert side is explicit. Class-experts are set equal to the number of anomaly classes, with UCF-Crime using 13 experts; a cluster-expert variant based on K-Means is also explored. Each expert receives the 1024D task-aware logits and consists of a transformer block followed by an MLP head. The transformer block applies LayerNorm, 2-head self-attention, residual addition, another LayerNorm, linear projection to 512, ReLU, linear projection back to 1024, and residual addition. The MLP reduces $\{G_i\}$9, uses GELU between the last two layers, and ends with sigmoid to produce per-snippet expert prediction $\hat I_i$0. Each expert has approximately 0.5M parameters.

Aggregation is performed by a gate rather than softmax-normalized expert weights. The gate takes concatenated expert scores $\hat I_i$1, projects them to 1024D, and performs bi-directional cross-attention with the task-aware logits $\hat I_i$2. One direction uses projected expert scores as values and task-aware logits as keys and queries; the other reverses the roles. The outputs are concatenated to 2048D, processed by a transformer block with 4 attention heads, then passed through an MLP and sigmoid to produce final anomaly scores $\hat I_i$3. The gate has approximately 1M parameters.

Temporal Gaussian Splatting is the supervisory core. Peaks are selected as local maxima in abnormal scores using thresholding on prominence relative to the previous two and next two snippets. For each peak $\hat I_i$4 at position $\hat I_i$5, the width is $\hat I_i$6, where $\hat I_i$7 and $\hat I_i$8 count monotonic score increase and decrease around the peak. A binary gate $\hat I_i$9 is then defined around the peak using local score band and width, with standard deviation $$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$0 computed from scores around $$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$1 within $$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$2. The temporal Gaussian is

$$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$3

and snippet-level pseudo-labels are rendered by

$$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$4

with clipping to $$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$5. Training uses

$$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$6

together with the same smoothness and sparsity regularizers as Sultani et al. (2018). The first epoch is a warm-up with $$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$7 only.

The empirical profile is strong. On UCF-Crime, GS-MoE reports 91.58% AUC and 83.86% AUC_A. On XD-Violence it reports 82.89% AP and 85.74% AP_A. On MSAD it reports 87.72% AUC, 69.54% AUC_A, 68.26% AP, and 76.68% AP_A. Component ablations on UCF-Crime AUC and XD-Violence AP_A progress from 86.97/82.91 for UR-DMU baseline, to 87.84/83.39 with TGS fine-tuning, to 89.53/84.16 with class experts, to 91.58/85.74 for the full GS-MoE. Threshold sensitivity on UCF-Crime peaks at 0.2, with AUC values 90.34, 91.08, 91.58, 91.23, and 90.75 for thresholds 0.1, 0.15, 0.2, 0.25, and 0.3. The computational profile on UCF-Crime with 13 experts is 4.133 GFLOPs, 16.02M parameters, and 9.57 FPS end-to-end, with the note that experts are processed sequentially in the current implementation.

5. Shared mathematical structure and GS-derived supervisory signals

Despite their domain differences, these systems share a recognizable pattern. In the pose-free and dynamic 3DGS settings, splatting starts from 3D Gaussians with means, covariances, and opacity, projects them to image space via Jacobian-linearized covariance, and uses front-to-back compositing to create either renderings or routing features. In the WSVAD setting, splatting instead places Gaussian kernels along the temporal axis around peak anomaly positions and sums them into pseudo-labels (Jeong et al., 22 Mar 2026, Jin et al., 22 Oct 2025, D'Amicantonio et al., 8 Aug 2025).

In 2Xplat, GS-derived supervision is dual. The appearance expert is supervised by differentiable 3DGS rendering losses on target views, using MSE and perceptual loss. The geometry expert is supervised explicitly by relative pose and intrinsics terms and implicitly by rendering loss because pose errors change renderings and gradients backpropagate through pose-conditioned modules. The paper’s proposed extension to a formal GS-MoE makes this explicit by introducing a gating network $$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$8 conditioned on GS-derived signals such as projected Gaussian footprints $$L_{\text{render}} = L_{\text{MSE}}(\hat I, I) + \lambda_{\text{perc}} L_{\text{perc}}(\hat I, I),$$9, per-ray alpha or opacity profiles, depth ordering, view angle, and uncertainty from the geometry expert. The proposed mixture weights are written as

$$L_{\text{cam}} = \frac{1}{N(N-1)} \sum_{i \ne j} \left( \lambda_R L_R(i,j) + \lambda_t L_t(i,j) \right) + \frac{\lambda_K}{N} \sum_{j=1}^N L_K(j).$$0

$$L_{\text{cam}} = \frac{1}{N(N-1)} \sum_{i \ne j} \left( \lambda_R L_R(i,j) + \lambda_t L_t(i,j) \right) + \frac{\lambda_K}{N} \sum_{j=1}^N L_K(j).$$1

In MoE-GS for dynamic scenes, the same compositing machinery is repurposed for routing rather than only photometric synthesis. Gaussian-level weights are splatted into pixel-space features using the same transmittance and alpha visibility rules that produce RGB renderings. In WSVAD, the corresponding operation maps detected abnormal peaks into a temporally extended supervision field rather than a spatial one. A plausible implication is that GS-MoE unifies expert systems through a splatting operator that transports structured latent evidence—geometry, opacity, temporal prominence, or uncertainty—into a domain on which optimization can act directly.

6. Misconceptions, limitations, and prospective directions

A recurrent misconception is that GS-MoE necessarily denotes a formal softmax-gated MoE. The literature does not support that equivalence. In 2Xplat, there is no explicit gating module; routing is deterministic through predicted camera poses and pose-aware conditioning. In MoE-GS for dynamic Gaussian splatting, gating is explicit, softmax-normalized, and pixel-wise. In WSVAD, aggregation is input-dependent and dense, but there is no explicit softmax-normalized expert weight vector; fusion is learned through bi-directional cross-attention, a transformer block, and an MLP (Jeong et al., 22 Mar 2026, Jin et al., 22 Oct 2025, D'Amicantonio et al., 8 Aug 2025).

A second misconception is that Gaussian splatting in GS-MoE always refers to 3D scene rendering. The anomaly-detection formulation uses “splatting” on the temporal axis, where peaks become centers $$L_{\text{cam}} = \frac{1}{N(N-1)} \sum_{i \ne j} \left( \lambda_R L_R(i,j) + \lambda_t L_t(i,j) \right) + \frac{\lambda_K}{N} \sum_{j=1}^N L_K(j).$$2 and widths and variances define temporal kernels that produce pseudo-labels. This broadens the term from a graphics primitive to a more general structured aggregation mechanism.

The principal limitations are domain-specific. In dynamic 3DGS MoE, increased model capacity and the total number of Gaussians reduce FPS and raise memory cost; naïve multi-pass rendering scales roughly with the number of experts $$L_{\text{cam}} = \frac{1}{N(N-1)} \sum_{i \ne j} \left( \lambda_R L_R(i,j) + \lambda_t L_t(i,j) \right) + \frac{\lambda_K}{N} \sum_{j=1}^N L_K(j).$$3, and gating errors can route pixels to suboptimal experts. In 2Xplat, pose accuracy slightly trails specialized pose estimators, and challenging cases include complex translucency, specularity, and severe intrinsics errors, though end-to-end training and evaluation-time pose alignment mitigate these issues. In WSVAD, performance depends on stable peak detection; thresholds below 0.1 cause many spurious peaks early, thresholds above 0.3 miss peaks, and end-to-end speed is limited because experts are processed sequentially in the current implementation.

The future directions proposed in these papers are convergent. The pose-free 3DGS work suggests adding multiple appearance sub-experts specialized for “texture,” “thin/transparent,” “specular,” and “background,” and multiple geometry sub-experts for “wide-baseline alignment,” “loop closure,” and “intrinsics calibration,” with gating driven by GS footprints, alpha profiles, view angles, and pose uncertainty. MoE-GS for dynamic scenes proposes adaptive expert specialization, tighter coupling to motion cues such as optical flow features, explicit temporal smoothing in gates, learned expert priors, and curriculum strategies. The anomaly-detection framework proposes cluster-experts when anomaly classes are unknown or mixed. Taken together, these directions indicate an emerging view of GS-MoE as a modular strategy for separating heterogeneous subproblems while retaining a common splatting-based supervisory substrate.