FuseMoE: Fleximodal Mixture-of-Experts

• FuseMoE is a multimodal fusion framework that combines modality-specific encoders with a sparse MoE layer and Transformer backbone to effectively integrate irregular and missing data.
• It demonstrated superior performance on MIMIC-IV benchmarks by achieving higher AUROC scores in tasks such as 48-hour in-hospital mortality, length of stay, and phenotype prediction.
• The innovative Laplace gating mechanism offers faster convergence and improved parameter estimation compared to traditional softmax gating in mixture-of-experts models.

FuseMoE is a mixture-of-experts Transformer framework for fleximodal fusion, introduced to integrate a diverse number of modalities while handling missing modalities, irregularly sampled data trajectories, and the scarcity of high-quality training samples. It combines modality-specific encoders, a sparse Mixture-of-Experts fusion layer, and a Transformer backbone, with a Laplace gating function proposed in place of conventional softmax gating. In the reported MIMIC-IV benchmarks—48-hour in-hospital mortality, length of stay, and 25-phenotype prediction—the Laplace-gated variant attains the best AUROC among the FuseMoE variants and exceeds the best reported baseline AUROC on all three tasks, while the theoretical analysis states faster parameter-estimation convergence than softmax gating (Han et al., 2024).

1. Problem formulation and design objective

FuseMoE is situated in a multimodal setting where predictive models must process heterogeneous signals that are both incomplete and temporally irregular. The motivating setting emphasizes critical fields in which machine learning models increasingly grapple with multimodal data, yet the available observations may be sparse, asynchronously sampled, or entirely missing for some modalities. The framework is therefore designed around two coupled requirements: multimodal integration across a variable set of inputs, and robustness to partial observability (Han et al., 2024).

The model’s use of “fleximodal” fusion denotes support for a diverse number of modalities rather than a fixed, always-present modality tuple. This is operationalized through modality-specific encoders, token-level expert routing, and a missing-modality mechanism based on learned indicator embeddings. A plausible implication is that FuseMoE is intended not merely as a fusion block, but as an end-to-end multimodal representation learner whose routing structure adapts to both modality identity and modality absence.

2. Architecture and data pathway

At a high level, FuseMoE consists of modality-specific encoders, discretization over a fixed window of length $\gamma$, a sparse MoE fusion layer, a backbone Transformer, and a task-specific head. For each modality $j$, discretization yields an embedding $Z_j \in \mathbb{R}^{\gamma \times d_e}$, after which each token is routed through the MoE layer and the resulting multimodal representation is passed to the Transformer backbone (Han et al., 2024).

Modality	Encoder	Output dimension
Time-series / lab values	Unified Temporal Discretization Embedding (UTDE) mixing simple imputation and mTAND	$d_e$ after discretization
Text notes	BioClinicalBERT	768-d
Chest X-ray	DenseNet-121	1024-d
ECG	1D CNN autoencoder	256-d

The tokenization and fusion path is explicit. Denoting the flattened or pooled modality-token sequence by $\{x_{j,t}\}_{t=1}^{\gamma}$, each token is processed by a set of $S$ expert MLPs, where each expert is $E_i:\mathbb{R}^{d_e}\to\mathbb{R}^{d_e}$. The MoE output for a token is

$$y_{j,t} = \sum_{i=1}^{S} G(x_{j,t})_i\,E_i(x_{j,t}).$$

The fused outputs are then concatenated or summed across modalities and passed through the Transformer backbone. The architecture may optionally stack multiple MoE-augmented Transformer blocks, followed by a task head for classification or regression (Han et al., 2024).

This structure separates modality-local feature extraction from cross-modal fusion. The design suggests that expert specialization occurs after each encoder has already mapped its modality into a common embedding regime, allowing routing to depend on token content rather than on raw modality format.

3. Gating mechanism and expert routing

The defining technical component of FuseMoE is its sparse top-$K$ gating network. For each modality embedding $x \in \mathbb{R}^{d_e}$, the gate computes logits and routes $j$0 to the top $j$1 experts, whose outputs are combined by a weighted sum. Let $j$2 denote gating parameters. The baseline softmax top-$j$3 gate is

$j$4

A Gaussian alternative, following Xu et al. 1994, replaces inner-product scoring with squared Euclidean distance: $j$5

FuseMoE proposes Laplace gating: $j$6

The final MoE output is

$j$7

The substantive difference from standard top-$j$8 softmax gating is therefore the metric used to define expert affinity. In FuseMoE, routing is based on $j$9 rather than $Z_j \in \mathbb{R}^{\gamma \times d_e}$0. The paper’s central claim is that this change is not merely heuristic: it is linked to improved convergence rates in the corresponding estimation problem (Han et al., 2024).

4. Theoretical analysis

The theoretical development is framed through a Laplace-gated Gaussian MoE model. Under Assumption 4.1, the data $Z_j \in \mathbb{R}^{\gamma \times d_e}$1, $Z_j \in \mathbb{R}^{\gamma \times d_e}$2, are i.i.d. from a model with $Z_j \in \mathbb{R}^{\gamma \times d_e}$3 true experts,

$Z_j \in \mathbb{R}^{\gamma \times d_e}$4

Theorem 4.1 states that, under standard boundedness conditions on $Z_j \in \mathbb{R}^{\gamma \times d_e}$5,

$Z_j \in \mathbb{R}^{\gamma \times d_e}$6

where $Z_j \in \mathbb{R}^{\gamma \times d_e}$7 is Total Variation distance and $Z_j \in \mathbb{R}^{\gamma \times d_e}$8 is the density of the MLE $Z_j \in \mathbb{R}^{\gamma \times d_e}$9. Theorem 4.2 extends this to the over-specified regime $d_e$0 by introducing a Voronoi-based loss $d_e$1 and establishing

$d_e$2

The most emphasized rate comparison concerns the parameters themselves. In the over-specified setting, Laplace gating yields $d_e$3 rates for the gating parameters $d_e$4 and expert slopes $d_e$5, independent of how many fitted components collapse into one true expert, and parametric $d_e$6 for intercepts $d_e$7. By contrast, under softmax gating, estimators of $d_e$8 converge as $d_e$9, where $\{x_{j,t}\}_{t=1}^{\gamma}$0 grows rapidly with component multiplicity. The proof sketch proceeds through empirical-process arguments for the $\{x_{j,t}\}_{t=1}^{\gamma}$1 density bound, a lower bound connecting Total Variation distance to a Voronoi loss on parameters, and a Taylor expansion of $\{x_{j,t}\}_{t=1}^{\gamma}$2 together with a contradiction argument based on non-vanishing lowest-order terms (Han et al., 2024).

These claims place FuseMoE in a comparatively unusual position among multimodal fusion models: its routing rule is justified not only by downstream task metrics but also by asymptotic estimation-rate arguments. This suggests that the choice of gating geometry is treated as a statistical identifiability issue as much as an architectural one.

5. Irregular sampling, missing modalities, and optimization

FuseMoE addresses temporal irregularity in time-series inputs through a Unified Temporal Discretization Embedding that combines imputation and mTAND. The multi-time attention network embeds each irregular timestamp $\{x_{j,t}\}_{t=1}^{\gamma}$3 via

$\{x_{j,t}\}_{t=1}^{\gamma}$4

uses queries on a regular grid $\{x_{j,t}\}_{t=1}^{\gamma}$5, keys on observed $\{x_{j,t}\}_{t=1}^{\gamma}$6, and values on $\{x_{j,t}\}_{t=1}^{\gamma}$7, and produces $\{x_{j,t}\}_{t=1}^{\gamma}$8. In parallel, an imputation baseline applies forward-fill or global-mean if no prior value exists. The two views are combined by a learned gate: $\{x_{j,t}\}_{t=1}^{\gamma}$9 This yields a discretized representation that can be aligned with the other modalities in the common fusion pipeline (Han et al., 2024).

Missing modalities are handled at the embedding level. If modality $S$0 is absent for an entire sample, its embedding $S$1 is replaced with a learnable missing-indicator vector $S$2. A per-modality router then assigns these missing embeddings to a small set of experts, and entropy regularization is used to concentrate missing cases onto under-used experts. The description further states that per-modality routers plus entropy loss push $S$3 to under-used experts so that when a modality is absent its experts have low gate weights.

The training objective combines task loss, entropy regularization, and parameter regularization: $S$4 with

$S$5

Cross-entropy is used for binary or multilabel classification tasks, including 48-IHM, LOS, and 25-PHE, and MSE is used for continuous LOS if formulated as regression. Optimization uses AdamW with learning rates $S$6 for the time-series encoder, $S$7 for BERT, and $S$8 for the rest; batch size is 2 with gradient accumulation to an effective 32; training runs for 8 epochs with early stopping on validation AUROC. The implementation summary also lists $S$9 experts, top-$E_i:\mathbb{R}^{d_e}\to\mathbb{R}^{d_e}$0 with disjoint top-$E_i:\mathbb{R}^{d_e}\to\mathbb{R}^{d_e}$1, 3 MoE layers, FFN size 512, and GeLU activation (Han et al., 2024).

6. Empirical results, scalability, and naming ambiguity

FuseMoE is evaluated against five baselines on three ICU tasks from MIMIC-IV: 48-hour in-hospital mortality (48-IHM), length of stay (LOS), and 25-phenotype multi-label prediction (25-PHE). The reported AUROC results are as follows (Han et al., 2024).

Task	Best FuseMoE result	Best baseline
48-IHM	Laplace MoE 81.03%	79.65%
LOS	Laplace 82.91 AUROC	82.58
25-PHE	Laplace 71.23 AUROC	70.55

For 48-IHM, the best AUROC is 81.03% for Laplace MoE, compared with 80.76% for the Gaussian variant and 79.65% for the best baseline. The F1 score on the same task is 46.53 for the Laplace variant versus 46.86 for the best baseline. When the modality set is expanded from 2 to 4 modalities by adding chest X-ray and ECG, Laplace gating shows a +2–4 point AUROC gain on 48-IHM. Ablation studies report that performance plateaus at $E_i:\mathbb{R}^{d_e}\to\mathbb{R}^{d_e}$2, motivating the choice top-$E_i:\mathbb{R}^{d_e}\to\mathbb{R}^{d_e}$3; per-modality routers slightly outperform a joint router when $E_i:\mathbb{R}^{d_e}\to\mathbb{R}^{d_e}$4; and the combination of per-modality routing with entropy regularization recovers performance in missing-modality settings, often outperforming “no-missing” models by leveraging more data. In compute terms, FuseMoE is described as mid-range in GPU memory relative to dense MoE or cross-attention baselines, and its sparse MoE layer adds approximately 10–20% more parameters but only approximately 5% more FLOPs due to top-$E_i:\mathbb{R}^{d_e}\to\mathbb{R}^{d_e}$5 selection (Han et al., 2024).

A recurring source of ambiguity is the name. In this usage, FuseMoE refers to “Mixture-of-Experts Transformers for Fleximodal Fusion” and denotes a multimodal fusion architecture centered on Laplace-gated sparse routing (Han et al., 2024). Separate later work uses closely related terminology for inference-kernel optimization rather than multimodal modeling: the TritonMoE paper describes a fused MoE dispatch kernel written in Triton for portable expert routing, token permutation, expert GEMMs, and weighted output combination across NVIDIA A100 and AMD MI300X (Mitra, 7 Apr 2026), while AdaFuse addresses token-level pre-gating and fused kernel optimization for dynamic adapter inference in open-source LLMs, with a single global routing decision reused throughout the network (Li et al., 12 Mar 2026). The overlap in naming therefore reflects a terminological collision, not a shared problem formulation.