mmFormer: Robust Incomplete Multimodal Transformer

• The paper introduces mmFormer, a unified Transformer-CNN hybrid that integrates modality-specific convolutional encoders with intra- and inter-modal Transformers for robust segmentation under any missing modality configuration.
• It achieves significant improvements, including a 19.07% Dice increase on enhancing tumor segmentation in single-modality scenarios, by leveraging feature alignment and a Dice-based training objective.
• The model’s versatility is further demonstrated by its adaptation to liver fibrosis staging, where a missing-modality compensation module extends its applicability from segmentation to classification tasks.

Searching arXiv for the core mmFormer papers and closely related variants. mmFormer is a multimodal medical Transformer for incomplete multimodal learning, introduced for brain tumor segmentation from multi-contrast MRI under arbitrary missing-modality patterns. Its defining design combines hybrid modality-specific encoders, which couple convolutional feature extraction with intra-modal Transformer blocks, with a modality-correlated encoder that models long-range cross-modal dependencies and a decoder that reconstructs robust segmentation masks from modality-invariant representations (Zhang et al., 2022). In subsequent work, the same backbone was adapted from segmentation to classification for liver fibrosis staging, where the original modality-specific and cross-modal encoding scheme was retained and extended with explicit missing-modality compensation (Zhang et al., 19 Sep 2025).

1. Clinical setting and formulation of incomplete multimodal learning

The original formulation of mmFormer addresses brain tumor segmentation in the BraTS setting, where four MRI modalities—FLAIR, T1c, T1, and T2—provide complementary information about tumor extent and tissue characteristics (Zhang et al., 2022). The central difficulty is that clinical acquisitions are often incomplete because of scanning protocol differences, corrupted scans, time and cost constraints, or patient condition. This produces an inference regime in which the model must operate correctly for any non-empty subset of modalities rather than only for the full four-channel input.

Within that setting, incomplete multimodal learning is not merely a missing-data nuisance variable; it changes the representation problem itself. Multimodal segmentation relies on complementary cues, so naive fusion degrades when one or more inputs are absent. The original mmFormer paper frames this as a limitation of prevailing multimodal segmentation pipelines, which often either assume complete inputs or depend on mean/variance aggregation, missing-modality generation, or distillation strategies that complicate training and deployment (Zhang et al., 2022).

A useful characterization of mmFormer is therefore as a unified model for combinatorial modality absence. Instead of training separate subnetworks for each subset, it is trained once and evaluated on all 15 non-empty subsets of the four BraTS modalities. This design choice is central to its identity: robustness is not a post hoc wrapper around a full-modality model, but an explicit architectural and training objective.

2. Core architecture

mmFormer uses an encoder-decoder organization with three principal components: hybrid modality-specific encoders, a modality-correlated encoder, and a convolutional decoder (Zhang et al., 2022). The architecture is multimodal but not monolithic; each modality first traverses its own pathway before cross-modal interaction occurs.

For each modality $$m \in \{ \mathrm{FLAIR}, \mathrm{T1c}, \mathrm{T1}, \mathrm{T2} \}$$, the input volume is represented as

$$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$

and a modality-specific convolutional encoder extracts local-context feature maps,

$$\mathbf{F}^{local}_m = \mathcal{F}^{conv}_m(\mathbf{X}_m; \theta^{conv}_m).$$

This branch is U-Net-like, with 5 stages, two convolutional blocks per stage, group norm, ReLU, $3\times 3$ convolutions, stride-2 downsampling, and channel sizes 16, 32, 64, 128, and 256. Its role is to preserve fine-grained spatial structure, boundaries, and texture cues that are essential for dense prediction.

The local feature maps are then tokenized for intra-modal global modeling:

$$\mathbf{F}^{token}_m = \mathbf{F}^{local}_m\mathbf{W}_m + \mathbf{P}_m,$$

where $\mathbf{P}_m$ is a learnable positional embedding. The intra-modal Transformer applies multi-head self-attention and feed-forward processing to model long-range dependencies within a single modality. This yields modality-specific global features $\mathbf{F}^{global}_m$, which complement the convolutional branch rather than replacing it.

Cross-modal fusion is handled by the modality-correlated encoder, described in the original paper as an inter-modal Transformer. Its input is the concatenation of modality-specific global features, modulated by modality-level dropout:

$$\mathbf{F}^{token} = [\delta_{FLAIR}\mathbf{F}^{global}_{FLAIR}, \delta_{T1c}\mathbf{F}^{global}_{T1c}, \delta_{T1}\mathbf{F}^{global}_{T1}, \delta_{T2}\mathbf{F}^{global}_{T2}]\mathbf{W} + \mathbf{P},$$

with $\delta_m \in \{0,1\}$. During training, $\delta_m=0$ simulates a missing modality; missing modalities are represented as zero vectors. Transformer self-attention over this token sequence is used to align long-range cross-modal correlations and produce modality-invariant features with global tumor semantics (Zhang et al., 2022).

The decoder is a convolutional decoder with a U-Net-like symmetric structure. The output of the inter-modal Transformer is reshaped into feature maps, progressively upsampled, and fused with skip connections. The skip pathways concatenate modality-specific encoder features at each level, so the decoder combines global modality-invariant semantics with restored spatial detail.

3. Representation learning strategy and robustness mechanisms

The architectural logic of mmFormer is based on a layered division of labor. Convolutional encoders model local spatial context, intra-modal Transformers model within-modality long-range structure, and the inter-modal Transformer models cross-modal alignment at a global scale (Zhang et al., 2022). This implies that robustness to missing modalities is pursued through feature alignment and representation invariance rather than through explicit image synthesis.

The original paper strengthens this design with auxiliary regularizers in both the encoder and decoder. On the encoder side, each modality-specific convolutional encoder is encouraged to segment tumors independently by attaching a shared-weight decoder and supervising its output. On the decoder side, deep supervision is applied at intermediate decoding stages. These regularizers serve several purposes simultaneously: they discourage over-reliance on the most discriminative modalities, improve optimization, and help preserve performance when the available input subset is sparse.

The training objective is Dice-based. The paper writes the segmentation loss as

$$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$0

and the total loss as

$$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$1

These terms formalize the model’s dual objective: accurate final segmentation and stable intermediate representations under modality dropout.

A recurring misconception is to treat mmFormer as a pure Transformer model. The original formulation does not support that reading. It is explicitly a Transformer-CNN hybrid, and its performance claims are tied to the interaction between convolutional locality, intra-modal global context, inter-modal alignment, and auxiliary regularization rather than to attention alone (Zhang et al., 2022).

4. BraTS 2018 evaluation and empirical behavior

The original evaluation is conducted on BraTS 2018, comprising 285 multi-contrast MRI scans with modalities T1, T1c, T2, and FLAIR, and segmentation targets grouped into Enhancing Tumor (ET), Tumor Core (TC), and Whole Tumor (WT) (Zhang et al., 2022). Input volumes are resized to $$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$2; training uses batch size 1, random flip, crop, intensity shifts, the Adam optimizer with initial learning rate 0.0002, and 1000 epochs on 4 NVIDIA Tesla V100 GPUs. The reported model size is 106M parameters with 748G FLOPs.

The evaluation covers all 15 non-empty modality subsets: four single-modality cases, six two-modality cases, four three-modality cases, and the full four-modality configuration. This exhaustive subset protocol is important because mmFormer’s intended contribution is not only high full-modality accuracy but unified performance under arbitrary incompleteness.

The main reported average Dice scores over the 15 modality combinations are as follows.

Method	ET	TC	WT
HeMIS	46.10	62.57	74.05
U-HVED	46.76	64.84	79.16
mmFormer	59.85	72.97	82.94

These averages indicate that mmFormer outperforms HeMIS and U-HVED on almost all incomplete-modality settings; the paper states that it is stronger on ET and TC for all 15 modality combinations and on WT for 12 of 15 combinations (Zhang et al., 2022). The improvement is especially pronounced when information is sparse. The paper highlights an average 19.07% Dice improvement on enhancing tumor segmentation when only one modality is available.

The paper also positions mmFormer against ACN, a distillation-based method. ACN requires 672 hours, 144M parameters, and 1 teacher plus 14 students, whereas mmFormer uses a single unified model trained in 25 hours with 106M parameters. Even with that efficiency difference, the reported average Dice scores remain close: ACN achieves 61.21, 77.62, and 85.92 on ET, TC, and WT, versus 59.85, 72.97, and 82.94 for mmFormer (Zhang et al., 2022). This supports the interpretation of mmFormer as a computationally simpler unified alternative rather than a distillation-heavy ensemble.

In the full-modality setting, mmFormer is also compared with TransBTS. The reported Dice scores are 72.66 / 72.69 / 79.99 for TransBTS and 77.61 / 85.78 / 89.64 for mmFormer on ET / TC / WT. The original paper further reports that removing the intra-modal Transformer, the inter-modal Transformer, or the auxiliary regularizer reduces performance, which ties the empirical gains to the full composite design rather than to any single component (Zhang et al., 2022).

5. Extension to liver fibrosis staging with missing-modality compensation

A later work adapts mmFormer from segmentation to classification for the CARE 2025 Liver Fibrosis Staging task and preserves the original backbone structure: hybrid modality-specific encoders and a modality-correlated encoder (Zhang et al., 19 Sep 2025). In that adaptation, each MRI modality—T1WI, T2WI, and DWI—has its own branch comprising a 3D convolutional encoder and an intra-modality Transformer, and the resulting modality tokens are concatenated and processed by a modality-correlated encoder for cross-modal interaction.

The principal addition is a missing-modality compensation module. Missing modalities are replaced by zero-filled volumes, and a binary modality-availability mask specifies whether each modality is present. A learnable affine calibration is then applied to modality tokens using per-channel mean $$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$3, scale-like parameter $$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$4, reliability weights $$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$5, and $$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$6. A reference token is built by averaging the calibrated features of all available modalities,

$$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$7

and this reference sequence is passed through an MSA block to produce proxy features for missing modalities. After a second calibration and FFN refinement, the proxy is attenuated by a fixed factor $$\mathbf{X}_m \in \mathbb{R}^{1\times D\times H\times W},$$8. The intended role of this mechanism is feature-level compensation rather than full modality reconstruction.

The segmentation decoder of the original mmFormer is replaced with a classification head consisting of layer normalization, global average pooling over tokens, a linear projection with dropout, an MLP, and a final linear layer. The work also uses a 4-fold cross-validation ensemble with soft voting over class probabilities at inference time (Zhang et al., 19 Sep 2025).

On the CARE 2025 LiFS benchmark using the LiQA dataset, the study reports the following in-distribution vendor results:

Task	ACC	AUC
Cirrhosis Detection	66.67%	71.73%
Substantial Fibrosis Detection	74.17%	68.48%

For the out-of-distribution vendor subset, the reported values are 64.29% ACC and 68.83% AUC for Cirrhosis Detection, and 91.43% ACC and 71.38% AUC for Substantial Fibrosis Detection (Zhang et al., 19 Sep 2025). The paper notes that the proposed model is competitive rather than uniformly dominant: it improves accuracy by 6.67% over CE for cirrhosis detection, achieves second-best accuracy for substantial fibrosis detection, and is 0.04% behind the third-place method in AUC on the substantial fibrosis task. This suggests that the mmFormer backbone is portable beyond segmentation, but that classification under real-world vendor shift remains an open problem.

6. Nomenclature, related models, and scope boundaries

The name “mmFormer” admits several potential ambiguities. The most important is that the medical model “mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation” (Zhang et al., 2022) is distinct from “MMFormer: Multimodal Transformer Using Multiscale Self-Attention for Remote Sensing Image Classification” (Zhang et al., 2023). In the supplied record for the latter, the associated text is described as a LaTeX template-style document titled “CAMST: Cross Attention and Multi-scale Transformer for Classification of Hyperspectral and LiDAR Fusion data,” and it explicitly does not contain the MMFormer architecture, tokenization equations, self-attention or multiscale attention formulas, training protocol, or benchmark tables. For that reason, the remote-sensing item does not provide a reliable basis for technical exposition of a separate MMFormer architecture.

mmFormer is also distinct from FMMformer, an efficient-transformer model inspired by the fast multipole method (Nguyen et al., 2021). FMMformer decomposes attention into near-field and far-field components, modeling the former by a banded matrix and the latter by a low-rank matrix, with linear complexity in sequence length. Despite the superficial orthographic similarity, its problem setting, mathematical construction, and evaluation domain are unrelated to incomplete multimodal medical segmentation.

A second boundary concerns what mmFormer is not. In its original form, it is not a modality-generation model, although it is evaluated under missing modalities. Its strategy is to use modality-level dropout, inter-modal alignment, and auxiliary regularization to learn modality-invariant semantics from arbitrary subsets (Zhang et al., 2022). Only later derivative work adds explicit feature-level proxy synthesis for absent inputs (Zhang et al., 19 Sep 2025). Treating the original model as a missing-modality imputation architecture would therefore conflate the 2022 segmentation framework with subsequent compensation-based adaptations.

Within multimodal medical image analysis, mmFormer is best understood as a unified Transformer-CNN architecture whose central contribution is to make incomplete-modality segmentation a first-class modeling target. Its significance lies not in replacing convolutional segmentation pipelines wholesale, but in showing that local convolutional priors, intra-modal global context, inter-modal long-range alignment, and regularized subset training can be integrated into a single model that remains effective across the full lattice of available modality combinations (Zhang et al., 2022).