Efficient Multimodal Diffusion Transformer (E-MMDiT)

• The paper introduces E-MMDiT, which employs 32× visual compression and multi-path token reduction to drastically cut computation in multimodal diffusion.
• Its architecture incorporates Position Reinforcement, Alternating Subregion Attention, and AdaLN-affine to maintain synthesis quality while optimizing efficiency.
• Experiments on 512px and 1024px outputs demonstrate that a 304M-parameter model can achieve competitive quality and high throughput compared to heavier systems.

Searching arXiv for the E-MMDiT paper and closely related multimodal diffusion transformer efficiency papers. Efficient Multimodal Diffusion Transformer (E-MMDiT) is a lightweight text-to-image diffusion transformer introduced in “E-MMDiT: Revisiting Multimodal Diffusion Transformer Design for Fast Image Synthesis under Limited Resources” (Shen et al., 31 Oct 2025). It is an efficiency-first redesign of the multimodal diffusion transformer (MMDiT) that targets fast image synthesis under limited training and inference resources through aggressive token reduction, a highly compressive visual tokenizer, a multi-path compression module, Position Reinforcement, Alternating Subregion Attention (ASA), and AdaLN-affine. The reported system has 304M parameters, supports 512px and 1024px generation, is trained on 25M public text-image pairs for the 512px model, and the 512px model is reported to be trained from scratch in 1.5 days on a single node of 8 AMD MI300X GPUs (Shen et al., 31 Oct 2025).

1. Definition and design objective

E-MMDiT is defined as an “efficient and lightweight multimodal diffusion model with only 304M parameters for fast image synthesis requiring low training resources” (Shen et al., 31 Oct 2025). Its design premise is that, for diffusion transformers, token count is the dominant systems bottleneck, because self-attention has quadratic complexity, written in the paper as $O(N^2)$, where $N$ is the number of tokens (Shen et al., 31 Oct 2025). The architecture therefore centers on reducing the number of visual tokens before and within the backbone, rather than relying only on post hoc compression or distillation.

The model is positioned against heavier text-to-image systems that either require large-scale data and compute or remain burdened by high-latency transformer structures (Shen et al., 31 Oct 2025). In that sense, E-MMDiT is not merely a smaller MMDiT; it is a token-efficient reorganization of the backbone. This suggests that its principal contribution lies in architectural cost control rather than in changing the underlying multimodal diffusion paradigm.

The backbone remains MMDiT-based. The paper states that E-MMDiT builds on MMDiT, where different modalities use separate sets of weights, and inter-modality interaction is achieved by a joint attention mechanism over concatenated features (Shen et al., 31 Oct 2025). Text prompts are encoded by Llama 3.2-1B, images are encoded by DC-AE, and denoising is performed in latent space with Rectified Flow (Shen et al., 31 Oct 2025).

2. Architectural pipeline

The end-to-end pipeline begins by encoding the prompt with Llama 3.2-1B and the target image with DC-AE, which uses a 32× compression ratio (Shen et al., 31 Oct 2025). The diffusion transformer then operates on latent tokens rather than pixels. The paper states that the main backbone contains 24 Transformer blocks, 24 attention heads, and 32 channels per head, with an FFN multiplier of 3 rather than 4 (Shen et al., 31 Oct 2025).

The 24 blocks are partitioned into three stages: $[N_1, N_2, N_3] = [4, 16, 4]$ The first 4 blocks operate at the original latent-token resolution; then a multi-path compression module compresses the visual token stream; the next 16 blocks process the compressed representation; a token reconstructor restores the original token resolution; and the final 4 blocks refine the reconstructed sequence before prediction (Shen et al., 31 Oct 2025). The model figure described in the paper places Position Reinforcement after token reconstruction and AdaLN-affine as the block-conditioning mechanism (Shen et al., 31 Oct 2025).

Rectified Flow is used in latent space. The forward process is written as

$x_t=(1-\sigma_t)x_0+\sigma_t\bm{\epsilon}$

where $x_0$ is the clean image latent and $\bm{\epsilon}\sim\mathcal{N}(0,I)$ (Shen et al., 31 Oct 2025). The training loss is given as

$\mathcal{L}_\text{RF}(\bm{\theta}) := \mathbb{E}_{\bm{\epsilon}\sim\mathcal{N}(0, I), t}\left\| (\bm{\epsilon}-x_0)-v_{\bm{\theta}(x_t,t) \right\|_2^2$

and the full objective is

$$\mathcal{L} := \mathcal{L}_\text{RF} + \lambda\mathcal{L}_\text{REPA}$$

with REPA used as an auxiliary regularizer in stage 1 training (Shen et al., 31 Oct 2025).

The architectural logic is explicitly coarse in its first tokenization decision and fine in later restoration. This suggests a design in which the model preserves multimodal fusion while shifting most heavy computation into a compressed token regime.

3. Token-efficiency mechanisms

The most consequential design choice is the use of DC-AE as a highly compressive visual tokenizer. The paper contrasts this with common DiT-style pipelines that combine 8× latent compression with patch size 2, yielding an effective 16× reduction, whereas E-MMDiT uses 32× downsampling without further patchification (Shen et al., 31 Oct 2025). The paper states that this yields a 75% reduction in token count relative to the 16× setup (Shen et al., 31 Oct 2025).

Within the backbone, the multi-path compression module further reduces visual tokens by processing two compressed branches in parallel: 2× compression and 4× compression (Shen et al., 31 Oct 2025). The compressor is inspired by TokenShuffle and uses local token merging along the channel dimension followed by small MLPs; the reconstructor uses three MLPs, two for upsampling and one for token fusion, together with a skip connection from early blocks (Shen et al., 31 Oct 2025). The paper states that this achieves a token reduction comparable to MicroDiT’s deferred masking, specifically 68.5% (Shen et al., 31 Oct 2025).

The ablation results make the compression design unusually concrete. On ImageNet 256, the proposed two-branch module achieves 89.77G FLOPs, 343M parameters, FID 22.42, and IS 58.65, outperforming 2× only, 4× only, stacked 2×, and w/o skip variants (Shen et al., 31 Oct 2025). The degradation of the 4× only setting to FID 33.52 and IS 41.43 indicates that aggressive one-branch compression is too destructive, while the strong drop in the w/o skip variant shows that the skip connection is structurally important (Shen et al., 31 Oct 2025).

Position Reinforcement is introduced because compression and reconstruction weaken spatial cues. The model uses absolute positional embeddings constructed from sine and cosine functions and re-applies them after reconstruction (Shen et al., 31 Oct 2025). The best ablation is PR only on reconstructed tokens, reported as FID 22.42 and IS 58.65; removing PR gives FID 24.78 and IS 53.85, while applying PR on compressed tokens degrades performance further to FID 26.56 and IS 51.23 (Shen et al., 31 Oct 2025). This indicates that positional restoration is helpful after reconstruction but harmful if imposed inside the compressed stage.

Alternating Subregion Attention (ASA) is the attention-efficiency mechanism. The implementation divides the token sequence $x \in \mathbb{R}^{B\times L\times C}$ into subregions through the tuple $(\text{region\_num}, \text{chunk\_size})$, with the paper giving the implementation $[N_1, N_2, N_3] = [4, 16, 4]$2 (Shen et al., 31 Oct 2025). In the final design, every three blocks use the schedule

$N$0

which the paper describes as one full-attention block followed by two subregion attention blocks (Shen et al., 31 Oct 2025). In ablation, attention FLOPs fall from 12.9G without ASA to 6.4G with the chosen schedule, while quality remains competitive: FID 23.50, IS 59.40 versus FID 23.33, IS 58.18 without ASA (Shen et al., 31 Oct 2025). By contrast, all-subregion attention $N$1 reduces attention FLOPs further to 3.2G but degrades to FID 26.54 and IS 55.16 (Shen et al., 31 Oct 2025).

4. Conditioning and optimization

E-MMDiT uses AdaLN-affine as a lightweight alternative to heavier per-block adaptive modulation. The paper first recalls AdaLN-single: $N$2 and then defines AdaLN-affine as

$N$3

where $N$4 is the shared global modulation vector, $N$5 is a learned block-specific bias, and $N$6 is a learned block-specific scale (Shen et al., 31 Oct 2025). In ablation, AdaLN-affine retains the same 89.77G FLOPs and 343M parameters as AdaLN-single, but improves quality from FID 22.94, IS 56.60 to FID 22.42, IS 58.65 (Shen et al., 31 Oct 2025).

Training uses AdamW, batch size 2048, and a two-stage main schedule: 100k iterations in stage 1 with learning rate 3e-4, followed by 50k iterations in stage 2, with EMA enabled in stage 2 and REPA omitted there (Shen et al., 31 Oct 2025). The data are entirely public. For text-to-image training, the paper lists SA1B: 11.1M, JourneyDB: 4.4M, and FLUXDB: 9.5M, totaling 25M text-image pairs (Shen et al., 31 Oct 2025). The paper also states that image and text features are pre-computed to accelerate training (Shen et al., 31 Oct 2025).

The REPA term is described with the printed form

$N$7

where $N$8 is a pretrained visual encoder such as DINOv2, $N$9 is a diffusion-model feature, and $[N_1, N_2, N_3] = [4, 16, 4]$0 is a projection head (Shen et al., 31 Oct 2025). The notation is visibly corrupted in the extracted text, but the stated role of REPA as representation alignment is explicit.

The post-training stage uses GRPO for 2k iterations, with reward based on GenEval and HPSv2.1 (Shen et al., 31 Oct 2025). Distillation is also applied following Nitro-1, using 1M synthetic data from the teacher and supporting 1–4 step generation (Shen et al., 31 Oct 2025). The paper does not specify the exact distillation loss or the exact GRPO objective, so a more detailed formalization would be speculative.

5. Reported performance and efficiency

The main reported results are summarized below.

Model	Key reported result	Efficiency figure
E-MMDiT-512	GenEval 0.66	18.83 samples/s, 398 ms, 0.08 TFLOPs
E-MMDiT-512-GRPO	GenEval 0.72	post-training variant
E-MMDiT-1024	GenEval 0.66	5.54 samples/s, 432 ms, 0.25 TFLOPs
E-MMDiT-1024-GRPO	GenEval 0.71	post-training variant

At 512px, the paper reports GenEval 0.66, IR 0.97, HPS 29.82, DPG 81.60, 18.83 samples/s, 398 ms latency, 304M parameters, and 0.08 TFLOPs for E-MMDiT-512 (Shen et al., 31 Oct 2025). The GRPO version reaches GenEval 0.72 and DPG 82.04 (Shen et al., 31 Oct 2025). At 1024px, E-MMDiT-1024 is reported at GenEval 0.66, IR 0.98, HPS 30.16, DPG 82.35, 5.54 samples/s, 432 ms, and 0.25 TFLOPs, with the GRPO version reaching GenEval 0.71 and IR 1.00 (Shen et al., 31 Oct 2025).

Relative to other lightweight baselines, the paper reports PixArt-Σ at GenEval 0.52 for 512px and 0.54 for 1024px, PixArt-α at 0.48 and 0.47, Sana-0.6B at 0.64 and 0.64, and MicroDiT at 0.46 for 512px (Shen et al., 31 Oct 2025). E-MMDiT is therefore presented as having the best reported GenEval among the listed lightweight baselines at both resolutions (Shen et al., 31 Oct 2025).

The throughput comparison is particularly sharp. At 512px, the paper reports 18.83 samples/s for E-MMDiT-512, versus 6.13 for Sana-0.6B, 4.98 for SDv2, 3.58 for SDv1.5, 3.02 for PixArt-α/Σ, and 0.70 for MicroDiT (Shen et al., 31 Oct 2025). At 1024px, it reports 5.54 samples/s for E-MMDiT-1024, versus 1.88 for Sana-0.6B, 0.52 for PixArt-Σ, and 0.54 for PixArt-α (Shen et al., 31 Oct 2025). The distilled models further improve throughput to 39.36 at 512px and 11.7 at 1024px, while keeping GenEval at 0.67 and 0.65, respectively (Shen et al., 31 Oct 2025).

These numbers establish the paper’s central empirical claim: a 304M-parameter MMDiT-based model trained on 25M public data can remain competitive on prompt-following metrics while delivering unusually high throughput on a single AMD MI300X GPU (Shen et al., 31 Oct 2025).

6. Position within efficient multimodal diffusion transformer research

E-MMDiT belongs to a broader line of work that seeks to reduce the cost of multimodal diffusion transformers without abandoning MMDiT-style joint multimodal attention. “EDiT: Efficient Diffusion Transformers with Linear Compressed Attention” introduces MM-EDiT, which applies linear compressed attention to image-to-image interactions while keeping standard scaled dot-product attention for prompt-related interactions; it reports up to 2.2× on-device speedup after distillation on SD-v3.5M (Becker et al., 20 Mar 2025). Relative to MM-EDiT, E-MMDiT places more emphasis on token reduction, compression/reconstruction, and subregion attention, rather than replacing the attention operator itself.

Other efficiency directions preserve MMDiT-like backbones but change where compute is spent. “Elastic Diffusion Transformer” equips each block with a lightweight router for sample-dependent skipping and adaptive MLP width reduction, reporting roughly $[N_1, N_2, N_3] = [4, 16, 4]$1 acceleration while keeping generation quality near the base model (Wang et al., 15 Feb 2026). “Hybrid Diffusion Transformer for Instruction-Guided Audio Editing via Rectified Flow” uses low-resolution DSJA-MMDiT blocks for coarse multimodal alignment and alternates them with AZCA-DiT refinement blocks at high resolution, thereby reducing the cost of full joint attention in audio editing (Gao et al., 18 Jun 2026). These papers differ in domain and mechanism, but all share the premise that full all-layer multimodal attention is often an inefficient default.

E-MMDiT also differs from training-free inference-control studies such as “Unraveling MMDiT Blocks” (Li et al., 5 Jan 2026), which identifies semantically critical early blocks and less critical middle blocks, and from position-control methods such as “Stitch” (Bader et al., 30 Sep 2025), which manipulate attention masks and branch scheduling at inference time. E-MMDiT is instead a trained backbone redesign with public-data reproducibility as a first-order objective (Shen et al., 31 Oct 2025).

The paper’s limitations are mostly implicit. It does not provide a fully specified optimizer recipe beyond the main schedule; exact scheduler details, AdamW betas, weight decay, guidance scale, and some post-training specifics are deferred (Shen et al., 31 Oct 2025). The strongest alignment numbers also require GRPO, and some engineering details—especially for exact reproduction of post-training and distillation—are not fully formalized in the paper text (Shen et al., 31 Oct 2025). A plausible implication is that E-MMDiT serves most clearly as a practical baseline: compact, fast, and open, but not necessarily the final word on the quality ceiling of efficient MMDiT systems.

In summary, E-MMDiT designates a specific efficiency-oriented multimodal diffusion transformer architecture in which 32× visual compression, two-branch token compression, reconstruction-time positional reinforcement, alternating subregion attention, and AdaLN-affine are combined into a 304M-parameter MMDiT backbone for fast text-to-image generation (Shen et al., 31 Oct 2025). Its reported significance lies in demonstrating that competitive multimodal diffusion can be trained on public data with comparatively modest hardware, while retaining the joint-attention logic of MMDiT and achieving strong throughput at both 512px and 1024px.