Elastic Multimodal Parallelism (EMP)

• EMP is a systems paradigm that decouples modalities and stages, enabling dynamic GPU allocation for diverse multimodal tasks.
• It employs modular decoupling, elastic resource redistribution, and independent layout selection to optimize heterogeneous parallelism.
• EMP delivers significant throughput and latency improvements in MLLM serving and training, ensuring inference and training equivalence.

Searching arXiv for the cited EMP-related papers to ground the article in current literature. Querying arXiv metadata for the main EMP-related papers. $$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$8 $$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$9 Elastic Multimodal Parallelism (EMP) denotes a class of system designs in which multimodal workloads are not bound to a single static parallelization layout, but are instead decomposed by modality, stage, or module and executed with elastic resource allocation and parallelism choices. In the serving literature, EMP is explicitly defined as a paradigm for multimodal LLMs (MLLMs) that decouples modality groups and inference stages and dynamically reallocates GPUs and parallelism accordingly (Liu et al., 14 Jul 2025). In training-oriented work, a closely aligned abstraction appears as heterogeneous parallelism, where each module in a multimodal end-to-end graph can use an independent layout and rank placement, with explicit mechanisms to preserve activation and gradient semantics across layout boundaries (Karnati et al., 26 May 2026).

1. Definition and conceptual scope

EMP is most concretely formalized for MLLM serving as a two-level elastic scheduling space. At the modality level, requests are separated into groups such as text-only and multimodal. At the stage level, instances inside each group are partitioned into encoding, prefill, and decode stages. Decoupling and elasticity are applied at both levels: groups have separate queues and execution, while stage parallelism and GPU allocation are adjusted at runtime (Liu et al., 14 Jul 2025).

The formal decomposition used for MLLM inference separates the model into an encoder, prefill stage, and decode stage:

$$g_E: \mathcal{I} \rightarrow \mathcal{V}, \qquad g_P: \mathcal{V} \times \mathcal{T} \rightarrow \mathcal{H} \times \mathcal{K}, \qquad g_D: \mathcal{H} \times \mathcal{K} \rightarrow \mathcal{O},$$

with sequential execution written as

$f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$

Within EMP, these components may execute on different instances and under different parallel configurations while preserving inference equivalence (Liu et al., 14 Jul 2025).

A training-side analogue appears in multimodal large-model training, where modules such as encoders, projectors, embedding assembly, and the autoregressive decoder are treated as graph components with module-local logical layouts and physical rank sets:

$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$

This formulation makes layout a per-module decision rather than a model-wide constant, which closely matches the usual intuition behind the adjective “elastic” in EMP (Karnati et al., 26 May 2026).

A broad reading of the literature therefore treats EMP not as a single algorithm, but as a systems principle: parallelism itself becomes a schedulable and reconfigurable resource.

2. Architectural principles

The central motivation for EMP is that multimodal systems contain components with incompatible compute, memory, and communication profiles. In MLLM serving, multimodal requests add vision encoders, projectors, longer prompts, larger KV caches, and, for encoder–decoder models, cross-attention overhead. Text-only and multimodal traffic also exhibit different temporal behavior, with real traces described as showing fairly stable text-only traffic and bursty multimodal traffic. A single tightly coupled serving stack therefore couples heavy and light requests, raises time-to-first-token (TTFT), and reduces GPU utilization (Liu et al., 14 Jul 2025).

In training, the mismatch is especially pronounced when a global layout is chosen around the LLM and inherited by encoders. Large LLMs may require high tensor parallelism and, at long context, context parallelism, but smaller encoders may receive little benefit from inheriting those choices. The training paper states that as context length grows the LLM needs CP, while encoder sequence length is fixed; keeping encoder $\mathrm{CP}=1$ can turn LLM context-parallel ranks into encoder data-parallel replicas (Karnati et al., 26 May 2026).

Three architectural principles recur across the literature:

Module or stage decoupling: encode, prefill, decode, diffusion transformer, or modality-specific encoders are separated into independently managed execution units. This is explicit in ElasticMM’s stage-level manager and in disaggregated diffusion serving, where encoder, DiT, and decoder are run as separate services (Liu et al., 14 Jul 2025, Zha et al., 25 May 2026).

Elastic resource redistribution: GPU instances are reassigned between modality groups or stages according to measured demand. ElasticMM uses proactive and reactive mechanisms across modality groups, while DisagFusion adjusts the instance counts $(g_E,g_T,g_D)$ to rebalance pipeline stages under changing workloads (Liu et al., 14 Jul 2025, Zha et al., 25 May 2026).

Independent layout selection: adjacent parts of the graph may use different logical layouts. In training, this is formalized by independent $G_m$ and $R_m$ per module; in diffusion serving, GF-DiT treats the execution layout of each trajectory task as a scheduling decision; in ElasticMM, stage-level parallelism is adjusted with data-parallel instance allocation and preemption (Karnati et al., 26 May 2026, Qiang et al., 11 Jun 2026).

These principles distinguish EMP from static TP/DP/PP/CP configurations, whose shared-layout assumption is well suited to homogeneous transformer stacks but becomes restrictive once modalities, sequence lengths, or stage costs diverge.

3. Core mechanisms and execution models

ElasticMM implements EMP through a modality-aware load balancer, a stage-level manager, unified multimodal prefix caching, and non-blocking encoding. Its proactive modality allocation objective is expressed through burst tolerance

$bt(i)=\frac{N_i^{\text{peak}}}{N_i^{\text{avg}}},$

and idle instances are greedily assigned to maximize the minimum burst tolerance across modality groups. Reactive decisions compare intra-group adaptation with inter-group preemption using explicit gain–cost models (Liu et al., 14 Jul 2025).

For prefill scaling, the benefit of adding an instance is modeled as

$$\text{Gain}=\sum_{r\in R_p}\frac{T(R_p,E_p)-T(R_p,E_p\cup\{e_{\max}\})}{r.\text{input\_len}},$$

while the decode-side harm is

$$\text{Cost}=\sum_{r\in B_d}\frac{M(e_{\max})+w\cdot L(B_d,E_d\setminus\{e_{\max}\})}{r.\text{output\_len}}.$$

Decode scaling uses an analogous gain–cost comparison. The default stage-level strategy is data parallelism, with tensor parallelism used only when a single GPU cannot host the model (Liu et al., 14 Jul 2025).

Training systems centered on heterogeneous parallelism solve a different but related problem: preserving tensor semantics when adjacent modules use different layouts or placements. The key abstraction is the boundary communicator, which implements a forward layout transform $f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$0 and a backward transform $f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$1. The design supports both colocated execution on shared GPUs and non-colocated execution on disjoint rank sets. It also distinguishes equal-DP, fan-in, and fan-out transformations according to the relation between source and destination data-parallel degrees (Karnati et al., 26 May 2026).

Scheduling is correspondingly specialized. For colocated training layouts, a three-phase schedule is used: encoder forward plus colocated forward transform; detached LLM pipeline execution with standard 1F1B; and boundary-gradient handoff plus encoder backward. For non-colocated layouts, the scheduler operates over a PP-stage graph whose edges may be intra-module P2P or inter-module boundary communicators (Karnati et al., 26 May 2026).

Adjacent work on diffusion serving pushes the same idea further toward runtime programmability. DisagFusion introduces asynchronous pipeline parallelism and elastic scheduling for disaggregated diffusion serving, with encoder, DiT, and decoder as separate microservices. GF-DiT decomposes each request into independently schedulable trajectory tasks and lets the scheduler choose an execution layout per task, arguing that GPU parallelism should be treated as a first-class schedulable resource (Zha et al., 25 May 2026, Qiang et al., 11 Jun 2026).

4. Communication substrates, migration, and scheduling granularity

A recurrent systems problem in EMP is that elastic scheduling is only practical if layout changes, handoffs, and group formation can occur at low overhead.

In training, non-colocated heterogeneous execution reduces cross-module communication by using boundary leaders for each DP shard rather than naively connecting the Cartesian product of TP and CP groups. Cross-module traffic then scales with DP and global batch rather than the TP$f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$2CP product. In colocated mode, the paper reports boundary overhead of less than 1% of step time (Karnati et al., 26 May 2026).

In disaggregated serving, DisagFusion separates control and data planes. Metadata is passed through RDMA-backed ring buffers, while tensors use Mooncake Transfer Engine for zero-copy GPU–GPU movement through GPUDirect/RDMA. The system is fully asynchronous: stages enqueue metadata, send or receive latent tensors non-blockingly, and continue scheduling ready work instead of waiting for synchronous handoff completion (Zha et al., 25 May 2026).

GF-DiT addresses a more specific obstacle: the overhead of dynamic collective-group formation. Standard NCCL group initialization and the first collective on a new subgroup are too expensive for per-task reconfiguration. GF-DiT therefore introduces group-free collectives (GFC), which separate one-time world-level communication setup from lightweight logical group descriptors created at runtime. The paper reports that GFC reduces communication-group setup overhead from 778 ms to approximately 60 $f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$3s, while also enforcing pairwise-consistent ordering through per-edge double-buffered signaling (Qiang et al., 11 Jun 2026).

The systems differ in target workload, but a common pattern is evident: EMP requires semantic task boundaries at which migration, repartitioning, or GPU reassignment can occur without corrupting execution state. In MLLM serving those boundaries are stages such as encoding, prefill, and decode; in GF-DiT they are trajectory tasks such as encoder, a denoising step, or decoder; in training they are module boundaries equipped with explicit forward and backward transforms.

5. Representative systems and empirical evidence

The main empirical literature on EMP and closely aligned systems spans serving, training, and adjacent multimodal or diffusion runtimes.

System	Domain	Elastic object
ElasticMM	MLLM serving	Modality pools; encode/prefill/decode partitioning
Heterogeneous parallelism	Multimodal training	Per-module layouts and placements
DisagFusion / GF-DiT	Diffusion serving	Stage instance ratios; per-task execution layout

ElasticMM reports that, on ShareGPT-4o, TTFT is reduced by up to 4.2$f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$4 for Qwen2.5-VL-7B and by up to 3.5$f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$5 for LLaMA3.2-Vision-11B relative to vLLM; on VisualWebInstruct, TTFT reduction reaches 3.7$f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$6 and 2.9$f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$7, respectively. Under service-level objectives, throughput reaches up to 4.5$f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$8 that of vLLM for Qwen2.5-VL-7B and 3.2$f_{\mathcal{M}}(i,t)=g_D(g_P(g_E(i),t)).$9 for LLaMA3.2-Vision-11B. An ablation comparing EMP against static allocation policies reports throughput improvements of 1.8$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$0 for Qwen2.5-VL and 2.3$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$1 for LLaMA3.2-Vision (Liu et al., 14 Jul 2025).

The training work on heterogeneous parallelism reports that colocated heterogeneity improves TFLOPS/GPU by up to 49.3%, while non-colocated heterogeneity improves aggregate token throughput by up to 13.0% and TFLOPS/GPU by up to 9.6%. In long-context regimes, examples include gains of 34.3–49.3% TFLOPS/GPU for a 1B encoder plus 7B LLM at 64K sequence length, and 29.8–34.0% TFLOPS/GPU for a 1B encoder plus 14B LLM at 128K. The paper also identifies an operating-range transition: for small encoders with a 70B or 120B LLM, colocated heterogeneous execution is best, whereas from about 6B encoder scale onward non-colocated mode becomes superior (Karnati et al., 26 May 2026).

DisagFusion reports throughput improvements of 3.4$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$2–20.5$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$3 and end-to-end latency reductions of 18.5$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$4 relative to a monolithic baseline, while its dynamic experiments show that optimal stage ratios can change from 1:6:1 to 1:5:2 as the diffusion step count changes. GF-DiT reports throughput gains up to 6.01$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$5, mean latency reductions up to 95%, and SLO violation-rate reductions up to 90% compared with fixed-pipeline execution with static parallelism (Zha et al., 25 May 2026, Qiang et al., 11 Jun 2026).

A plausible implication is that EMP is most valuable precisely where heterogeneity is structurally unavoidable: mixed request types, long contexts, large encoder–decoder asymmetries, or dynamic service conditions that shift bottlenecks over time.

6. Correctness, terminology, and open issues

The literature repeatedly treats semantic preservation as a first-order requirement. ElasticMM formalizes inference equivalence between standard execution and EMP execution, proves stage-level and modality-level equivalence under deterministic data movement, and reports bit-identical outputs across 1,000 prompts. Its prefix caches satisfy

$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$6

so cache reuse is defined as exact reuse rather than approximation (Liu et al., 14 Jul 2025).

Training-oriented heterogeneous parallelism validates correctness through step-level FP32 parity against homogeneous baselines and through LLaVA-style training runs whose loss curves consistently track one another across many colocated and non-colocated layouts. DynaTrain, while not itself an EMP paper, further demonstrates that fast online switching across arbitrary multi-dimensional parallelism can preserve convergence, reporting reconfiguration of a 70B dense model in under 2s and of a 235B MoE model in 4.36s (Karnati et al., 26 May 2026, Wang et al., 12 May 2026).

Several misconceptions are therefore not supported by the published record. EMP is not merely a synonym for static TP/DP/PP tuning; its distinguishing feature is runtime or module-level decoupling of modalities, stages, or graph components. Nor is EMP confined to inference: closely aligned abstractions already exist for multimodal training, even when the acronym is not used (Karnati et al., 26 May 2026). At the same time, the acronym is historically ambiguous. In older architecture literature, “EMPA” denotes the Explicitly Many-Processor Approach, a hardware model based on quasi-threads, supervisor-managed core rental, and special meta-instructions such as QCreate, QTerm, and QAlloc; this usage is conceptually separate from contemporary multimodal systems usage (Végh, 2016).

Open issues are explicit in the current literature. ElasticMM is evaluated on a single 8$$G_m=(\mathrm{TP}_m,\mathrm{DP}_m,\mathrm{PP}_m,\mathrm{CP}_m,\ldots), \qquad R_m \subset \{0,\dots,W-1\}.$$7A800 node, and multi-node distributed scenarios are left to future work. DisagFusion assumes a fixed three-stage encoder–DiT–decoder pipeline and does not provide arbitrary DAG execution. GF-DiT dynamically varies execution-group size for diffusion tasks, but its dynamic parallelism space is narrower than a full multimodal system with jointly tunable data, tensor, and pipeline parallelism. DynaTrain assumes a fixed model architecture across transitions and does not incorporate long-lived activations or KV caches into its virtual state abstraction (Liu et al., 14 Jul 2025, Zha et al., 25 May 2026, Qiang et al., 11 Jun 2026, Wang et al., 12 May 2026).

Taken together, these results place EMP as an emerging systems paradigm rather than a single framework: multimodal computation is decomposed into semantically coherent units, parallelism becomes an elastic runtime decision, and correctness is maintained by exact state-transfer or boundary-transform mechanisms.