ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving

Abstract: Current LLM inference systems universally deploy ultra-large-scale models using a combination of Tensor Parallelism (TP) and Pipeline Parallelism (PP). However, existing systems treat the model parallelism topology as a static configuration that cannot be flexibly adjusted at runtime. This rigid design creates a fundamental contradiction with the dynamically changing inference workloads in real-world scenarios. State-of-the-art systems lack online reconfiguration capabilities and can only switch configurations by restarting the service, resulting in several minutes of service interruption, KV cache loss, and prohibitive recomputation overhead. To address this problem, this paper presents ReMP, a runtime model parallelism reconfiguration framework that supports low downtime. ReMP achieves dynamic adjustment through three key techniques: (1) decoupling the model parallelism topology from runtime state to avoid full service reconstruction; (2) designing a two-dimensional KV cache migration mechanism to preserve reusable cache states after TP/PP changes; and (3) implementing end-to-end online reconfiguration. Experiments demonstrate that ReMP can complete most topology switches within 1-7 seconds on models ranging from 7B to 70B parameters, achieving speedups of tens to over a hundred times compared to the restart approach. Moreover, ReMP significantly outperforms fixed configurations under dynamic workloads, delivering superior performance in terms of TTFT, TPOT, and output throughput.

• The paper introduces ReMP, a framework that decouples runtime state from fixed TP/PP topology, enabling rapid topology reconfiguration.
• It employs concurrent model loading and layer-wise KV cache migration to minimize downtime and avoid redundant recomputation.
• Evaluation on 7B to 70B LLMs shows ReMP achieves topology switching in 1–7 seconds, significantly outperforming restart-based methods.

ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving

Motivation and Problem Statement

Traditional LLM serving systems rigidly bind the model parallelism topology, determined by tensor parallelism (TP) and pipeline parallelism (PP), to launch-time configuration. This static assignment creates a structural impedance mismatch with dynamic, fluctuating inference workloads, as evidenced by distinct traffic patterns in real-world deployments (Figure 1). Existing solutions, lacking online topological reconfiguration, mandate disruptive service restarts to alter TP/PP degrees. Such restarts incur minute-scale downtime, KV cache loss, and redundant recomputation—precluding practical workload-adaptive serving.

Figure 1: Token Usage Profile of INFINIGENCE Model Service, illustrating daily traffic variability and motivating dynamic topology adaptation.

ReMP Architecture and Decoupling Mechanisms

ReMP is conceived to allow online, low-downtime TP/PP reconfiguration by decoupling system runtime state from the fixed topology. It introduces six architectural components: Reconfiguration Controller, Shared Weight Store, MPU State Space, Worker Lifecycle Manager, KV Migration Engine, and Scheduler Adapter (Figure 2).

Figure 2: ReMP architecture, illustrating the decoupling of model weights, KV cache, communication groups, and worker lifetimes from static TP/PP topology.

Key decoupling strategies include:

• Model weights: Persisted in CPU shared memory, enabling GPU shard re-materialization without disk I/O under topology changes (Figure 3).
• KV cache: Migrated via a two-dimensional mechanism—along layer and head dimensions—to preserve request state across TP/PP transitions (Figure 4).
• Communication groups: Parallel-state snapshots preconstructed for candidate topologies, applied instantly during reconfiguration.
• Worker lifetimes: Standby/wakeup model leverages existing processes, minimizing process management overhead.

  Figure 3: Shared model weight store; full model state resident in CPU shared memory, facilitating efficient topology-motivated GPU shard reconstruction.

  

  Figure 4: Two-dimensional KV cache migration process, enabling seamless redistribution of live cache across TP (head) and PP (layer) axes.

Transactional Runtime Switching and State Migration

Each TP/PP topology switch is framed as a controlled transaction, comprising quiescing the scheduler, applying new parallel state, migrating KV cache, loading GPU shards, and resuming execution. Critical optimizations underpin this process:

• Concurrent Model Loading and KV Migration: These disjoint operations are run in parallel, reducing latency from the sequential sum to their maximum duration.
• Layer-wise Streaming: KV migration is conducted layer-by-layer, constraining memory usage to manageable levels and averting OOM risks during migration.
• Worker Management: Workers transition only after their relevant KV slices are properly migrated, ensuring correctness and maintaining low downtime.

Evaluation: Reconfiguration Cost and Serving Performance

ReMP's empirical evaluation spanned models from 7B to 70B parameters and two 8-GPU platforms (H100 and RTX 5090). The core claims substantiated by experimental data are:

• ReMP achieves 1–7 second topology switching—tens to $>$100$\times$ faster than restart-based approaches. (Figure 5, Figure 6)
• For Llama2-70B, ReMP consistently stays within several seconds for most TP/PP transitions, despite substantial model-shard reconstruction requirements.
• Overlapping model-shard reloading and KV cache migration is instrumental in minimizing disruption and mitigating critical path overhead.

Figure 5: H100 platform.

Figure 6: H100 platform.

Figure 7: Serving performance comparison on the H100 platform; ReMP adaptively selects optimal TP/PP under each request rate, outperforming fixed baselines in both latency and throughput.

Adaptive Serving: Throughput and Latency Benefits

Serving experiments reveal that the best TP/PP configuration varies with model architecture and request rate: pipeline-heavy configurations can optimize throughput but suffer increased TTFT and TPOT; tensor-heavy configurations minimize latency but may bottleneck throughput.

• ReMP dynamically selects the superior topology, eclipsing fixed TP1PP8 and TP2PP4 baselines in TTFT, TPOT, and output throughput. This is consistent across all evaluated models and platforms (Figure 8).
• The ability to switch rapidly enables ReMP to avoid the tradeoff inherent in static deployments, maintaining optimal throughput/latency ratios even under variable request pressure.

  Figure 8: Serving performance comparison on the RTX 5090 platform; ReMP consistently achieves lower latency and higher throughput compared to static TP/PP baselines.

Implications and Future Directions

By transforming TP/PP topology from static deployment parameters into runtime-adjustable resources, ReMP enables adaptive LLM serving, with direct gains in latency, throughput, and operational resilience. Practically, this can improve user experience (lower TTFT/TPOT), system efficiency (higher throughput), and cloud cost optimization (dynamic provisioning and resource utilization).

Theoretically, the decoupling mechanisms introduced—shared memory model state, layer/head KV migration, and parallel-state snapshotting—set a precedent for elastic LLM serving and could be extended to heterogeneous cluster environments and fine-grained topology adaptation, including MoE and hybrid parallelism paradigms. Recent works on pipeline reconfuguration [pipelive] and dynamic scheduling [llumnix] will likely integrate similar architectural concepts.

Future research directions include optimal probe-and-select mechanisms for online topology selection, amortized distributed group management for arbitrary topology sets, and advanced KV cache retention schemes for ultra-long context LLM workloads.

Conclusion

ReMP enables second-level, low-downtime TP/PP reconfiguration in production LLM serving systems by decoupling runtime state from topology. Experimental evidence demonstrates substantial speedups over restart-based adaptation and improved serving performance across dynamic workloads. The framework advances adaptive, workload-aware LLM inference, laying the foundation for highly elastic, efficient AI services.

(2606.18741)