SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference

Abstract: The rapid adoption of LLMs has shifted a substantial portion of inference workloads into throughput-oriented offline regimes, where fully utilizing GPU compute requires large batch sizes. However, existing deployments face a structural tension. Data parallelism (DP) scales throughput well but replicates model weights, leaving limited GPU memory for key-value (KV) cache and constraining batch size. Model parallelism reduces per-device weights, but requires fine-grained synchronization that erodes DP's independence and scheduling flexibility. We present SiDP, a memory-efficient data-parallel paradigm for offline LLM inference that treats weights as a bandwidth-backed shared resource inside a DP group. Instead of storing the full model on every GPU, SiDP organizes weights as a distributed pool: each layer is owned by a single GPU, and other replicas access its weights on demand via two complementary execution modes: a Weight-as-a-Service (WaS) mode that streams remote weights over NVLink into a small cache in the large-batch regime, and a Compute-as-a-Service (CaS) mode that ships activations to owners in the small-batch tail. Evaluated on NVIDIA H20, H200, and B200 GPUs with Qwen3-32B, Qwen2.5-72B, and Llama-3.1-70B, SiDP increases usable KV capacity by up to 1.8x under the same configurations, and converts this into up to 1.5x higher end-to-end throughput over baselines (vLLM) for offline workloads.

• The paper introduces Albireo, a novel SiDP approach that improves memory efficiency by distributing FFN weights across GPUs.
• It leverages dual-mode execution—WaS for large batches and CaS for tail jobs—to optimize throughput and reduce NVLink bottlenecks.
• Empirical results show up to 1.8× KV cache expansion and 1.5× throughput gains, validating the paradigm for offline LLM inference.

SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference

Motivation and Problem Definition

The escalating adoption of LLMs has led to a bifurcation in inference workloads: latency-critical online services and throughput-oriented offline jobs. The latter—encompassing large-scale dataset processing, evaluation, synthetic data generation, and RL rollouts—predominantly seek maximal aggregate throughput, with batch size serving as the primary lever. In these settings, increasing batch size improves arithmetic intensity and kernel utilization, yielding significant throughput gains up to a hardware-dependent threshold ($B_e$), after which marginal utility stagnates. However, memory constraints—chiefly from model weights (W) and key-value (KV) cache for active tokens—become the dominant bottleneck, especially as batch sizes and sequence lengths grow. Data parallelism (DP), while optimal for scaling throughput, incurs redundant weight replication across devices, severely limiting available memory for KV cache and constraining $B$. Conversely, model parallelism (MP) shards the model to reduce per-device weight footprint but mandates step synchronization, reducing scheduling flexibility and throughput.

Albireo: Architectural Principles and Algorithms

Albireo, the presented SiDP solution, directly addresses this tension. It organizes model weights within a DP group into a distributed pool, assigning each layer to a specific GPU and allowing peers to access weights via two execution paradigms:

1. Weight-as-a-Service (WaS): In large-batch regimes, non-owner GPUs asynchronously fetch required FFN weights from the owner's HBM over NVLink, caching them locally. Local computation is preserved, and remote weight fetches are overlapped with compute, minimizing contention. Decentralized, staggered prefetching (peak shifting) avoids NVLink hotspots, ensuring bandwidth-efficient operation.
2. Compute-as-a-Service (CaS): In the long-tail, small-batch regime, activations are shipped to the weight-owning GPU, which performs the GEMM and returns results. Batch fusion, dummy skipping, and orchestrator-controlled mode switching optimize bandwidth and tail job completion.

Initialization parses the model, assigns layer ownership, exports remote buffer handles, and wraps FFN modules transparently. Attention weights remain locally replicated due to their smaller parameter footprint and locality constraints. All weight transfers use CUDA IPC and operate on dedicated streams via asynchronous events, allowing seamless overlap with primary computation.

Figure 1: Overview of the Albireo architecture with layer ownership, asynchronous prefetching, and dual WaS/CaS modes.

Figure 2: Memory layout and per-layer workflows for WaS and CaS in Albireo, illustrating DP group interaction and buffer management.

Empirical Evaluation: Throughput and Memory Efficiency

Albireo was benchmarked on NVIDIA H20, H200, and B200 nodes with Qwen3-32B, Qwen2.5-72B, and Llama-3.1-70B under offline workloads with large sequences. The most salient results include:

• Increased KV Cache Capacity: Albireo enlarges the available KV cache by up to $1.8\times$ under identical DP/TP configurations. For large DP settings (e.g., TP=1, DP=8), the baseline vLLM fails to accommodate Llama-3.1-70B or Qwen2.5-72B, whereas Albireo provides $\sim$1$$M KV tokens. <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2605-28095/results_kv_tokens_sidp_vs_vllm_3subplots.png" alt="Figure 3" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 3: Albireo enables substantially greater KV cache capacity compared to vLLM, with improvements up to$$1.8\times$$under memory constraints.</p></li> <li><strong>End-to-End Throughput Gains:</strong> Albireo converts expanded KV capacity into up to$$1.5\times$$higher throughput than strong TP+DP baselines. Gains scale with batch size, sequence length, and DP degree; the advantage is most pronounced for memory-bound, long-context workloads. <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2605-28095/results_tpdp_sidp_H20_vs_seqlen.png" alt="Figure 4" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 4: Throughput improvement with Albireo across parallelism configurations and sequence lengths on H20.</p> <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2605-28095/results_tpdp_sidp_H200_vs_seqlen.png" alt="Figure 5" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 5: Throughput scaling behavior of Albireo on H200, demonstrating consistent gains.</p> <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2605-28095/results_tpdp_sidp_B200_vs_seqlen.png" alt="Figure 6" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 6: Throughput scaling with Albireo on B200, supporting large batch sizes and sequence lengths.</p></li> <li><strong>Mode Adaptivity and Design Validation:</strong> Peak shifting reduces NVLink incast bottlenecks, delivering up to$$3.4\times$$throughput gains at DP=8 compared to naive prefetching. WaS dominates for large batches, but CaS substantially improves tail iteration latency; orchestrator-driven mode-switching further amplifies end-to-end throughput ($$27\%$–$32\%$$) compared to WaS alone. <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2605-28095/ablation_gpu_iter_fetch_time.png" alt="Figure 7" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 7: Prefetch latency vs. iteration time, confirming the feasibility of overlapped weight transfer in WaS mode.</p> <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2605-28095/ablation_dp_shift_vs_dp.png" alt="Figure 8" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 8: Impact of peak shifting on throughput as DP degree increases.</p> <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2605-28095/ablation_iter_time_vs_batch.png" alt="Figure 9" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 9: Iteration time breakdown across WaS, CaS, Albireo, and vLLM as batch size varies, demonstrating adaptive efficiency.</p></li> <li><strong>Robustness and Short-Context Applicability:</strong> Albireo maintains benefits for short-context settings and lower memory budgets, providing$$B$0–$B$$1 throughput improvement for mainstream LLMs. <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2605-28095/ablation_two_model_seq_tp_singlecol.png" alt="Figure 10" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 10: Throughput gains with Albireo under reduced GPU memory utilization, confirming effectiveness in short-context scenarios.</p></li> <li><strong>Ablation Analysis:</strong> CaS optimizations (P2P, GEMM fusion, dummy skipping) collectively deliver$$B$2 speedup in extreme small-batch regimes, overcoming the limitations of FSDP-like sharding in inference.

  Figure 11: Throughput gain with and without mode switching, highlighting the critical contribution of CaS.

  

  Figure 12: End-to-end single-request completion time under different implementations for tail workloads.

  

  Figure 13: Distribution of batch sizes and mode switches in Albireo, showing WaS suffices for bulk iterations, with CaS invoked only briefly for tail-completion.

Practical and Theoretical Implications

Albireo fundamentally changes the scaling dynamics of offline LLM inference. By transforming FFN weights into a bandwidth-backed, shared resource within DP groups, it eliminates redundant memory consumption while retaining the independence and scalability of DP. This translates directly into larger effective batch sizes, higher throughput, and extended feasible sequence lengths for fixed hardware—without requiring global synchronization or sacrificing scheduler flexibility.

The design leverages the NVLink interconnect, making it suited primarily for intra-node deployments. Extending Albireo to multi-node or cross-cluster environments, or adapting for slower interconnects, will require further research into communication scheduling and remote weight sharing.

Theoretically, the architectural strategy demonstrates that memory bottlenecks in LLM inference are not solely a function of parameter count, but also of data-parallel weight replication. By sharding only FFN weights—responsible for the dominant parameter fraction—Albireo provides a clean path to higher throughput without increasing communication granularity beyond what is overlappable with compute.

Further generalization to mixture-of-experts models and integration with expert-parallel load balancing are promising avenues. Albireo can enable greater elasticity and memory efficiency in expert-parallel MoE deployments, reducing redundant expert replication within DP groups. The impact on scheduling, workload co-design, and robustness in heterogeneous environments remains a subject for future exploration.

Conclusion

SiDP, instantiated by Albireo, establishes a new memory-efficient data parallelism paradigm for offline LLM inference. The dual WaS/CaS execution modes, distributed weight pool layout, and asynchronous prefetch protocol collectively yield up to $B$3 higher KV cache capacity and $B$4 higher throughput than state-of-the-art baselines—robustly across multiple LLM architectures and hardware generations. The implications extend to both theoretical scalability limits and practical deployment scenarios for large-scale, throughput-oriented LLM applications.