Serving Chain-structured Jobs with Large Memory Footprints with Application to Large Foundation Model Serving

Abstract: As a current trend in AI, large foundation models are increasingly employed as the core of AI services. However, even after training, serving such models at scale remains a challenging task due to their heavy resource footprints, particularly in terms of GPU memory. While recent works revealed unique characteristics of systems serving foundation models that distinguish them from traditional distributed computing systems, there is still a lack of fundamental understanding of the underlying system management problems. This work aims at addressing this gap by extracting a novel problem of "server chain composition" via block placement and cache allocation for serving chainstructured jobs with large memory footprints, which models a fundamental problem in serving large foundation models through pipeline parallelism. After showing the NP-hardness of the optimal solution, the focus is turned to developing scalable algorithms with guaranteed performance under state-of-the-art load balancing. Application of the proposed solution to a distributed LLM serving system shows significant reduction of response times compared to state-of-the-art solutions.

• The paper presents the GBP-CR approach that optimally assigns model blocks to servers with reserved cache to mitigate severe memory constraints.
• It integrates Greedy Cache Allocation (GCA) and the JFFC load balancing policy to effectively reduce mean response times under high concurrency.
• Empirical evaluations show up to 77.8% reduction in 95th percentile response times, validating the method’s efficiency in real distributed GPU settings.

Serving Chain-Structured Jobs with Large Memory Footprints for Foundation Model Serving

Introduction and Motivation

The proliferation of large foundation models, including LLMs and multimodal architectures, has induced substantial system-level challenges in inference serving due to extreme GPU memory demands. The prevalent paradigm of pipeline parallelism, which partitions models at layer boundaries across distributed servers, transforms inference requests into chain-structured jobs that traverse a sequence of physical hosts. Unlike traditional compute-bound workloads, transformer-based models shift the resource bottleneck to memory—both for storing model parameters and per-request intermediate values (e.g., KV cache)—posing unique compositional and allocation problems unseen in legacy server management.

This paper formulates the fundamental "server chain composition" problem for memory-bound chain-structured jobs in distributed model serving. By abstracting models as sequences of blocks and requests as jobs requiring block traversal, it systematically tackles (i) block placement across servers, (ii) cache space allocation for job concurrency, and (iii) dynamic job dispatching. The NP-hardness of the joint optimization is established, motivating scalable algorithmic solutions with guaranteed performance bounds under advanced load balancing frameworks.

Problem Formulation: Chain Composition and Resource Contention

The model is formalized as a set of $J$ servers, each with memory constraint $M_j$, computation time $\tau^p_j$, and communication time $\tau^c_j$. The service comprises $L$ blocks (layers), each of footprint $s_m$, and per-job cache requirements $s_c$ per block. Inference serving requires jobs to be assigned to chains that collectively host all blocks in sequence, while reserving cache allocation for each job traversing the chain.

Block placement is encoded as contiguous block intervals per server, with chain feasibility determined by adjacency rules. Memory contention arises since placement consumes static memory ($s_m$ per block), leaving residual for dynamic cache allocation ($s_c$ per job per block). The block placement and cache allocation are jointly constrained to ensure the composability of server chains for concurrent job execution.

This resource allocation is intertwined with online load balancing: given a composed set of job servers (server chains with assigned capacities and service rates), inference requests must be dispatched efficiently to minimize mean response time under a central queue, while ensuring system stability for arrival rate $\lambda$.

Algorithmic Solution: Block Placement, Cache Allocation, and Load Balancing

Block Placement under Cache Reservation

The NP-hardness of the block placement and cache allocation problem motivates tractable surrogate formulations. The paper proposes Greedy Block Placement with Cache Reservation (GBP-CR), in which blocks are sequentially assigned to servers, sorted by amortized per-block service time, and cache slots are reserved per placed block for concurrency parameter $M_j$0. GBP-CR optimally composes chains in homogeneous memory configurations and provides near-optimal solutions otherwise. Parameter $M_j$1 mediates the tradeoff between chain length (service time) and concurrency (waiting time).

Figure 1: Impact of cache capacity requirement $M_j$2 on chain composition and server concurrency for $M_j$3 servers and block set $M_j$4.

Cache Allocation and Chain Composition

Post-placement, residual memory permits further concurrency optimization by constructing additional chains or augmenting existing ones via Greedy Cache Allocation (GCA). GCA iteratively identifies the fastest chains, allocates maximal feasible job capacity, and updates server memory accordingly. This method restricts the utilized chains to a polynomial subset compatible with advanced load balancing policies (Join-the-Fastest-Free-Server), ensuring both scalability and optimality in chain selection.

Figure 2: Optimization opportunity: additional concurrency achievable beyond reserved chain allocation after block placement.

Load Balancing: Join-the-Fastest-Free-Chain Policy

For online dispatch, the adapted Join-the-Fastest-Free-Chain (JFFC) policy is employed, prioritizing assignment to the fastest chain with available capacity and falling back to central queueing. Steady-state analysis under Poisson arrivals and exponential service times yields tight upper and lower bounds on system occupancy and response time. These bounds directly feedback into the selection of the cache reservation parameter $M_j$5 for offline placement, optimizing the end-to-end policy.

Figure 3: Mean response time comparison for JFFC versus JSQ, JIQ, and advanced policies under simulated heterogeneous servers.

Performance Evaluation: Simulation and Real-System Validation

Model-Driven Simulations

Under synthetic workloads matching large LLM serving constraints (e.g., BLOOM-176B layer counts, realistic GPU types), the proposed pipeline—GBP-CR + GCA + JFFC—consistently outperforms PETALS and state-of-the-art solutions (BPRR) in mean response time, particularly under scarce server and high-arrival regimes. Parameter $M_j$6 exhibits non-monotonic impact, with optimal values increasing with arrival intensity, validating the need for analytical tuning.

Figure 4: Optimal cache reservation parameter $M_j$7 as a function of arrival rate, reflecting the interplay between service and waiting times.

PETALS-Based Experiments

The algorithmic stack was integrated with PETALS on an emulated deployment (A100 Multi-Instance GPU slices) and evaluated using Azure LLM trace data. Despite deviations from theoretical service and arrival time assumptions (burstier arrivals, less variable service), the system registered robust improvements: up to 77.8% reduction in 95th percentile response times and over 97% reduction in waiting times compared to PETALS heuristics. Analysis confirmed that improved memory utilization (even for small models) substantially increased job parallelism per fast server, reducing both queueing and service delays.

Figure 5: Median and P95 response time improvement across algorithms for LLaMA-2-7B on emulated distributed GPU clusters.

Figure 6: Communication time as a function of output length, matching modeled autoregressive communication delays.

Implications and Future Directions

This work establishes the necessity of explicit chain composition and cache allocation in distributed foundation model serving. By formally disentangling the roles of block placement and memory concurrency, and integrating analytical bounds into orchestration, the approach enables response time reduction and resource efficiency that extend beyond heuristic system implementations.

Practically, server providers can tailor memory allocation and concurrency strategies to match dynamic workloads, adjusting chain composition in response to time-varying demand or heterogeneous infrastructure. Theoretically, similar formalizations can be extended to tensor-parallel or hybrid inference serving, as well as adaptive scheduling under complex job and memory dynamics.

Future developments may include:

• Adaptive algorithms for online reconfiguration under varying arrival rates and model workloads.
• Integration with advanced scheduling (e.g., non-FCFS policies) to maximize throughput and fairness.
• Exploration of block placement under hardware heterogeneity and cross-datacenter networking constraints.

Conclusion

The paper rigorously addresses resource allocation for chain-structured, memory-bound inference jobs, fundamental for scalable large foundation model serving. The combination of tractable algorithms (GBP-CR, GCA), theoretical bounds, and empirical validation demonstrates significant gains over prior art. This establishes a principled foundation for serving transformer-based models on distributed infrastructure, with broad applicability to emerging AI deployment architectures.