LoRAServe: Dynamic Adapter Placement

• LoRAServe is a workload-aware framework that dynamically manages heterogeneous LoRA adapters in distributed large language model inference systems.
• It leverages rank-specific operating points, fractional bin-packing, and GPU Direct RDMA to optimize placement, reduce tail latency, and prevent overprovisioning.
• The framework achieves up to 16× reduction in per-node adapter storage and substantial throughput gains, enabling efficient multi-tenant deployments.

LoRAServe is a workload-aware dynamic adapter placement and routing framework for serving heterogeneous Low-Rank Adaptation (LoRA) adapters in distributed LLM inference systems. It targets the multi-tenant setting in which a single base LLM is shared by hundreds of adapters, and it treats adapter rank heterogeneity as a primary systems variable because higher-rank adapters incur larger memory footprints and higher per-token cost. Its central claim is that cluster-level placement and routing, rather than only kernel-level optimization or local caching, are necessary to prevent performance skew, reduce tail latency, and avoid overprovisioning in real multi-adapter deployments (Jaiswal et al., 28 Nov 2025).

1. Problem setting and scope

LoRAServe is motivated by the standard multi-LoRA serving scenario: a pool of base LLM instances, many LoRA adapters for each base, and incoming requests that specify which adapter to use. In this setting, “adapter heterogeneity” is primarily rank heterogeneity: different tasks use different ranks such as 4, 8, 16, 32, 64, or 128, and higher rank implies larger adapter matrices together with more FLOPs and memory traffic per token. The paper reports that, for Llama-7B, a rank-128 adapter takes approximately $2.7\times$ the prefill time of rank-8 at 2k token prompts; for larger models the penalty from higher rank grows to about 45% TTFT increase relative to low rank; and even with tensor parallelism up to 8, rank-128 is still about 20% slower TTFT than rank-8 (Jaiswal et al., 28 Nov 2025).

The immediate systems consequence is that heterogeneous co-batching is not benign. Under Punica-like or S-LoRA-like kernels, low-rank requests inherit the cost profile of the highest-rank request in the batch. The paper’s microbenchmarks show that co-serving rank-8 and rank-128 on a single Llama-7B instance increases p95 TTFT for rank-8 by 84% compared to a pure rank-8 workload. This converts rank diversity into throughput loss, tail-latency inflation, and SLO violations for otherwise inexpensive adapters (Jaiswal et al., 28 Nov 2025).

The production setting used to motivate LoRAServe is similarly skewed. In traces from “Company X,” popular base models host hundreds of adapters each, across three main base models there are more than 1000 adapters, and popularity is highly concentrated: for one base model, the top 5 adapters account for more than 70% of requests, while the remaining long tail of roughly 1000 adapters each receives less than 1% of traffic. Popularity is also region-specific and drifts over time, with diurnal patterns, increasing or decreasing trends, and sudden surges. This makes both global replication and static placement unattractive (Jaiswal et al., 28 Nov 2025).

The name also has a broader literature-level role. Predictive-LoRA describes itself as a design blueprint for a “LoRAServe”-style system, and FASTLIBRA uses “LoRAServe” as representative of vLLM + S-LoRA-style multi-LoRA engines (Ni et al., 23 Dec 2025, Zhang et al., 19 Apr 2025). This suggests that the term functions both as the name of a specific cluster-level framework and as shorthand for the broader class of systems that share a base model while dynamically serving many LoRA variants.

2. System architecture and operating model

LoRAServe is a cluster-level orchestration layer built on top of S-LoRA. Each server hosts a base LLM instance, often tensor-parallel across GPUs on that node, while the cluster orchestrator continuously observes adapter-level demand, uses pre-profiled rank-specific operating points, computes adapter-to-server placement, updates a routing table, and manages a distributed adapter pool. The system is explicitly workload-aware, rank-aware, and dynamic: placement is recomputed every time step, routing probabilities are updated accordingly, and newly assigned adapters are migrated on first access via GPU Direct RDMA (Jaiswal et al., 28 Nov 2025).

A central input to the control loop is the rank-specific operating point $OP[r]$, defined as the maximum tokens per second that a single LLM server can process for rank $r$ while respecting the SLO. At runtime, the orchestrator estimates per-adapter demand in tokens per second, $\text{demandTPS}[a]$, from recent request history. The framework then maps each request’s adapter_id to one or more candidate servers and routes with probabilities $\phi_i$, so the routing table entries take the form $(\text{adapter\_id}, \text{server\_id}, \phi)$ (Jaiswal et al., 28 Nov 2025).

On each server, LoRAServe does not replace the underlying batch execution model. S-LoRA remains responsible for continuous batching, unified paging of KV cache and adapters between GPU and CPU, and heterogeneous LoRA kernels. LoRAServe instead changes which adapters are colocated on which servers, and therefore which adapters are likely to share a batch. The design premise is that placement can shape batch composition before kernel execution begins, reducing interference without modifying the LoRA kernels themselves (Jaiswal et al., 28 Nov 2025).

The distributed adapter pool is another defining feature. Rather than replicating all adapters everywhere, each server stores only the adapters it actively serves during the current epoch; the union across servers covers all active adapters; and missing adapters are fetched remotely on demand. This turns cluster memory into a shared adapter reservoir and separates logical assignment from physical residence (Jaiswal et al., 28 Nov 2025).

3. Rank-aware placement and routing algorithm

The placement algorithm begins by translating recent adapter demand into rank-specific utilization. For each rank $r$, LoRAServe computes the effective server-equivalents required under the SLO as

$$\text{rankUtil}[r] = \frac{\sum_{a:\,\text{rank}(a)=r}\text{demandTPS}[a]}{OP[r]}.$$

It then sums across ranks,

$$\text{targetUtilTotal} = \sum_r \text{rankUtil}[r],$$

and defines the target average utilization per server over $N$ servers as

$OP[r]$0

From this it derives a server budget for each rank,

$OP[r]$1

The interpretation is direct: if a rank accounts for a fixed fraction of the weighted load, it should receive approximately the same fraction of servers, so similar ranks are grouped rather than arbitrarily mixed (Jaiswal et al., 28 Nov 2025).

Within each rank class, LoRAServe uses a fractional bin-packing step. For an adapter with load $OP[r]$2, the assignment function is

$OP[r]$3

where the $OP[r]$4 are selected servers and the $OP[r]$5 are traffic fractions satisfying $OP[r]$6. This produces the routing table used at request time: the orchestrator looks up all entries for the incoming adapter and forwards the request to server $OP[r]$7 with probability $OP[r]$8 (Jaiswal et al., 28 Nov 2025).

Adapters that are not fully assigned by the rank-budgeted bin packing are handled as leftovers. These are sorted by descending rank and placed on servers that already have the highest maximum rank but currently the least utilization. The heuristic deliberately contains high-rank interference within already high-rank-heavy servers instead of allowing a few large adapters to perturb many low-rank servers (Jaiswal et al., 28 Nov 2025).

To reduce churn, the algorithm applies a permuteAssignment(assignment, prevAssignment) step that adjusts the new placement to minimize deviation from the previous one. In practice this means retaining an adapter on its current server when capacity allows, reducing migration traffic and preserving locality. The resulting policy is neither purely static nor purely myopic: it is periodic, workload-responsive, and explicitly stabilized against unnecessary movement (Jaiswal et al., 28 Nov 2025).

4. Distributed adapter pool and RDMA-based migration

LoRAServe distinguishes logical placement from physical storage through an adapter location table,

$OP[r]$9

If a request is routed to server $r$0 but adapter $r$1 is not in $r$2’s local host memory, the orchestrator consults the location table to find another server $r$3 that currently holds $r$4. Migration then proceeds in three stages: copy $r$5 from host memory to GPU memory on $r$6, transfer it from GPU on $r$7 to GPU on $r$8 with GPU Direct RDMA over InfiniBand, and store $r$9 in host memory on $\text{demandTPS}[a]$0 for future use; the system may optionally delete it from $\text{demandTPS}[a]$1 if the adapter is no longer assigned there (Jaiswal et al., 28 Nov 2025).

The transport assumption is not incidental. The paper reports that fetching a tensor over InfiniBand with GPU Direct RDMA has latency comparable to loading from local host memory to GPU, whereas NVMe or SSD-based storage is much slower in both latency and bandwidth. This is why LoRAServe treats the union of node-local host memories as the fast distributed backing store and avoids local disk in the serving fast path (Jaiswal et al., 28 Nov 2025).

This storage model sharply changes memory economics. Without LoRAServe, systems such as Toppings may replicate all adapters in every server’s CPU memory. With LoRAServe, each server holds only a fraction of the adapters, and the paper reports up to $\text{demandTPS}[a]$2 reduction in per-node adapter storage footprint relative to Toppings. The effect is especially important when the long tail is large: the details give an example in which a 200B-parameter 8-bit model is about 200 GB, LoRA is often about 1% of the base or about 2 GB per adapter, and 500 adapters would amount to roughly 1 TB of adapter parameters, making universal replication infeasible or wasteful (Jaiswal et al., 28 Nov 2025).

The migration mechanism is also central to handling workload drift. As $\text{demandTPS}[a]$3 changes across time steps, hot adapters acquire more replicas, cold adapters are consolidated, and newly popular adapters migrate toward the servers to which routing probabilities now point. In this respect, LoRAServe treats remote adapter access not as an exception but as a first-class cluster operation (Jaiswal et al., 28 Nov 2025).

5. Position within the LoRA-serving literature

LoRAServe belongs to a rapidly expanding systems literature on multi-LoRA inference, but it occupies a distinct layer of the design space. S-LoRA establishes the per-node substrate with Unified Paging, heterogeneous LoRA kernels, and a tensor-parallel strategy for serving thousands of adapters from CPU memory (Sheng et al., 2023). CaraServe addresses the cold-start path by early-starting activated adapters on CPUs for prefilling while they are being loaded onto GPUs, together with a rank-aware scheduling algorithm (Li et al., 2024). Predictive-LoRA adds an LSTM-based traffic predictor and a page-based adapter memory manager for serverless environments (Ni et al., 23 Dec 2025). FASTLIBRA unifies LoRA and KV caching with a dependency-aware cache manager and a performance-driven cache swapper (Zhang et al., 19 Apr 2025). InfiniLoRA takes a different step entirely by decoupling LoRA execution from base-model inference through a shared LoRA Server (Chen et al., 8 Apr 2026).

System	Primary mechanism	Scope
S-LoRA (Sheng et al., 2023)	Unified Paging; MBGMM/MBGMV; tensor parallelism	Per-node multi-LoRA runtime
CaraServe (Li et al., 2024)	CPU-assisted prefill; rank-aware scheduling	Cold-start mitigation within servers
Predictive-LoRA (Ni et al., 23 Dec 2025)	LSTM prediction; proactive prefetch; page-based GPU memory	Serverless multi-LoRA serving
FASTLIBRA (Zhang et al., 19 Apr 2025)	Unified LoRA+KV cache; dependency-aware manager	TTFT-oriented cache orchestration
LoRAServe (Jaiswal et al., 28 Nov 2025)	Dynamic adapter placement and routing; GPU Direct RDMA	Cluster-level rank-aware orchestration
InfiniLoRA (Chen et al., 8 Apr 2026)	Shared LoRA Server; GPU-initiated communication	Disaggregated LoRA execution

The paper’s own comparison is especially pointed against Toppings. Toppings introduces CPU-assisted LoRA compute for prefill and a rank-aware request scheduler, but it assumes that all adapters are replicated on each server’s CPU memory. LoRAServe argues that this is not viable when there are hundreds or thousands of adapters, and that request-level rank awareness is insufficient if placement still exposes all servers to all ranks (Jaiswal et al., 28 Nov 2025).

InfiniLoRA later makes the distinction explicit: it states that LoRAServe “optimizes LoRA placement among LLM instance clusters by dynamically rebalancing adapters across GPUs,” but treats this as fundamentally different from disaggregation, where LoRA is removed from the LLM instances entirely and executed as a shared service (Chen et al., 8 Apr 2026). The contrast clarifies LoRAServe’s place in the lineage: it is not a serverless design, not a unified LoRA+KV cache manager, and not a disaggregated LoRA server, but a cluster scheduler for coupled multi-LoRA engines.

6. Evaluation, limitations, and significance

LoRAServe is evaluated on both production traces from Company X and synthetic traces derived from the Azure public dataset. The main production setting uses a 4-node cluster with 50, 100, or 200 adapters, defaulting to Llama-7B with tensor parallelism 4 and Poisson arrivals unless otherwise noted. Under a fixed TTFT SLO, LoRAServe can serve up to 20% more requests than Toppings, and relative to S-LoRA Random or S-LoRA Contiguous it reaches up to $\text{demandTPS}[a]$4 throughput. Across the evaluation, the headline gains are up to $\text{demandTPS}[a]$5 higher throughput, up to $\text{demandTPS}[a]$6 lower TTFT, up to 50% fewer GPUs to meet SLOs, and up to $\text{demandTPS}[a]$7 reduction in per-node adapter storage footprint versus Toppings (Jaiswal et al., 28 Nov 2025).

The behavior under synthetic workload drift is equally central. Across six combinations of uniform versus Poisson arrivals and uniform versus shifting-skew versus exponential rank popularity, LoRAServe consistently outperforms the baselines. S-LoRA Contiguous performs acceptably under uniform popularity but fails under skew, while S-LoRA Random is moderately robust yet can fail when skew changes over time. By contrast, LoRAServe tracks shifting popularity and maintains lower TTFT because it jointly models rank and demand rather than relying on static allocation (Jaiswal et al., 28 Nov 2025).

Scalability results show near-linear weak scaling: with 4 servers the system can handle about 32 RPS within the p95 TTFT SLO, with 8 servers about 64 RPS, and with 12 servers about 96 RPS. The reported interpretation is that cluster orchestrator overhead and RDMA costs do not introduce noticeable bottlenecks. Sensitivity studies also show that the relative benefits persist for Llama-7B, 30B, and 70B, and across different tensor-parallel settings; the larger the model, the more pronounced the cost of high ranks and thus the more important rank-aware placement becomes (Jaiswal et al., 28 Nov 2025).

The framework’s assumptions are correspondingly clear. It presumes high-speed InfiniBand with GPU Direct RDMA, a deployment model centered on one base LLM instance per server, periodic rather than continuous rebalancing, and rank together with tokens-per-second as the primary cost features. Other per-adapter characteristics, such as prompt-length distributions or KV-cache behavior, are not explicitly modeled. Very abrupt shifts can therefore cause temporary misplacement until the next time step, and the design does not directly address multi-base colocated clusters or non-RDMA environments (Jaiswal et al., 28 Nov 2025).

Within the broader research trajectory, LoRAServe’s lasting significance is that it reframes multi-LoRA serving from a purely kernel or local-memory problem into a cluster placement problem. Later systems continue to optimize page allocators, dependency-aware caches, serverless pre-loading, or full disaggregation, but LoRAServe isolates a separate systems fact: when adapter rank is heterogeneous and popularity drifts, batch composition is governed as much by cluster routing and adapter placement as by the efficiency of the kernels that eventually execute the LoRA update (Jaiswal et al., 28 Nov 2025).