Punica: Multi-Tenant LoRA Serving

Updated 30 March 2026

Punica is a scalable multi-tenant architecture that serves thousands of LoRA adapters by decoupling base model weights from tenant-specific LoRA matrices.
It employs a segmented gather matrix-vector kernel to batch heterogeneous adapter computations, optimizing GPU memory utilization and throughput.
Advanced scheduling, compression techniques, and unified memory management ensure low latency and fair resource allocation across diverse tenant workloads.

Low-rank adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of LLMs, resulting in the proliferation of thousands of fine-tuned LoRA adapters derived from a common base model. Serving these adapters in a multi-tenant environment at scale presents unique systems and algorithmic challenges, including memory management, latency minimization, throughput maximization, and fairness amid adapter heterogeneity. Punica is a system architecture specifically designed to address these issues, enabling practical, high-throughput serving of thousands of LoRA models in GPU-backed inference clusters (Chen et al., 2023).

1. Architectural Foundations of Punica

Punica is structured as a cluster-level orchestrator mediating between tenant-facing frontend APIs and a pool of GPU-backed inference servers. The central architectural innovation lies in its separation of the base model weights from tenant-specific LoRA adapters. Each GPU process hosts a single backbone LLM copy and dynamically loads LoRA matrices $(A_i, B_i)$ for each request. This design allows a single GPU to serve arbitrarily many adapters without maintaining separate model copies per tenant.

Core System Components

Frontend Servers: Handle (model_id, prompt) requests, streaming tokens back via REST/gRPC.
Central Scheduler: Tracks per-GPU “working sets,” manages KV-cache allocation, assigns or migrates requests, and triggers scale-up/scale-down actions.
GPU Runners: Each process loads the shared base model, manages LoRA/KV paging in GPU memory, and batches inference using fused custom CUDA kernels.

To maximize arithmetic intensity and reduce redundant memory transfers, requests are sorted by adapter identity before batching and partitioned into segments per unique adapter. Punica’s Segmented Gather Matrix-Vector (SGMV) kernel then applies all requisite LoRA addons in a single pair of kernel launches, fusing the operations for maximum efficiency (Chen et al., 2023).

2. Multi-Model Batching and CUDA Kernel Design

The core kernel is the SGMV, which breaks the $xA_iB_i$ operation for each adapter into sequential “shrink” (project input to rank- $r$ ) and “expand” (project back to output dimension) stages. This enables Punica to batch heterogeneous LoRA computations, maximizing hardware utilization even in non-uniform tenant workloads.

// segment_idx = adapter batch boundaries
// A_ptrs, B_ptrs = pointers to each adapter’s matrices
kernel_SGMV_shrink(x, segment_idx, A_ptrs, out v);
kernel_SGMV_expand(v, B_ptrs, out y);

All $A_i, B_i$ matrices are staged in flat device buffers addressable by per-batch pointer arrays. See [(Chen et al., 2023), Fig. 3] for layout details.

3. Memory Management, Paging, and Rank Heterogeneity

Punica leverages demand-paging for LoRA adapter weights: adapters are paged in and out of GPU memory as needed, using a unified memory pool that also serves the KV cache. Designs such as S-LoRA and Predictive-LoRA (P-LoRA) formalize this with page-based allocators that minimize fragmentation, support heterogeneous adapter ranks, and enable prefetch- or eviction-driven policies to optimize both time-to-first-token (TTFT) and overall GPU memory utilization (Sheng et al., 2023, Ni et al., 23 Dec 2025).

A critical challenge addressed is the performance skew induced by batching adapters of diverse rank in the same kernel invocation. Punica’s scheduler, informed by performance models, ensures that each GPU is assigned adapters of compatible rank and distributes load to avoid tail-latency blowups (Jaiswal et al., 28 Nov 2025).

Dependent Caching and Unified Pools

Extensions such as FASTLIBRA further unify LoRA and KV cache management by constructing dependency-aware cache graphs (trie-based) and optimizing swap-in/swap-out via cost models that combine recent access patterns and predicted future demand (Zhang et al., 19 Apr 2025).

4. High-Throughput Scheduling and Placement

Punica’s cluster-wide scheduler embodies a fractional assignment optimization: adapter traffic is assigned to GPU servers so as to maximize throughput minus tail-latency, subject to hard GPU memory and compute constraints, while mitigating the impact of rank-induced heterogeneity. The routing plan specifies, for each $(adapter, server)$ tuple, the fraction of requests to send, enabling both load balancing and hot-adapter replication across machines.

Key constraints:

Per-GPU memory and compute limits ( $\sum_{a:x_{a,g}>0} size(a) \le M_g, \sum_{a} D_a\cdot x_{a,g} \le C_g$ ).
Optimal placement minimizes maximum predicted P95 latency $L_{a,g}(x)$ .
RDMA-based just-in-time paging avoids full replication of all adapters and enables rapid migration (Jaiswal et al., 28 Nov 2025).

5. Compression and Communication-Optimized Serving

When the number of LoRA adapters grows to the thousands, paging alone is insufficient—adapter state cannot fully reside on GPU. Punica incorporates joint compression methods to factorize the collection ${\Delta W_i}$ into a shared low-rank basis $B$ and per-adapter scaling matrices $S_i$ (JD-Full/JD-Diag), with optional clustering for enhanced compression fidelity. In serving, only the compact scaling matrices must be loaded per request, slashing per-adapter memory and PCIe I/O. E.g., with $N=1000$ , $d=4096$ , $r=16$ , and $C=8$ clusters, the compressed adapter footprint is $<1$ % of the uncompressed equivalent, with $<1\%$ downstream loss and $75$–$90$% of single-LoRA throughput maintained (Brüel-Gabrielsson et al., 2024).

Punica also admits specialized LoRA architectures designed to eliminate inter-device communication overhead in multi-GPU/tensor-parallel settings. Block-Diagonal LoRA (BD-LoRA) matches the sharding of LoRA weights to the tensor-parallel partitioning of the base model, permitting fully local computation and avoiding the all-gather/all-reduce collectives customary in standard LoRA batching. This results in strictly higher throughput over e.g., S-LoRA, with negligible expressiveness loss (Wang et al., 27 Oct 2025).

6. Performance Evaluation and Empirical Results

Punica achieves substantial improvements over existing systems:

Throughput: 12× improvement in aggregate throughput (e.g., 1k tok/s for 7B models), compared to vLLM/PEFT with naive LoRA support (Chen et al., 2023, Sheng et al., 2023).
Overhead: Additional per-token latency is $\leq 2$ ms, mainly from PCIe copy; overlapped over tens of ms decode steps.
Adapter scalability: Batching and unified paging support $>1000$ concurrent adapters in GPU DRAM; compressed serving (JD-Diag/JD-Full) enables $10^3$ – $10^4$ adapters with 75–90% original throughput and sub-1% accuracy loss (Brüel-Gabrielsson et al., 2024).
Rank-aware fairness: SLO attainment rates up to 99%, with tail TTFT reduced by up to 9× and GPU count cut in half relative to baseline batchers (Jaiswal et al., 28 Nov 2025, Li et al., 2024).
Serverless environments: Integration of proactive adapter prefetching, page-based memory managers, and combined LRU/frequency/predicted-demand eviction policies yield 1.5× higher throughput and 35% lower median TTFT in high-concurrency regimes (Ni et al., 23 Dec 2025).
Edge and distributed deployments: Innovations in adaptive adapter routing, heterogeneous memory pools, and batch LoRA inference extend scalability to resource-constrained scenarios, achieving 2–4× throughput gains over baseline edge-serving stacks (Shen et al., 2 Jul 2025).

7. Limitations, Extensions, and Future Directions

Punica’s modular design admits multiple avenues for further specialization. Extensions to other parameter-efficient fine-tuning methods (e.g., prefix-tuning, prompt-tuning) require addition of new cost and latency models in the placement engine. Enhanced memory management—e.g., anticipatory replication, joint KV and adapter paging, and QoS-aware caching—offers room for further improvement in SLO satisfaction and memory elasticity, especially as adapter heterogeneity increases (Jaiswal et al., 28 Nov 2025, Zhang et al., 19 Apr 2025).

Deployments in multi-tenant retrieval settings demonstrate the broader applicability of the LoRA serving paradigm: query-side LoRA adaptation enables index-preserving personalization for thousands of tenants, maintaining isolation and negligible per-tenant overhead in IR contexts (Jain et al., 8 Jan 2026).

A plausible implication is that as LoRA usage broadens, Punica-style architectures equipped with compression-aware, fragmentation-robust, and communication-minimizing primitives will emerge as the standard for multi-tenant, high-density LLM adaptation serving.

References

(Chen et al., 2023) "Punica: Multi-Tenant LoRA Serving"
(Sheng et al., 2023) "S-LoRA: Serving Thousands of Concurrent LoRA Adapters"
(Brüel-Gabrielsson et al., 2024) "Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead"
(Wang et al., 27 Oct 2025) "Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving"
(Jaiswal et al., 28 Nov 2025) "Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems"
(Li et al., 2024) "CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference"
(Ni et al., 23 Dec 2025) "Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs"
(Zhang et al., 19 Apr 2025) "Improving the Serving Performance of Multi-LoRA LLMs via Efficient LoRA and KV Cache Management"
(Shen et al., 2 Jul 2025) "EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices"
(Jain et al., 8 Jan 2026) "Succeeding at Scale: Automated Multi-Retriever Fusion and Query-Side Adaptation for Multi-Tenant Search"