Block-Diagonal LoRA for Efficient LLM Serving

• Block-Diagonal LoRA is a structured variant of Low-Rank Adaptation that constrains key adapter factors to block-diagonal form to align with tensor-parallel sharding.
• It eliminates extra inter-device communication by partitioning one LoRA factor into shard-local blocks while maintaining comparable downstream performance to standard LoRA.
• Empirical evaluations show BD-LoRA achieves similar accuracy with improved runtime efficiency, offering up to 1.78x speed-up in decoding under multi-adapter scenarios.

Block-Diagonal LoRA (BD-LoRA) is a structured variant of Low-Rank Adaptation in which selected LoRA factors are constrained to be block-diagonal so that adapter computation aligns with the tensor-parallel shard structure of a base LLM during multi-adapter online serving. In the formulation introduced for this setting, the method targets the case of one frozen base LLM and many LoRA adapters served concurrently, where adapters cannot simply be merged into the base weights because adapter swapping would create overhead and requests using different adapters could not be batched. Its defining systems property is that, under tensor parallelism, BD-LoRA adds no additional communication for LoRA computation beyond the base model’s own collectives, while achieving downstream performance similar to standard LoRA at a similar parameter budget (Wang et al., 27 Oct 2025).

1. Problem setting and design objective

BD-LoRA was introduced for the serving regime in which a single base LLM is shared across several different LoRA adapters and executed with tensor parallelism across multiple GPUs. In that regime, the serving system computes the adapter path on the fly as $XW + XAB$, rather than permanently merging $W+\Delta W$ into the base model. The motivation is operational rather than purely algebraic: if requests switch between adapters, repeatedly subtracting the old adapter contribution and adding the new one creates overhead, and if adapters are merged into weights then batched inference works naturally only when all requests in a batch use the same adapter (Wang et al., 27 Oct 2025).

The immediate precursor discussed in the same context is S-LoRA, which shards LoRA adapters in a way compatible with Megatron-LM tensor parallelism but still incurs blocking communication. For one attention module, the reported extra communication is

$\frac{5(N-1)rS}{N},$

where $N$ is the number of devices, $r$ the LoRA rank, and $S$ the number of tokens, whereas the base model communication is

$\frac{2(N-1)d_HS}{N}.$

Although this appears small when $r \ll d_H$, the paper emphasizes that larger rsLoRA ranks are useful and that communication startup latency is ignored by bandwidth-only estimates. BD-LoRA addresses this by constraining certain LoRA factors to be block-diagonal so that the adapter path follows the base model’s tensor-parallel dataflow and requires no extra inter-device communication (Wang et al., 27 Oct 2025).

2. Algebraic construction

In the paper’s row-major notation, a standard LoRA-adapted linear layer with base weight

$W \in \mathbb{R}^{d_{in}\times d_{out}}$

uses

$W' = W + \Delta W, \qquad \Delta W = AB,$

with

$W+\Delta W$0

and forward pass

$W+\Delta W$1

During training, the implementation uses rsLoRA scaling

$W+\Delta W$2

BD-LoRA does not retain fully dense LoRA factors. Instead, it constrains one factor in each LoRA pair to be block-diagonal while the other factor remains dense but is sharded compatibly (Wang et al., 27 Oct 2025).

The underlying locality principle is the standard block-diagonal matrix identity. If

$W+\Delta W$3

with

$W+\Delta W$4

and

$W+\Delta W$5

then

$W+\Delta W$6

so each device computes its shard independently with no communication. BD-LoRA engineers the adapter factors so that the LoRA path inherits the same property (Wang et al., 27 Oct 2025).

For a Megatron-style MLP with

$W+\Delta W$7

and forward pass $W+\Delta W$8, the first projection $W+\Delta W$9 is column-parallel and the second projection $\frac{5(N-1)rS}{N},$0 is row-parallel. BD-LoRA uses the following asymmetric construction.

For the first projection,

$\frac{5(N-1)rS}{N},$1

and

$\frac{5(N-1)rS}{N},$2

Then

$\frac{5(N-1)rS}{N},$3

which matches the column-sharded layout of $\frac{5(N-1)rS}{N},$4 (Wang et al., 27 Oct 2025).

For the second projection,

$\frac{5(N-1)rS}{N},$5

and

$\frac{5(N-1)rS}{N},$6

Because the intermediate activation is already sharded over the $\frac{5(N-1)rS}{N},$7 dimension, each device computes $\frac{5(N-1)rS}{N},$8 locally, and the resulting shard layout matches the base row-parallel $\frac{5(N-1)rS}{N},$9 path (Wang et al., 27 Oct 2025).

3. Sharding behavior and communication elimination

The central systems claim of BD-LoRA is not merely reduced communication but zero additional communication for the LoRA computation under tensor parallelism. In S-LoRA, communication arises because partial rank-space results must be assembled across devices. For the first MLP projection, S-LoRA shards both $N$0 and $N$1 column-wise; after each device computes its local $N$2, the full rank dimension is needed before multiplication by $N$3, so an all-gather is required. For the second projection, S-LoRA shards $N$4 row-wise and $N$5 column-wise; after multiplication by $N$6, devices hold partial contributions over the shared rank dimension and must combine them via all-reduce. For a basic MLP, this yields two additional communication operations on top of the base model, one all-gather and one all-reduce. For GLU MLP variants, the paper reports two all-gathers and one all-reduce; for attention, three all-gathers and one all-reduce, or fused variants with larger payloads (Wang et al., 27 Oct 2025).

BD-LoRA avoids these collectives by ensuring that each adapter intermediate stays inside the local tensor-parallel shard. The appendix-level device-local forward structure is:

$N$7

followed by local addition to $N$8. The second projection proceeds analogously:

$N$9

again with local addition. The only collective is the final

$r$0

which is already required by the base Megatron-style model. In that sense, BD-LoRA makes the LoRA path communication-free relative to the base execution graph (Wang et al., 27 Oct 2025).

The paper also gives an equivalent interpretation: BD-LoRA is equivalent to attaching independent LoRA adapters of rank $r$1 to each shard of the base model. This interpretation is operationally important because it turns the method into standard LoRA training on a modified architecture composed of shard-specific linear layers, rather than a separate PEFT formalism (Wang et al., 27 Oct 2025).

4. Parameterization, training, and inference

The block-diagonal constraint reduces parameter count at fixed nominal rank because the constrained factors contain only shard-local blocks. For one basic MLP module, the per-device parameter count of S-LoRA is

$r$2

while BD-LoRA uses

$r$3

where $r$4 is the BD-LoRA rank. Equal parameter count is obtained when

$r$5

The paper further notes that, at the same rank, BD-LoRA has roughly half the number of trainable parameters as standard LoRA because some matrices are block-diagonal. For a matched parameter budget, compute is comparable: for one MLP module, S-LoRA requires

$r$6

whereas BD-LoRA requires

$r$7

The stated argument is that end-to-end runtime gains come not from lower matmul complexity at equal effective budget but from removal of communication kernels (Wang et al., 27 Oct 2025).

Training does not require a distinct optimization framework. The implementation rewrites each sharded projection as separate linear layers and attaches standard LoRA adapters of rank $r$8 to each shard in Hugging Face Transformers + PEFT. Because BD-LoRA is interpreted as $r$9 independent rank-$S$0 adapters, the scaling factor becomes

$S$1

The reported fine-tuning setup uses AdamW with $S$2; for GLUE, learning rate $S$3, early stopping, and maximum sequence length 128; for OpenOrca, 20k training and 20k evaluation examples with learning rate $S$4 (Wang et al., 27 Oct 2025).

At inference time, adapter tensors are loaded and sharded in vLLM, but block-diagonal matrices are stored without zeros. The paper states that $S$5 is stored in compact packed form as shape $S$6 by placing blocks side by side, and $S$7 as shape $S$8 by stacking blocks. BD-LoRA keeps S-LoRA’s memory management, scheduling, request handling, and caching mechanisms unchanged. A major limitation is that the tensor-parallel degree $S$9 must be known at training time because the adapter structure depends on deployment sharding (Wang et al., 27 Oct 2025).

The expressivity tradeoff is explicit. BD-LoRA is not mathematically equivalent to standard dense LoRA: by design it removes cross-shard coupling, because each shard has an independent local adapter. The paper’s position is pragmatic rather than theoretical: for a similar number of effective parameters, the reduction in cross-shard interactions is acceptable if downstream quality remains close while serving efficiency improves (Wang et al., 27 Oct 2025).

5. Empirical behavior

The fine-tuning experiments use Llama-3.2-1B and Llama-3.1-8B on GLUE tasks—MNLI, SST-2, QNLI, QQP, MRPC, CoLA, RTE, and STS-B—and on OpenOrca language modeling evaluated with perplexity. The reported summary result is that, for a similar number of trainable or effective parameters, BD-LoRA and standard LoRA achieve very similar downstream performance. On Llama-3.1-8B, LoRA rank 16 uses 41.9M parameters and obtains OpenOrca perplexity 2.32 with GLUE average 75.8, whereas BD-LoRA rank 32 uses 36.2M parameters and obtains OpenOrca perplexity 2.32 with GLUE average 75.6. At larger budgets, LoRA rank 64 gives 167.8M parameters, OpenOrca 2.30, GLUE 76.0, while BD-LoRA rank 128 gives 144.7M parameters, OpenOrca 2.30, GLUE 76.2; LoRA rank 256 gives 671.1M parameters, OpenOrca 2.29, GLUE 76.4, while BD-LoRA rank 512 gives 578.8M parameters, OpenOrca 2.29, GLUE 76.6 (Wang et al., 27 Oct 2025).

The runtime evaluation is conducted in vLLM on AWS p4d systems with 8 NVIDIA A100 GPUs and NVLink Switch, and on AWS g5 systems with 8 NVIDIA A10G GPUs and PCIe interconnect. Measurements include throughput, end-to-end latency, decoding latency, and prefill latency under both single cached-adapter and different-adapter-per-request regimes. The headline abstract results on 8 A100 GPUs report up to 1.79x end-to-end speed-up with 0.87x adapter parameters, and up to 1.23x speed-up with 1.74x adapter parameters, for Llama-3.1-70B; for Llama-3.1-8B, up to 1.63x speed-up with 0.86x adapter parameters, and up to 1.30x with 1.73x adapter parameters. On 4 A100 GPUs, Llama-3.1-8B reaches up to 1.27x speed-up with 1.03x parameters. On 8 A10G GPUs, the reported gains are up to 1.36x with 0.86x parameters or 1.27x with 1.73x parameters (Wang et al., 27 Oct 2025).

The paper emphasizes that BD-LoRA strictly Pareto-dominates S-LoRA in performance-versus-runtime plots: for matched downstream quality, BD-LoRA is faster. The speedup is concentrated in decoding rather than prefill, because decoding communicates small tensors for which startup latency dominates. The advantage grows with higher rank, smaller batch size, generation-heavy workloads, higher tensor-parallel degree, and attention-heavy adaptation, since S-LoRA communication overhead is then more consequential. When each request uses a different adapter, speedup is reduced because disk-loading overhead is shared by all methods, but the paper still reports up to 1.78x (Wang et al., 27 Oct 2025).

6. Relation to adjacent “block” LoRA methods

The literature described here uses the word block for several distinct constructions. Only some of them impose a block-diagonal constraint inside the adapted matrix; others operate at the level of network topology, per-layer rank allocation, latent-space diagonal modulation, or full block-grid routing.

Method	Structural locus	Relation to BD-LoRA
Block-wise LoRA (Li et al., 2024)	U-Net stage selection in Stable Diffusion	Block-wise/per-stage, not block-diagonal
FIM-LoRA (Sathyavageeswaran, 16 May 2026)	Per-layer rank allocation over standard LoRA	Complementary rank-budget method
BoRA (Li et al., 9 Aug 2025)	Full $\frac{2(N-1)d_HS}{N}.$0 block interactions with $\frac{2(N-1)d_HS}{N}.$1	Broader block-partitioned framework
Ouroboros (Jaber et al., 2 Apr 2026)	$\frac{2(N-1)d_HS}{N}.$2 with dynamic controller	Dynamic diagonal latent modulation
Localized LoRA-MoE (Barazandeh et al., 6 Jul 2026)	Full block-grid local experts with routing	More general than block-diagonal

The distinction is especially sharp for Stable Diffusion “block-wise LoRA.” That method partitions the U-Net into four input blocks, one mid-block, and four output blocks, then activates or skips ordinary LoRA or LoCon modules in selected stages by setting the rank to zero in skipped blocks. The paper explicitly does not define a block-diagonal $\frac{2(N-1)d_HS}{N}.$3, $\frac{2(N-1)d_HS}{N}.$4, or $\frac{2(N-1)d_HS}{N}.$5; the decomposition is architectural over network stages, not algebraic within a single weight matrix (Li et al., 2024).

FIM-LoRA changes a different design axis. It keeps the standard LoRA parameterization and uses calibration-time gradient-variance estimates of the LoRA-$\frac{2(N-1)d_HS}{N}.$6 matrices to redistribute a fixed total rank budget across layers or modules. The resulting adapter is still “a standard LoRA with a per-layer rank pattern,” so it is best understood as heterogeneous rank allocation rather than block-diagonal structure (Sathyavageeswaran, 16 May 2026).

BoRA is closer in algebraic spirit because it rewrites LoRA as a block matrix multiplication. It partitions

$\frac{2(N-1)d_HS}{N}.$7

and defines

$\frac{2(N-1)d_HS}{N}.$8

for all $\frac{2(N-1)d_HS}{N}.$9. Because BoRA retains all $r \ll d_H$0 block interactions, it is not block-diagonal in the usual sense. However, the paper explicitly states that LoRA is a special case when $r \ll d_H$1 for all $r \ll d_H$2, and MELoRA is a special case when $r \ll d_H$3 for $r \ll d_H$4 and $r \ll d_H$5 for $r \ll d_H$6. This makes BoRA a broader block-partitioned framework within which a diagonal-only interaction pattern can be represented as a restriction (Li et al., 9 Aug 2025).

Ouroboros and Localized LoRA-MoE move in still different directions. Ouroboros uses

$r \ll d_H$7

where a controller generates step- and input-dependent diagonal modulation vectors over frozen low-rank bases; the structure is diagonal in the rank dimension, not block-diagonal in weight space (Jaber et al., 2 Apr 2026). Localized LoRA-MoE partitions the whole update into a $r \ll d_H$8 grid of cells, equips each cell with local low-rank experts, and allows routed off-diagonal block updates; it is therefore more general than canonical block-diagonal LoRA and explicitly designed to preserve cross-field interactions (Barazandeh et al., 6 Jul 2026).

A plausible implication is that BD-LoRA, heterogeneous rank allocation, and dynamic coefficient modulation occupy complementary design axes rather than mutually exclusive categories. The direct BD-LoRA paper, however, is narrower and more specific: its contribution is a serving-oriented block-diagonal construction whose purpose is to align LoRA with tensor-parallel execution and eliminate additional communication overhead (Wang et al., 27 Oct 2025).