---
title: Block-Diagonal LoRA for Efficient LLM Serving
url: https://www.emergentmind.com/topics/block-diagonal-lora-bd-lora
type: topic
---

# Block-Diagonal LoRA for Efficient LLM Serving

Block-Diagonal LoRA (BD-LoRA) is a structured variant of Low-Rank Adaptation in which selected LoRA factors are constrained to be block-diagonal so that adapter computation aligns with the tensor-parallel shard structure of a base large language model during multi-adapter online serving. In the formulation introduced for this setting, the method targets the case of one frozen base LLM and many LoRA adapters served concurrently, where adapters cannot simply be merged into the base weights because adapter swapping would create overhead and requests using different adapters could not be batched. Its defining systems property is that, under tensor parallelism, BD-LoRA adds no additional communication for LoRA computation beyond the base model’s own collectives, while achieving downstream performance similar to standard LoRA at a similar parameter budget [2510.23346].

## 1. Problem setting and design objective

BD-LoRA was introduced for the serving regime in which a single base LLM is shared across several different LoRA adapters and executed with tensor parallelism across multiple GPUs. In that regime, the serving system computes the adapter path on the fly as \(XW + XAB\), rather than permanently merging \(W+\Delta W\) into the base model. The motivation is operational rather than purely algebraic: if requests switch between adapters, repeatedly subtracting the old adapter contribution and adding the new one creates overhead, and if adapters are merged into weights then batched inference works naturally only when all requests in a batch use the same adapter [2510.23346].

The immediate precursor discussed in the same context is S-LoRA, which shards LoRA adapters in a way compatible with Megatron-LM tensor parallelism but still incurs blocking communication. For one attention module, the reported extra communication is
$$
\frac{5(N-1)rS}{N},
$$
where \(N\) is the number of devices, \(r\) the LoRA rank, and \(S\) the number of tokens, whereas the base model communication is
$$
\frac{2(N-1)d_HS}{N}.
$$
Although this appears small when \(r \ll d_H\), the paper emphasizes that larger rsLoRA ranks are useful and that communication startup latency is ignored by bandwidth-only estimates. BD-LoRA addresses this by constraining certain LoRA factors to be block-diagonal so that the adapter path follows the base model’s tensor-parallel dataflow and requires no extra inter-device communication [2510.23346].

## 2. Algebraic construction

In the paper’s row-major notation, a standard LoRA-adapted linear layer with base weight
$$
W \in \mathbb{R}^{d_{in}\times d_{out}}
$$
uses
$$
W' = W + \Delta W, \qquad \Delta W = AB,
$$
with
$$
A \in \mathbb{R}^{d_{in}\times r}, \qquad B \in \mathbb{R}^{r\times d_{out}},
$$
and forward pass
$$
XW' = XW + XAB, \qquad X \in \mathbb{R}^{S\times d_{in}}.
$$
During training, the implementation uses rsLoRA scaling
$$
\gamma_r = \frac{\alpha}{\sqrt{r}}.
$$
BD-LoRA does not retain fully dense LoRA factors. Instead, it constrains one factor in each LoRA pair to be block-diagonal while the other factor remains dense but is sharded compatibly [2510.23346].

The underlying locality principle is the standard block-diagonal matrix identity. If
$$
X = [X^1 \mid X^2 \mid \cdots \mid X^N] \in \mathbb{R}^{S\times d_{in}},
$$
with
$$
X^i \in \mathbb{R}^{S\times d_{in}/N},
$$
and
$$
W =
\begin{bmatrix}
W^1 & 0 & \cdots & 0\\
0 & W^2 & \cdots & 0\\
0 & 0 & \ddots & 0\\
0 & \cdots & 0 & W^N
\end{bmatrix},
\qquad
W^i \in \mathbb{R}^{(d_{in}/N)\times(d_{out}/N)},
$$
then
$$
XW = [X^1W^1 \mid X^2W^2 \mid \cdots \mid X^NW^N],
$$
so each device computes its shard independently with no communication. BD-LoRA engineers the adapter factors so that the LoRA path inherits the same property [2510.23346].

For a Megatron-style MLP with
$$
W_1 \in \mathbb{R}^{d_H\times d_I}, \qquad W_2 \in \mathbb{R}^{d_I\times d_H},
$$
and forward pass \(\sigma(XW_1)W_2\), the first projection \(W_1\) is column-parallel and the second projection \(W_2\) is row-parallel. BD-LoRA uses the following asymmetric construction.

For the first projection,
$$
A_1 = [A_1^{(1)} \mid A_1^{(2)} \mid \cdots \mid A_1^{(N)}],
\qquad
A_1^{(i)} \in \mathbb{R}^{d_H\times r/N},
$$
and
$$
B_1 =
\begin{bmatrix}
B_1^{(1)} & 0 & \cdots & 0\\
0 & B_1^{(2)} & \cdots & 0\\
0 & 0 & \ddots & 0\\
0 & \cdots & 0 & B_1^{(N)}
\end{bmatrix},
\qquad
B_1^{(i)} \in \mathbb{R}^{(r/N)\times(d_I/N)}.
$$
Then
$$
XA_1B_1 = [XA_1^{(1)}B_1^{(1)} \mid \cdots \mid XA_1^{(N)}B_1^{(N)}],
$$
which matches the column-sharded layout of \(XW_1\) [2510.23346].

For the second projection,
$$
A_2 =
\begin{bmatrix}
A_2^{(1)} & 0 & \cdots & 0\\
0 & A_2^{(2)} & \cdots & 0\\
0 & 0 & \ddots & 0\\
0 & \cdots & 0 & A_2^{(N)}
\end{bmatrix},
\qquad
A_2^{(i)} \in \mathbb{R}^{(d_I/N)\times(r/N)},
$$
and
$$
B_2 =
\begin{bmatrix}
B_2^{(1)}\\
B_2^{(2)}\\
\vdots\\
B_2^{(N)}
\end{bmatrix},
\qquad
B_2^{(i)} \in \mathbb{R}^{(r/N)\times d_H}.
$$
Because the intermediate activation is already sharded over the \(d_I\) dimension, each device computes \(YA_2^{(i)}B_2^{(i)}\) locally, and the resulting shard layout matches the base row-parallel \(YW_2\) path [2510.23346].

## 3. Sharding behavior and communication elimination

The central systems claim of BD-LoRA is not merely reduced communication but zero additional communication for the LoRA computation under tensor parallelism. In S-LoRA, communication arises because partial rank-space results must be assembled across devices. For the first MLP projection, S-LoRA shards both \(A_1\) and \(B_1\) column-wise; after each device computes its local \(XA_1^{(i)}\), the full rank dimension is needed before multiplication by \(B_1\), so an all-gather is required. For the second projection, S-LoRA shards \(A_2\) row-wise and \(B_2\) column-wise; after multiplication by \(A_2\), devices hold partial contributions over the shared rank dimension and must combine them via all-reduce. For a basic MLP, this yields two additional communication operations on top of the base model, one all-gather and one all-reduce. For GLU MLP variants, the paper reports two all-gathers and one all-reduce; for attention, three all-gathers and one all-reduce, or fused variants with larger payloads [2510.23346].

BD-LoRA avoids these collectives by ensuring that each adapter intermediate stays inside the local tensor-parallel shard. The appendix-level device-local forward structure is:
$$
Y_1^{(i)} \gets XW_1^{(i)}, \qquad
Z_1^{(i)} \gets XA_1^{(i)}, \qquad
Y_{1,\mathrm{adapter}}^{(i)} \gets Z_1^{(i)}B_1^{(i)},
$$
followed by local addition to \(Y_1^{(i)}\). The second projection proceeds analogously:
$$
Y_2^{(i)} \gets Y_1^{(i)}W_2^{(i)}, \qquad
Z_2^{(i)} \gets Y_1^{(i)}A_2^{(i)}, \qquad
Y_{2,\mathrm{adapter}}^{(i)} \gets Z_2^{(i)}B_2^{(i)},
$$
again with local addition. The only collective is the final
$$
Y \gets \mathrm{AllReduce}\big(\{Y_2^{(1)},\dots,Y_2^{(N)}\}\big),
$$
which is already required by the base Megatron-style model. In that sense, BD-LoRA makes the LoRA path communication-free relative to the base execution graph [2510.23346].

The paper also gives an equivalent interpretation: BD-LoRA is equivalent to attaching independent LoRA adapters of rank \(r/N\) to each shard of the base model. This interpretation is operationally important because it turns the method into standard LoRA training on a modified architecture composed of shard-specific linear layers, rather than a separate PEFT formalism [2510.23346].

## 4. Parameterization, training, and inference

The block-diagonal constraint reduces parameter count at fixed nominal rank because the constrained factors contain only shard-local blocks. For one basic MLP module, the per-device parameter count of S-LoRA is
$$
2(d_H+d_I)\frac{r}{N},
$$
while BD-LoRA uses
$$
2\left(d_H+\frac{d_I}{N}\right)\frac{r'}{N},
$$
where \(r'\) is the BD-LoRA rank. Equal parameter count is obtained when
$$
r' = r\frac{d_H+d_I}{d_H+\frac{d_I}{N}}.
$$
The paper further notes that, at the same rank, BD-LoRA has roughly half the number of trainable parameters as standard LoRA because some matrices are block-diagonal. For a matched parameter budget, compute is comparable: for one MLP module, S-LoRA requires
$$
4S(d_H+d_I)\frac{r}{N},
$$
whereas BD-LoRA requires
$$
4S\left(d_H+\frac{d_I}{N}\right)\frac{r'}{N}.
$$
The stated argument is that end-to-end runtime gains come not from lower matmul complexity at equal effective budget but from removal of communication kernels [2510.23346].

Training does not require a distinct optimization framework. The implementation rewrites each sharded projection as separate linear layers and attaches standard LoRA adapters of rank \(r/N\) to each shard in Hugging Face Transformers + PEFT. Because BD-LoRA is interpreted as \(N\) independent rank-\(r/N\) adapters, the scaling factor becomes
$$
\frac{\alpha\sqrt{N}}{\sqrt{r}}.
$$
The reported fine-tuning setup uses AdamW with \(\alpha=16\); for GLUE, learning rate \(10^{-5}\), early stopping, and maximum sequence length 128; for OpenOrca, 20k training and 20k evaluation examples with learning rate \(5\cdot 10^{-5}\) [2510.23346].

At inference time, adapter tensors are loaded and sharded in vLLM, but block-diagonal matrices are stored without zeros. The paper states that \(B_1\) is stored in compact packed form as shape \(\frac{r}{N}\times d_I\) by placing blocks side by side, and \(A_2\) as shape \(d_I\times \frac{r}{N}\) by stacking blocks. BD-LoRA keeps S-LoRA’s memory management, scheduling, request handling, and caching mechanisms unchanged. A major limitation is that the tensor-parallel degree \(N\) must be known at training time because the adapter structure depends on deployment sharding [2510.23346].

The expressivity tradeoff is explicit. BD-LoRA is not mathematically equivalent to standard dense LoRA: by design it removes cross-shard coupling, because each shard has an independent local adapter. The paper’s position is pragmatic rather than theoretical: for a similar number of effective parameters, the reduction in cross-shard interactions is acceptable if downstream quality remains close while serving efficiency improves [2510.23346].

## 5. Empirical behavior

The fine-tuning experiments use Llama-3.2-1B and Llama-3.1-8B on GLUE tasks—MNLI, SST-2, QNLI, QQP, MRPC, CoLA, RTE, and STS-B—and on OpenOrca language modeling evaluated with perplexity. The reported summary result is that, for a similar number of trainable or effective parameters, BD-LoRA and standard LoRA achieve very similar downstream performance. On Llama-3.1-8B, LoRA rank 16 uses 41.9M parameters and obtains OpenOrca perplexity 2.32 with GLUE average 75.8, whereas BD-LoRA rank 32 uses 36.2M parameters and obtains OpenOrca perplexity 2.32 with GLUE average 75.6. At larger budgets, LoRA rank 64 gives 167.8M parameters, OpenOrca 2.30, GLUE 76.0, while BD-LoRA rank 128 gives 144.7M parameters, OpenOrca 2.30, GLUE 76.2; LoRA rank 256 gives 671.1M parameters, OpenOrca 2.29, GLUE 76.4, while BD-LoRA rank 512 gives 578.8M parameters, OpenOrca 2.29, GLUE 76.6 [2510.23346].

The runtime evaluation is conducted in vLLM on AWS p4d systems with 8 NVIDIA A100 GPUs and NVLink Switch, and on AWS g5 systems with 8 NVIDIA A10G GPUs and PCIe interconnect. Measurements include throughput, end-to-end latency, decoding latency, and prefill latency under both single cached-adapter and different-adapter-per-request regimes. The headline abstract results on 8 A100 GPUs report up to 1.79x end-to-end speed-up with 0.87x adapter parameters, and up to 1.23x speed-up with 1.74x adapter parameters, for Llama-3.1-70B; for Llama-3.1-8B, up to 1.63x speed-up with 0.86x adapter parameters, and up to 1.30x with 1.73x adapter parameters. On 4 A100 GPUs, Llama-3.1-8B reaches up to 1.27x speed-up with 1.03x parameters. On 8 A10G GPUs, the reported gains are up to 1.36x with 0.86x parameters or 1.27x with 1.73x parameters [2510.23346].

The paper emphasizes that BD-LoRA strictly Pareto-dominates S-LoRA in performance-versus-runtime plots: for matched downstream quality, BD-LoRA is faster. The speedup is concentrated in decoding rather than prefill, because decoding communicates small tensors for which startup latency dominates. The advantage grows with higher rank, smaller batch size, generation-heavy workloads, higher tensor-parallel degree, and attention-heavy adaptation, since S-LoRA communication overhead is then more consequential. When each request uses a different adapter, speedup is reduced because disk-loading overhead is shared by all methods, but the paper still reports up to 1.78x [2510.23346].

## 6. Relation to adjacent “block” LoRA methods

The literature described here uses the word *block* for several distinct constructions. Only some of them impose a block-diagonal constraint inside the adapted matrix; others operate at the level of network topology, per-layer rank allocation, latent-space diagonal modulation, or full block-grid routing.

| Method | Structural locus | Relation to BD-LoRA |
|---|---|---|
| Block-wise LoRA [2403.07500] | U-Net stage selection in Stable Diffusion | Block-wise/per-stage, not block-diagonal |
| FIM-LoRA [2605.16800] | Per-layer rank allocation over standard LoRA | Complementary rank-budget method |
| BoRA [2508.06953] | Full \(b\times b\) block interactions with \(\Sigma_{i,j}\) | Broader block-partitioned framework |
| Ouroboros [2604.02051] | \(B\,\mathrm{diag}(\delta^{(t)})\,A\) with dynamic controller | Dynamic diagonal latent modulation |
| Localized LoRA-MoE [2607.05114] | Full block-grid local experts with routing | More general than block-diagonal |

The distinction is especially sharp for Stable Diffusion “block-wise LoRA.” That method partitions the U-Net into four input blocks, one mid-block, and four output blocks, then activates or skips ordinary LoRA or LoCon modules in selected stages by setting the rank to zero in skipped blocks. The paper explicitly does not define a block-diagonal \(A\), \(B\), or \(\Delta W\); the decomposition is architectural over network stages, not algebraic within a single weight matrix [2403.07500].

FIM-LoRA changes a different design axis. It keeps the standard LoRA parameterization and uses calibration-time gradient-variance estimates of the LoRA-\(B\) matrices to redistribute a fixed total rank budget across layers or modules. The resulting adapter is still “a standard LoRA with a per-layer rank pattern,” so it is best understood as heterogeneous rank allocation rather than block-diagonal structure [2605.16800].

BoRA is closer in algebraic spirit because it rewrites LoRA as a block matrix multiplication. It partitions
$$
A=[A_1,\dots,A_b], \qquad B=[B_1,\dots,B_b]^\top
$$
and defines
$$
\Delta W_{i,j}=B_i\Sigma_{i,j}A_j
$$
for all \(i,j\in[b]\). Because BoRA retains all \(b^2\) block interactions, it is not block-diagonal in the usual sense. However, the paper explicitly states that LoRA is a special case when \(\Sigma_{i,j}=I\) for all \(i,j\), and MELoRA is a special case when \(\Sigma_{i,j}=I\) for \(i=j\) and \(\Sigma_{i,j}=0\) for \(i\ne j\). This makes BoRA a broader block-partitioned framework within which a diagonal-only interaction pattern can be represented as a restriction [2508.06953].

Ouroboros and Localized LoRA-MoE move in still different directions. Ouroboros uses
$$
\Delta W_k^{(t)} = \frac{\alpha}{r}\, B_k\,\mathrm{diag}(\delta_k^{(t)})\,A_k,
$$
where a controller generates step- and input-dependent diagonal modulation vectors over frozen low-rank bases; the structure is diagonal in the rank dimension, not block-diagonal in weight space [2604.02051]. Localized LoRA-MoE partitions the whole update into a \(K_y\times K_x\) grid of cells, equips each cell with local low-rank experts, and allows routed off-diagonal block updates; it is therefore more general than canonical block-diagonal LoRA and explicitly designed to preserve cross-field interactions [2607.05114].

A plausible implication is that BD-LoRA, heterogeneous rank allocation, and dynamic coefficient modulation occupy complementary design axes rather than mutually exclusive categories. The direct BD-LoRA paper, however, is narrower and more specific: its contribution is a serving-oriented block-diagonal construction whose purpose is to align LoRA with tensor-parallel execution and eliminate additional communication overhead [2510.23346].

Source: https://www.emergentmind.com/topics/block-diagonal-lora-bd-lora