---
title: Bi-Dimensional Decomposition (BoRA)
url: https://www.emergentmind.com/topics/bi-dimensional-decomposition-bora
type: topic
---

# Bi-Dimensional Decomposition (BoRA)

Bi-dimensional Decomposition (BoRA) refers to a class of strategies for enhancing expressivity and efficiency in neural network adaptation and Boolean logic synthesis by exploiting structured, two-dimensional (bi-dimensional) decompositions—typically involving block diversity, symmetric magnitude modulation, or optimal variable partitioning. Three prominent lines in the literature employ the "BoRA" terminology: Block-Diversified Low-Rank Adaptation for parameter-efficient fine-tuning in deep learning [2508.06953], Bi-dimensional Weight-Decomposed Low-Rank Adaptation for magnitude-symmetric neural adaptation [2412.06441], and QBF-based Boolean Function Bi-Decomposition in logic synthesis [1112.2313]. All BoRA approaches share the use of two-way decompositions that increase structural or functional diversity and improve task performance under resource constraints.

## 1. Background and Motivation

Parameter-efficient fine-tuning (PEFT) has become essential for adapting large neural models with minimal overhead, typically using low-rank adaptation (LoRA). LoRA decomposes an update to a pretrained weight $W \in \mathbb{R}^{m \times n}$ as $BA$ with $A \in \mathbb{R}^{r \times n}, B \in \mathbb{R}^{m \times r}$, $r \ll \min(m, n)$, yielding an update of rank at most $r$ with $(m+n)r$ parameters. Increasing $r$ improves capacity but quickly increases costs.

Block-Diversified Low-Rank Adaptation (BoRA) [2508.06953] extends LoRA by imposing a two-dimensional block structure and introducing block-wise diagonal modulations to maximize expressivity with only a small increase in parameter count. Separately, the bi-dimensional weight-decomposed BoRA approach [2412.06441] introduces symmetric row- and column-wise magnitude modulation on adapter weights, and QBF-based BoRA in logic synthesis [1112.2313] seeks optimal variable partitioning for Boolean function bi-decomposition. In each context, bi-dimensionality (via explicit row/column or blockwise structure) enables a richer adaptation or decomposition.

## 2. Block-Diversified Low-Rank Adaptation (BoRA) in Neural Networks

BoRA [2508.06953] generalizes LoRA by treating its low-rank parameter update as a grid of $b \times b$ blocks. Specifically, $A$ is partitioned by columns and $B$ by rows:
- $A = [A_1, \dots, A_b]$, $A_j \in \mathbb{R}^{r \times (n/b)}$
- $B = [B_1; \dots; B_b]^\top$, $B_i \in \mathbb{R}^{(m/b) \times r}$

The update becomes
\[
\Delta W = BA = 
\begin{bmatrix}
B_1 A_1 & \dots & B_1 A_b \\
\vdots & \ddots & \vdots \\
B_b A_1 & \dots & B_b A_b
\end{bmatrix}
\]
Each sub-block $\Delta W_{i,j} = B_i A_j$. To break inter-block coupling and raise the effective rank, BoRA inserts for each block a diagonal matrix $\Sigma_{i,j} \in \mathbb{R}^{r \times r}$:
\[
\Delta W_{i,j}^{\text{BoRA}} = B_i \Sigma_{i,j} A_j
\]
With $b^2$ such matrices, total parameters increase only by $b^2 r$, since each $\Sigma_{i,j}$ is diagonal. The update generalizes to
\[
\Delta W = B' \Sigma' A'
\]
where $B' \in \mathbb{R}^{m \times br}$ concatenates all row blocks, $A' \in \mathbb{R}^{br \times n}$ stacks the column blocks, and $\Sigma' \in \mathbb{R}^{br \times br}$ is block-diagonal from all $\Sigma_{i,j}$.

**Effective rank:** BoRA raises the upper bound of the update rank from $r$ (the LoRA limit) to $br$. This allows higher expressivity at fixed or slightly increased parameter count.

**Parameter overhead:** Adding $b^2 r$ parameters for $\Sigma$ to $(m+n)r$ for $A$ and $B$ is negligible when $b$ and $r$ are small and $m,n$ are large.

**Implementation:** For each input, the computation is blockwise: project each input slice through its $A_j$, modulate via $\exp(\sigma_{i,j}/\mathrm{mean\_abs}(\sigma))$, combine across $j$ for each output block $i$, then project via $B_i$ and sum.

## 3. Bi-dimensional Weight-Decomposed Low-Rank Adaptation (BoRA) for Symmetric Modulation

The variant introduced in [2412.06441] addresses the asymmetry of DoRA, which only introduces column-wise (vertical) magnitude scaling. This BoRA symmetrically modulates both rows and columns:

1. **Adapter construction:** Given frozen $W_0 \in \mathbb{R}^{h_r \times h_c}$,
   \[
   W = W_0 + A B
   \]
   with learnable $A \in \mathbb{R}^{h_r \times r}$ and $B \in \mathbb{R}^{r \times h_c}$.

2. **Row-wise normalization and scaling:** Normalize each row of $W_0 + AB$ and scale by $m^r \in \mathbb{R}^{h_r \times 1}$:
   \[
   V^r = \frac{W_0 + AB}{\| (W_0 + AB)^{(*,r)} \|}, \quad H = \mathrm{diag}(m^r) V^r
   \]

3. **Column-wise normalization and scaling:** Normalize each column of $H$ and scale by $m^c \in \mathbb{R}^{1 \times h_c}$:
   \[
   V^c = \frac{H}{\| H^{(*,c)} \|},\quad W = \mathrm{diag}(m^c) V^c
   \]

This approach aligns the adaptation to both input (column) and output (row) sensitivities, providing a bi-dimensional symmetry that matches empirical patterns in full-parameter fine-tuning.

## 4. Rank, Parameter, and Compute Characteristics

| Method         | Update Rank | Trainable Parameters    | Extra Overhead    |
|----------------|-------------|------------------------|-------------------|
| LoRA           | $r$         | $(m+n) r$              | $(m+n) r$ FLOPs   |
| BoRA (block)   | $b r$       | $(m+n) r + b^2 r$      | $(m+n) r + b^2 r$ |
| DoRA           | $r$         | $(m+n) r + n$          | $\sim$2x LoRA     |
| BoRA (symm)    | $r$         | $2r(m+n) + m + n$      | 2 normalizations  |

With $b=8-16$, $r$ small, and $m,n\gg r$, additional costs from BoRA in either formulation are minor relative to gains in expressivity or transfer performance [2508.06953][2412.06441].

## 5. Empirical Results and Comparative Performance

Block-partitioned BoRA with $r=8$ and $b=8$–$16$ consistently outperforms vanilla LoRA (same rank) by $2$–$4\,\%$ absolute across natural language understanding (GLUE/commonsense reasoning) and reasoning (Gemma/LLaMA/Qwen) tasks, and often matches or exceeds LoRA at $4\times$ the rank [2508.06953].

Bi-dimensional magnitude-modulated BoRA outperforms LoRA and DoRA across MT-Bench and commonsense NLU with only slight parameter increase, e.g., Llama-2-7B: BoRA (2.35% params) achieves 6.76 (MT-Bench), compared to LoRA (2.32%) at 6.16 and DoRA (2.33%) at 6.38 [2412.06441].

Ablation studies indicate both diagonal parameterization and normalization in $\Sigma$ are crucial, with weak utilization or gradient issues if omitted. Singular value analysis confirms that BoRA achieves substantially more effective singular values in $\Delta W$, aligning with its theoretical rank increase.

## 6. QBF-Based BoRA in Boolean Function Bi-Decomposition

In digital logic, bi-decomposition refers to splitting a Boolean function $f(X)$ into two subfunctions combined by a binary gate. The QBF-based BoRA scheme [1112.2313] achieves optimal variable partitioning for
\[
f(X) = f_A(X_A, X_C) \,\mathrm{op}\, f_B(X_B, X_C)
\]
where $X$ is partitioned into $X_A$, $X_B$, $X_C$, and $\mathrm{op} \in \{\vee, \wedge, \oplus\}$. The method uses Quantified Boolean Formulas (QBF) over partition indicator variables $\{\alpha_x, \beta_x\}$ and universally-quantified copies of inputs to enforce the decomposition and desired metrics (disjointness, balancedness).

Empirically, this method yields strictly higher-quality decompositions versus prior SAT/BDD-based approaches, with optimal guarantees and scalable performance on industrial circuits. Disjointness, balancedness, and joint metrics improve across 13%–80% of test cases versus SAT-based baselines, at modest additional computational cost.

## 7. Extensions, Limitations, and Perspectives

BoRA offers a principled route to increased expressivity with controllable overhead in both neural PEFT and Boolean logic synthesis. In the adaptation setting:
- Scaling $b$ increases effective update rank until overfitting occurs.
- Extra diagonal parameters are negligible for practical $b$, $r$.
- The block/symmetric decompositions integrate with existing architectures without architectural redesign.

Potential extensions include grouping rows/columns for structured sharing, learning asymmetric row/column ranks, and combining with quantized or further-structured modules. Limitations include the possibility of overfitting if $b$ is too large or if not paired with sufficient regularization. Slight increases in computation (1–5%) are observed due to block or normalization operations.

In summary, bi-dimensional decomposition via BoRA defines a family of methods wherein blockwise or symmetric modulation/gating multiplies capacity and expressivity at minimal additional cost, with robust empirical advantages across architectures and domains [2508.06953][2412.06441][1112.2313].

Source: https://www.emergentmind.com/topics/bi-dimensional-decomposition-bora