BD Attention: Basis Decomposition Reformulation

• BD Attention is a basis decomposition-based reparameterization that transforms multi-head attention by reducing projection parameters while preserving exact output under arithmetic precision.
• It decomposes the QK and VO projection paths into compact basis and coefficient forms, significantly cutting floating-point multiplications and storage requirements.
• The method yields practical speedups (up to 1.33×) and minimal numerical error, enabling efficient, post-hoc acceleration of pre-trained large language and vision–language models.

Searching arXiv for the main paper and closely related attention-optimization references to ground the article. BD Attention (BDA) denotes a Basis Decomposition-based reformulation of multi-head attention in which the query–key and value–output projection paths are reparameterized into compact basis-and-coefficient forms while preserving the original attention mapping under exact arithmetic. In the formulation introduced for LLMs and vision–LLMs, BDA is an algorithmic, post-hoc, lossless method: it reduces projection parameters and floating-point multiplications, requires only a short offline conversion step, and does not require retraining (Zhao, 2 Oct 2025). The acronym “BDA” is also used in unrelated literatures for “Balanced Distribution Adaptation” in transfer learning (Wang et al., 2018) and “Bezier Deformable Attention” in road-topology understanding (Kalfaoglu et al., 2024); in attention acceleration, however, “BD Attention” refers specifically to Basis Decomposition (Zhao, 2 Oct 2025).

1. Position in the attention-optimization landscape

BDA occupies a distinct point in the design space of Transformer acceleration. Standard multi-head attention is expensive because the projection matrices for $Q$, $K$, $V$, and $O$ dominate parameter count and FLOPs in each layer, with costs growing with hidden dimension $d$, number of heads $n$, and per-head dimension $d_h$ (Zhao, 2 Oct 2025). BDA targets these projection costs directly.

The central distinction is between system-level exact methods and algorithmic approximations. I/O-aware kernels such as FlashAttention and related serving optimizations such as PagedAttention preserve the attention function but mainly improve execution by reordering computation and memory access; they do not reduce arithmetic operations or parameter count (Zhao, 2 Oct 2025, Dao et al., 2022, Kwon et al., 2023). By contrast, approximate algorithmic families—including linear attention, sparse attention, pruning, low-rank compression, and quantization—typically alter the function being computed and often require retraining or calibration (Zhao, 2 Oct 2025, Wang et al., 2020, Choromanski et al., 2020, Child et al., 2019, Beltagy et al., 2020, Kitaev et al., 2020).

BDA is unusual because it is both algorithmic and lossless. The method changes the parameterization and execution path of attention projections, yet preserves exact outputs in exact arithmetic. This suggests that the usual opposition between “exact but only systems-level” and “algorithmic but approximate” is not exhaustive: BDA introduces a third category, namely mathematically exact algorithmic reformulation (Zhao, 2 Oct 2025).

2. Basis Decomposition as the algebraic foundation

The method begins from a generic rank-$r$ matrix product

$$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$

Rather than storing $\mathbf{U}$ and $K$0 as an ordinary low-rank factorization, BD rewrites $K$1 in basis form. If $K$2 has rank $K$3, there exist $K$4 linearly independent rows or columns from which all remaining rows or columns can be reconstructed exactly. In row-based form, if $K$5 contains basis rows and $K$6 contains reconstruction coefficients, the paper gives the identities

$K$7

and similarly for column-based BD,

$K$8

This representation is exact rather than approximate, provided the chosen basis rows or columns are independent. The paper’s Theorem 3.1 states that an $K$9 random matrix whose entries are drawn from a distribution absolutely continuous with respect to Lebesgue measure is full rank with probability $V$0. In the intended setting, this motivates a GPU-friendly choice: contiguous first-$V$1 or last-$V$2 rows or columns can be used as the basis without expensive pivoting or SVD (Zhao, 2 Oct 2025).

BD also changes storage and reconstruction cost. For $V$3 of rank $V$4, full storage costs $V$5 parameters, ordinary low-rank storage costs $V$6, and BD storage costs

$V$7

A plausible implication is that BD is especially attractive when rank is structurally fixed and much smaller than ambient dimension, because the subtraction of the overlap term $V$8 makes BD more compact than standard two-factor low-rank storage (Zhao, 2 Oct 2025).

3. Reformulating multi-head attention

In standard multi-head attention, with input $V$9, number of heads $O$0, and head dimension $O$1 such that $O$2,

$O$3

with $O$4 and $O$5. Writing the projections per head,

$O$6

reveals two head-local products: $O$7 Each is a product of a $O$8 matrix and a $O$9 matrix, so each has rank at most $d$0. BDA exploits precisely this built-in low-rank structure (Zhao, 2 Oct 2025).

For the query–key path, BDA applies column-based BD to each head’s

$d$1

obtaining basis blocks $d$2 and coefficients $d$3, so that

$d$4

For the value–output path, it applies row-based BD to

$d$5

obtaining $d$6 and $d$7, so that

$d$8

The online inference path then avoids reconstructing the original $d$9 products. Aggregating headwise BD parameters, the computation is

$n$0

$n$1

$n$2

followed by ordinary headwise attention and

$n$3

The key equivalence claim is that, for every head $n$4,

$n$5

and likewise the value–output mapping is preserved. Consequently, the overall MHA layer mapping $n$6 is unchanged under exact arithmetic (Zhao, 2 Oct 2025).

4. Complexity, storage, and measured acceleration

The principal savings arise because BDA leaves the softmax attention kernel unchanged but reduces cost in the projection sublayers, especially $n$7 and $n$8. In standard MHA, the four projections together cost about $n$9 multiplies, ignoring constants. In BDA, the $d_h$0 and output projections remain of the same order, but the $d_h$1 and $d_h$2 projections become

$d_h$3

rather than $d_h$4. The ratio is

$d_h$5

For the DeepSeek-V2/V3 configuration reported in the paper, $d_h$6 and $d_h$7, so $d_h$8. This yields a theoretical $d_h$9 arithmetic reduction for those projection paths and a theoretical $r$0 speedup for the $r$1 operator (Zhao, 2 Oct 2025).

The empirical results reported on DeepSeek-V2-Lite (16B) on an NVIDIA A6000 closely match this analysis.

Quantity	Reported result	Context
Offline preparation	$r$2–$r$3 s	Whole model conversion
K/V projection speed	$r$4 faster	Abstract summary
$r$5 measured speedup	$r$6 FP16, $r$7 BF16	A6000 GPU
Single attention operator throughput	$r$8–$r$9	Sequence lengths $$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$0–$$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$1
Attention-weight reduction	$$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$2	DeepSeek configuration

Beyond arithmetic reduction, BDA also changes parameter storage. Because the coefficient matrices for $$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$3 and $$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$4 are $$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$5 rather than full $$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$6, the projection weights shrink by about $$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$7 in the cited DeepSeek setup. The paper further attributes the practical speedups to shared basis alignment across heads and to a fused Triton kernel for the $$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$8 path, which combines slicing, repeating, matrix multiplication, and addition into a single GPU kernel (Zhao, 2 Oct 2025).

5. Exactness, numerical behavior, and empirical validation

BDA is “lossless” in a strict algebraic sense only under exact arithmetic. In finite precision, the basis coefficients are obtained by linear solves, and the reformulated computation is therefore subject to floating-point noise. The paper measures this noise directly and finds very small reconstruction errors for the headwise $$\mathbf{W} = \mathbf{U}\mathbf{V}^{\top}, \qquad \mathbf{U}\in\mathbb{R}^{m\times r},\; \mathbf{V}^{\top}\in\mathbb{R}^{r\times n},\; r<\min(m,n).$$9 and $\mathbf{U}$0 products. Averaged over all heads and layers in DeepSeek-V2-Lite, the reported residual-min reconstruction MSEs are $\mathbf{U}$1 for QK and $\mathbf{U}$2 for VO in FP32, and $\mathbf{U}$3 for QK and $\mathbf{U}$4 for VO in FP16 (Zhao, 2 Oct 2025).

These residuals translate into negligible end-to-end perplexity changes on WikiText-2. The paper reports that DeepSeek-V2-Lite moves from $\mathbf{U}$5 to $\mathbf{U}$6 in FP32, corresponding to a $\mathbf{U}$7 relative increase, and from $\mathbf{U}$8 to $\mathbf{U}$9 in FP16, corresponding to a $K$00 relative increase. The abstract summarizes the same effect as a $K$01 increase in FP16 or $K$02 in FP32, characterizing the change as negligible (Zhao, 2 Oct 2025).

The reported evidence also suggests that BDA is compatible with training, although the method is primarily framed as an inference-time reformulation. On IWSLT’14 En–De with a standard Transformer, BLEU scores are described as essentially on par with, or slightly better than, ordinary MHA across learning-rate scales $K$03, $K$04, $K$05, and $K$06, without hyperparameter retuning (Zhao, 2 Oct 2025). A plausible interpretation is that exact forward equivalence in real arithmetic is sufficient to preserve optimization quality in practice even when gradient paths are no longer identical.

A second empirical result concerns composability with low-rank pruning. On LLaMA2-7B and 13B in FP16, applying BD on top of low-rank weights improves throughput and reduces memory relative to the already-compressed low-rank baselines, while leaving perplexity unchanged or negligibly changed. For example, in the 7B case without KV cache, throughput rises from $K$07 to $K$08 M tokens/s and memory falls from $K$09 GB to $K$10 GB; with KV cache, throughput rises from $K$11 to $K$12 and perplexity remains $K$13 (Zhao, 2 Oct 2025). This suggests that BD is not merely a stand-alone alternative to compression, but also a secondary optimization layer that can exploit existing low-rank structure.

6. Practical integration, constraints, and terminology

The deployment workflow is explicitly post-hoc. One loads a pre-trained model, computes per-head $K$14 and $K$15 products, applies column-based BD to the former and row-based BD to the latter, selects between first-$K$16 and last-$K$17 bases using the residual-min strategy, stores $K$18, $K$19, $K$20, and $K$21, and replaces the original attention layer with a BD-parameterized implementation (Zhao, 2 Oct 2025). The preparation step takes only seconds for a model of DeepSeek-V2-Lite scale.

The method is architecture-agnostic in the sense intended by the paper, but its benefits are configuration-dependent. When $K$22, the ratio $K$23 is small and the K/V savings are material. When $K$24, the benefit shrinks. Exactness also interacts with positional encoding. If positional information is injected only at the embedding layer, BDA remains fully exact. With vanilla RoPE, exactness for QK generally breaks because the effective interaction becomes $K$25. The paper notes that DeepSeek uses Decoupled RoPE, allowing BD to be applied exactly to non-RoPE QK channels and to all VO channels; for LLaMA models with vanilla RoPE, BD is exactly lossless for VO and only approximate for QK, though the numerical errors remain small (Zhao, 2 Oct 2025).

BDA is explicitly presented as complementary to FlashAttention rather than a replacement for it. FlashAttention reduces I/O and kernel overhead in the attention-score and softmax path, whereas BDA reduces arithmetic and parameters in the projection path (Zhao, 2 Oct 2025, Dao et al., 2022). The same complementarity is claimed for pruning and quantization. This suggests a layered optimization strategy in which algebraic reformulation, kernel engineering, and model compression operate on different bottlenecks rather than competing for the same one.

Finally, the term itself requires disambiguation. “BDA” has established, unrelated meanings in transfer learning and geometric perception: “Balanced Distribution Adaptation” is a feature-based unsupervised domain adaptation method that balances marginal and conditional distribution alignment (Wang et al., 2018), and “Bezier Deformable Attention” is a curve-structured cross-attention mechanism for lane-centerline and road-topology decoding (Kalfaoglu et al., 2024). In current LLM/VLM systems literature, however, “BD Attention” specifically denotes the Basis Decomposition reformulation of multi-head attention introduced as a theoretically exact acceleration method (Zhao, 2 Oct 2025).