Bit-Serial Enable Stage Fusion (BESF)

• BESF is a novel approach that unifies the prediction and execution stages of dynamic sparsity attention using bit-serial processing.
• It employs progressive, just-in-time pruning with adaptive token selection (LATS) to dramatically reduce memory traffic and computational workload.
• Experimental results in the BitStopper accelerator demonstrate up to 3.2× speedup and enhanced energy efficiency with minimal impact on LLM quality.

Bit-Serial Enable Stage Fusion (BESF) is an algorithmic and architectural mechanism introduced in the BitStopper accelerator to improve the efficiency of dynamic sparsity (DS) attention in LLMs. BESF fuses the traditional prediction and execution stages of DS attention by interleaving them at bit-level granularity, allowing for progressive, just-in-time pruning of trivial tokens and direct reuse of partial dot-product computations. This enables substantial reductions in memory traffic and computational workload, leading to marked improvements in throughput and energy efficiency over prior Transformer accelerators (Wang et al., 6 Dec 2025).

1. Conventional Dynamic-Sparsity Attention: Structure and Limitations

Traditional DS attention operates using a two-stage workflow:

• Prediction Stage: Computes a low-bitwidth approximation of the attention score matrix $A = QK^T$ using quantized queries ($Q$) and keys ($K$), typically with a 4-bit precision. A selection (e.g., top-$k$ or thresholding) then retains high-scoring $K$ vectors. However, this stage must read the entire $K$ matrix from DRAM, leading to high memory IO and power consumption.
• Execution Stage: Performs high-precision (e.g., INT12) dot-products on the retained $Q$-$K$ pairs to compute attention output $\mathbf{O} = \mathrm{softmax}(A/\sqrt{d_h}) V$. Despite reduced arithmetic, this stage duplicates certain compute and cannot reuse work from the predictor due to decoupling.

Fundamental constraints include excessive predictor IO, a lack of computational reuse between stages, and insufficient adaptivity due to static pruning thresholds.

2. BESF Algorithmic Principle and Pipeline

BESF replaces the two-stage DS pipeline with a single, bit-serial process that unifies prediction and execution:

• Bit-Serial Pruning: Both $Q$ and $K$ are quantized as $N$-bit signed integers. Each $K_j$ is decomposed into $N$ bit-planes:

$$K_j = \sum_{r=0}^{N-1} K_j^{[r]} 2^{N-1-r}, \quad K_j^{[r]} \in \{0,1\}^H$$

• For each query $Q_i$, a running partial inner-product $A_{i,j}^{(r)}$ is formed as the sum over received bit-planes:

$$A_{i,j}^{(r)} = \sum_{\kappa=0}^{r} Q_i \cdot K_j^{[\kappa]} 2^{N-1-\kappa}$$

• After each round $r$, BESF computes for each surviving token:
  • The exact partial sum $A_{i,j}^{(r)}$,
  • An upper-bound margin $M_{i,j}^{(r),\max}$ reflecting the possible contribution of unprocessed lower bit-planes.
  • Any candidate $j$ satisfying $A_{i,j}^{(r)} + M_{i,j}^{(r),\max} \leq \eta_i$ (threshold) is pruned.

No separate predictor is invoked; each dot-product partial result is directly reused for final attention weight computation in surviving tokens.

3. Algorithmic Steps and Pseudocode

The BESF process can be formalized as follows:

1. Precompute bit-margins (for each query $Q_i$): For $r = 0$ to $N - 1$, compute possible min/max dot-product contributions for unknown remaining bits.
2. Initialize survivors: Set all $S$ possible tokens as candidates.
3. Bit-plane rounds ($r = 0$ to $N-1$):
   • For each candidate $j$, fetch $K_j^{[r]}$ and update $A_{i,j}^{(r)}$; compute margin.
   • Determine dynamic pruning threshold $\eta_i$ (see LATS below).
   • Prune any $j$ where $A_{i,j}^{(r)} + M_{i,j}^{(r),\max} \leq \eta_i$.
4. Final computation: For survivors, complete the high-precision dot-product and output attention with $$\mathrm{softmax}(S_i/\sqrt{d_h}) \, V[\text{survivors}]$$.

The above is operationalized in highly parallel hardware. The bit-uncertainty margins are computed based on the process order and the sign of $Q_i$ (Wang et al., 6 Dec 2025).

4. Mathematical Analysis: Complexity and Memory-Access Gains

Analytical complexity for various attention architectures is summarized below:

Attention Type	Compute Operations	Memory Reads
Dense	$O(S^2 H)$	$Q$, $K$ (size $S \times H$), outputs
Two-stage DS	$O(S^2 H_\text{pred}} + S K_\text{ret} H_\text{exec})$	All $K$ for prediction, survivors $K$ for execution
BESF	$H \sum_{r=0}^{N-1} |C_r| \ll H N S$	$\sum_{r=0}^{N-1} |C_r|$ bit-planes of $K$

In practice, BESF achieves a $\sim 60\%$ reduction in both bit-planes fetched and computation, as for optimized $\alpha$, $\sum_r |C_r| \approx 0.4 N S$. BESF keeps average perplexity degradation below 0.1 for LLaMA2-7B (INT12 quant, $4\,\mathrm{K}$ sequence), confirming negligible model quality loss (Wang et al., 6 Dec 2025).

5. Adaptive Token Selection and Early Termination (LATS)

The LATS (Lightweight and Adaptive Token Selection) mechanism complements BESF by dynamically deriving the pruning threshold $\eta_i$ at each bit-plane round:

• The threshold is set as

$$\eta_i = \max_{j} \bigl(A_{i,j}^{(r),\min}\bigr) - \alpha \times \mathrm{radius}, \quad \alpha \in [0,1],\,\mathrm{radius}=5$$

where $A_{i,j}^{(r),\min}$ is a lower bound on score given processed bits.

• This ensures that pruning adapts to the current score distribution, avoiding over- or under-pruning.
• Only tokens whose upper-bounded scores cannot surpass $\eta_i$ are removed, eliminating memory fetches for their remaining bits.

The LATS module operates in tandem with the BESF pipeline, broadcasting $\eta_i$ per round to all processing elements.

6. Hardware Realization and Microarchitectural Features

BitStopper, the hardware implementation of BESF, is partitioned into specialized processing units:

• QK-PU (Query-Key Processing Unit):
  • 32 bit-serial PE lanes, each processing a candidate key.
  • A bit-serial reusable AND-tree (BRAT) for $Q_i \cdot K_j^{[r]}$ operations.
  • Scoreboard (on-chip buffer) and pruning engine coordinating partial sums, margins, and dynamic thresholding.
  • Bit Margin Generator and LATS module for bit-level adaptivity and early token termination.
• V-PU (Value Processing Unit):
  • Lookup-table-based softmax computation and 64× INT12 MAC array for final attention output.

Bit-level asynchronous processing (BAP) enables per-lane, out-of-order fetching for bit-planes, increasing PE utilization from $48\%$ to $83\%$ by overlapping memory access with computation.

Hardware cost: Margin generator plus LATS adds $4.9\%$ area and $6.9\%$ power, while Scoreboard plus pruning engines add $5.8\%$ area and $4.9\%$ power. The total die area is $6.84\,\text{mm}^2$ at 28 nm, 1 GHz. Peak energy efficiency is $11.36\,\text{TOPS/W}$.

7. Quantitative Performance and System-Level Impact

BESF, as realized in BitStopper, demonstrates the following system-wide benefits on LLaMA2-7B (INT12 quant, $4\,\mathrm{K}$ token sequence):

• Memory Access Reduction:
  • $\mathbf{2.9\times}$ (Sanger), $\mathbf{2.8\times}$ (SOFA w/o fine-tune), $\mathbf{2.1\times}$ (SOFA* w/ fine-tune)
• Throughput Gains:
  • $\mathbf{3.20\times}$ (dense baseline), $\mathbf{2.03\times}$ (Sanger DS), $\mathbf{1.89\times}$ (SOFA DS)
• Energy Efficiency:
  • $\mathbf{3.7\times}$ (dense baseline), $\mathbf{2.4\times}$ (Sanger), $\mathbf{2.1\times}$ (SOFA)
• Ablation Breakdown (LLaMA-7B, Dolly, $4\,\mathrm{K}$ tokens):
  • BESF core fusion: $1.25\times$ speedup
  • +BAP (async fetching): $1.63\times$ (increased PE utilization)
  • +LATS (adaptive pruning): total $3.20\times$ speedup
• Model Quality: Perplexity penalty remains below $0.1$ for appropriate $\alpha$, with $\sim 40\%$ of bit-planes retained on average.

In summary, BESF unifies prediction and execution in DS attention, realizes significant memory and computation reductions via bit-serial progressive pruning, and, with LATS and BAP, underpins the efficiency of the BitStopper accelerator. These advances establish a new standard for fine-grained Transformer optimization, delivering $2$–$3\times$ speedups and $2$–$4\times$ energy efficiency improvements with negligible hardware overhead (Wang et al., 6 Dec 2025).