---
title: 'ShiftAddLLM: Efficient Inference via Shift-and-Add'
url: https://www.emergentmind.com/topics/shiftaddllm
type: topic
---

# ShiftAddLLM: Efficient Inference via Shift-and-Add

ShiftAddLLM is a post-training reparameterization framework for large language models (LLMs) that eliminates dense multiplications in model inference by replacing them with hardware-efficient shift-and-add primitives. This approach enables efficient deployment of pretrained LLMs on resource-constrained hardware while maintaining competitive perplexity and accuracy, significantly reducing the memory footprint and energy consumption of inference. ShiftAddLLM achieves these gains through binary quantization of weight matrices, power-of-two (PoT) scaling, multi-objective optimization over quantization and activation error, and a global mixed-bit allocation strategy constrained by an overall bit budget [2406.05981].

## 1. Weight Matrix Reparameterization via Binary Coding and PoT Scaling

ShiftAddLLM reparameterizes each full-precision weight matrix $W \in \mathbb{R}^{m \times n}$ using a sum of $q$ binary matrices $\{b_i \in \{-1, +1\}^{m \times n}\}$ weighted by group-wise scaling factors $\{\alpha_i\}$:
$$
W_q = \sum_{i=1}^q \alpha_i b_i \approx W,
$$
where the quantization seeks to minimize the Frobenius norm $\|W - W_q\|_F^2$.

The quantization is performed via the Alternating Binary Code Quantization (BCQ) algorithm, which employs greedy residual minimization:
- At each step $i$, solve $\min_{\alpha_i, b_i} \|r_{i-1} - \alpha_i b_i\|^2$ where $r_{i-1} = W - \sum_{j < i} \alpha_j b_j$.
- The optimal $b_i$ is $\mathrm{sign}(r_{i-1})$, and $\alpha_i = r_{i-1} \cdot b_i / \|\mathbf{1}\|$.
- $\{\alpha_i\}$ are further optimized via least squares with fixed binaries, followed by alternating updates.

To facilitate hardware efficiency, each $\alpha_i$ is quantized via a greedy additive PoT decomposition:
$$
\alpha_k = \operatorname{POT}(r_{k-1}), \quad \operatorname{POT}(x) = \operatorname{sign}(x) \cdot 2^{\mathrm{round}(\log_2 |x|)}.
$$
Weight blocks ($8 \times 8$) utilize a shared scaling factor $s_g$, compactly representing $W_q = \sum_{g=1}^G s_g B_g$.

## 2. Multiplication-Free Inference: Shifts, Adds, and Look-Up Tables

Conventional general matrix-matrix multiplications (GEMM), $Y = X W$, are transformed with reparameterized $W_q$ into operations solely composed of shift and add instructions:
$$
Y = X \left( \sum_g s_g B_g \right ) = \sum_g (X s_g) B_g,
$$
where multiplication by $s_g$ (a PoT) reduces to integer bit-shifts on activations $X$, i.e., $X s_g = X \ll p_g$ or $X \gg |p_g|$.

Binary mask multiplication $(B_g)$ is implemented using precomputed $8$-bit lookup tables (LUTs):
- Each shifted activation row $x \in \mathbb{R}^8$ is indexed; $2^8 = 256$ unique patterns.
- LUT entries are computed as $\mathrm{LUT}_g[\mathrm{index}(x)] = \sum_{k=0}^7 B_g[k] x[k]$.
- The result $Y$ is accumulated by querying LUTs and summing outputs across groups.

In attention and MLP sublayers, every multiplication with model weights thus becomes a shift plus a LUT-indexed sum, avoiding floating-point multiplication entirely.

## 3. Joint Weight and Activation-Aware Multi-Objective Optimization

Unlike previous post-training quantization (PTQ) schemes that minimize only weight error $(\mathcal{L}_W = \|W - W_q\|_F^2)$ or output activation error $(\mathcal{L}_A = \|WX - W_q X\|_F^2)$ independently, ShiftAddLLM employs a combined, column-wise (or block-wise) multi-objective loss:
$$
\min_{\{\alpha_{i,j}, b_{i,:,j}\}} \|W_{:,j} - W_{q,:,j}\|^2 + \lambda \|W_{:,j} X - W_{q,:,j} X\|^2,
$$
where $\lambda$ balances the tradeoff, and $X$ represents a batch of sampled activations.

A single pass of alternating BCQ per column (or block) is executed with this hybrid loss for significantly reduced end-to-end distortion.

## 4. Automated Mixed-Bitwidth Allocation under Accuracy-Constrained Bit Budgets

Recognizing heterogenous sensitivity to quantization across transformer layers, ShiftAddLLM introduces a global bit allocation framework:
- Sensitivity analysis measures per-layer/block reconstruction error at 2/3/4 bits; attention Q/K and deeper layers are consistently most sensitive.
- For each layer $i$, the importance score $\mathrm{IS}_i$ is defined as $W_i[\mathrm{diag}(\mathrm{cholesky}((X_i X_i^T)^{-1}))]^{-1}$, leading to a cost metric
  $$
  C_i = \|\mathrm{IS}_i\|_F \times [\mathrm{STD}(\mathrm{IS}_i)]^2,
  $$
  estimating the effect of aggressive quantization.
- The discrete bit assignment $\beta_{i,b}$ is solved via integer programming:
  $$
  \min_{\{\beta_{i,b}\}} \sum_{i=1}^L \sum_{b=2}^4 \beta_{i,b} C_{i,b},\quad
  \sum_{b=2}^4 \beta_{i,b} = 1\,\,\forall\, i,\quad
  \sum_{i=1}^L \sum_{b=2}^4 \beta_{i,b} b \le B_\text{avg} L,
  $$
  yielding per-layer mixed-bit allocation that maximizes global accuracy under a fixed average bit-width.

## 5. Experimental Results across Architectures, Workloads, and Hardware Metrics

Empirical validation includes five LLM families: OPT (125M–66B), LLaMA-1/2/3 (7B–70B), Gemma 2B, Mistral 7B, BLOOM 3/7B, benchmarked on eight tasks (WikiText-2 perplexity, zero-shot task accuracy on ARC-Ch/E, BoolQ, COPA, PIQA, StoryCloze, RTE).

Key results:
- At 3 bits: ShiftAddLLM achieves an average WikiText-2 perplexity (PPL) reduction of 5.6 points versus the strongest 3-bit baselines, matching FP16 accuracy in many cases.
- At 2 bits: achieves ≈22.7 points average PPL gain over the most competitive 2-bit scheme (QuIP), with baselines otherwise failing catastrophically (PPL in thousands).
- Zero-shot task accuracy: yields average gains of 10–14 points over baselines for OPT-66B and LLaMA-2-70B.
- Latency: with block-wise scaling, achieves lower inference latency than LUT-GEMM on A100 GPU (6.3 ms for 125M, 20.9 ms for 13B at 2 bits vs 29.5 ms FP16).
- Resource savings: Energy consumption reduced by 80–90%, GPU memory usage reduced by 80–87% (e.g., OPT-66B: 23 GB at 3 bits vs 122 GB at FP16) [2406.05981].

| Metric             | ShiftAddLLM (2–3 bit) | Best Baseline (2–3 bit) | FP16      |
|--------------------|----------------------|-------------------------|-----------|
| WikiText-2 PPL (3b)| 5.6 pts better       | —                       | —         |
| Zero-shot Accuracy | +10–14 pts           | —                       | —         |
| Latency (ms, 13B)  | 20.9 (2b)            | —                       | 29.5      |
| GPU Memory (66B)   | 23 GB (3b)           | —                       | 122 GB    |
| Energy             | ↓ 80–90%             | —                       | baseline  |

## 6. Methodological Insights, Limitations, and Prospective Directions

The joint optimization of both quantization and activation error in a single post-training pass is crucial for controlling end-to-end distortion; decoupled objectives are unable to achieve comparable accuracy, especially at low bit-widths. Block-wise (8×8) scaling, supported by efficient CUDA kernels, enables significant speedup with a minor accuracy penalty (≈0.2–0.5 PPL), while column-wise scaling achieves maximal quality but lacks correspondingly optimized inference kernels.

Current limitations include the absence of a fast custom kernel for column-wise scaling; closing the accuracy/latency gap will require dedicated kernel development. Promising future research avenues include extending shift-and-add quantization to sparse or Mixture-of-Experts (MoE) transformer layers, investigating mixed-precision PoT representations, and pursuing hardware-in-the-loop fine-tuning of bit or PoT-level allocations.

ShiftAddLLM delivers a hardware-aligned, post-training quantization-reparameterization that converts dense multiplications to efficient shift, add, and LUT primitives, with global mixed-precision allocation—enabling scalable, low-resource inference for pretrained LLMs without retraining [2406.05981].

Source: https://www.emergentmind.com/topics/shiftaddllm