---
title: 'Batched LUT-GEMV: Efficient Lookup Matrix-Vector Multiply'
url: https://www.emergentmind.com/topics/batched-lut-based-general-matrix-vector-multiplication-lut-gemv
type: topic
---

# Batched LUT-GEMV: Efficient Lookup Matrix-Vector Multiply

Searching arXiv for papers on LUT-based GEMV, LUT-GEMM, and related FPGA/CPU/ASIC approaches.
arxiv_search:
{"query":"\"Batched LUT-based General Matrix-Vector Multiplication\" OR LUT-GEMV arXiv", "max_results": 10}
Batched LUT-based General Matrix-Vector Multiplication (LUT-GEMV) denotes a family of matrix-vector execution schemes in which multiply-accumulate is replaced, wholly or partly, by lookup-table access and accumulation. In the explicit terminology of SAIL, it is a CPU-oriented, SRAM-based processing-in-memory GEMV primitive that builds lookup tables from weight tiles and reuses them across multiple input vectors in a batch [2509.25853]. Closely related literature studies the same computational pattern under other names, including an FPGA “Approximate Multiplication Unit (AMU)” derived from MADDNESS, LUT-based inner-product decoding for nested-lattice quantization, GPU “LUT-GEMM” for weight-only quantized large language models, LUT-based mixed-precision GEMM tensor cores, and LUT-based ASIC mpGEMM for low-bit and ternary weights [2407.02362, 2505.13164, 2206.09557, 2408.06003, 2511.21910]. Across these variants, the common structure is to encode or quantize operands, index a precomputed table of partial products or partial sums, and accumulate the retrieved values.

## 1. Core computational principle

The defining transformation in LUT-GEMV is the replacement of direct multiplication by a sequence of table-oriented steps. In the MADDNESS-derived FPGA formulation, an input vector is divided into $C$ sub-vectors, each sub-vector is encoded by a learned decision tree, and the approximate dot product is reconstructed by summing one entry from each codebook-specific lookup table:
$$
\mathbf{a}^T\mathbf{w} \approx \sum_{c=1}^{C} \mathrm{LUT}_c[\mathrm{enc}(\mathbf{a}^{(c)})].
$$
The same work expresses the matrix product approximation as
$$
\tilde{f}(\mathbf{A},\mathbf{W})=\mathbf{A}\times \mathbf{W}+\mathbf{E},
$$
with small error $\mathbf{E}$, and frames the fully connected layer output as
$$
s_j=(\mathbf{W}^T\mathbf{a}_{in})_j
$$
before replacing the exact dot product by encoding, lookup, and accumulation [2407.02362].

In SAIL, LUT-GEMV is explained through low-bit activation patterns and weight-driven table construction. For three weights $W_0,W_1,W_2$ and three 4-bit activations $A,B,C$, the target expression
$$
AW_0 + BW_1 + CW_2
$$
is converted into LUT lookup, shift, and add. If the number of basis weights is $\mathrm{NBW}=3$, the LUT contains $2^{\mathrm{NBW}} = 8$ entries, one for every possible bit pattern of the $\mathrm{NBW}$ input bits. Runtime execution scans activation bits from LSB to MSB, uses each bit pattern to fetch a precomputed sum such as $W_2$, $W_0$, or $W_0+W_1+W_2$, left-shifts the fetched value according to bit position, and accumulates it [2509.25853].

A third formulation appears in hierarchical nested-lattice quantization, where the primitive is inner-product decoding over quantized codewords rather than direct low-bit arithmetic. If
$$
\hat{x}=\sum_{i=0}^{M-1} q^i \hat{x}_i,\qquad \hat{y}=\sum_{j=0}^{M-1} q^j \hat{y}_j,
$$
then
$$
\hat{x}^\top \hat{y} = \sum_{i=0}^{M-1}\sum_{j=0}^{M-1} q^{i+j}\,L(b_i(x),b_j(y)).
$$
Here a single LUT for the base code is reused across layers and across many blocks of a vector, so matrix products can be reconstructed from repeated small-table accesses rather than de-quantization followed by arithmetic [2505.13164].

These formulations differ in whether the approximation enters through quantization, codebook encoding, or weight-only low-bit representation. They nevertheless share a single operational kernel: table address generation followed by accumulation of precomputed values.

## 2. Principal formulations and related systems

The current literature does not present a single universal LUT-GEMV design. Instead, several closely related formulations implement the same basic pattern for different numerical regimes and hardware targets.

| System | Core lookup object | Reuse or batching mode |
|---|---|---|
| AMU | Prototype-weight partial dot products | Parallel codebooks and output packages [2407.02362] |
| Hierarchical nested-lattice quantization | Pairwise codeword inner products | Reuse of one small LUT across many chunks and queries [2505.13164] |
| LUT-GEMM | Partial dot products for binary-coded sub-vectors | Shared-memory LUT reuse in single-batch generation [2206.09557] |
| LUT Tensor Core | Low-bit weight-group dot products | Elongated tiling with large reuse dimension $N$ [2408.06003] |
| SAIL | Weight-subset sums indexed by activation bit patterns | LUT construction amortized across batched inputs [2509.25853] |
| Platinum | Chunk-level dot products indexed by weight codes | One LUT reused across many rows and multiple output columns [2511.21910] |

The FPGA AMU is explicitly an approximate matrix multiplication unit rather than a general exact GEMM engine. It reorganizes MADDNESS into a pipelined hardware structure that no longer performs elementwise multiply-accumulate operations, and it is presented as an alternative to the MVAU in FINN-style pipelines [2407.02362]. By contrast, hierarchical nested-lattice decoding is a quantization-centric construction in which quantized vectors are multiplied through LUT queries over codewords, with the central obstacle being the memory cost of storing all pairwise inner products at high rate [2505.13164].

GPU-oriented LUT-GEMM addresses weight-only quantized LLM inference. It reformulates quantized matrix multiplication as
$$
\mathbf{y} = \sum_{i=1}^{q}\left(\mathbf{\alpha}_i \circ (\mathbf{B}_i \cdot \mathbf{x})\right),
$$
uses a sub-vector length hyperparameter $\mu$, precomputes all corresponding partial dot products for each sub-vector of $\mathbf{x}$, and stores them in shared-memory LUTs [2206.09557]. LUT Tensor Core and Platinum extend the same general idea to mixed-precision GEMM and integer or ternary-weight mpGEMM, respectively, but emphasize co-designed hardware paths, tiling, and construction-path optimization [2408.06003, 2511.21910].

A plausible implication is that “batched LUT-GEMV” is best understood as a workload class rather than a single algorithm. The explicit SAIL terminology identifies one concrete member of that class, while the surrounding literature supplies several adjacent implementations that differ in exactness, quantization model, and table-generation strategy.

## 3. Batching, reuse, and table construction

Batching is the central mechanism that makes LUT-based matrix-vector execution attractive beyond isolated dot products. In SAIL, the key idea is that once a LUT is built for a given weight tile, it can be reused across multiple input vectors in the batch. The ISA extension therefore centers on a tiled GEMV of size
$$
[1,1024] \times [1024,1024],
$$
with larger matrix-vector products realized by chaining these tiles. The paper further describes tensor-level scheduling in which weights for one layer are loaded into the LLC once and then reused across different users’ requests in the batch before moving on [2509.25853].

AMU implements an FPGA-specific version of batching through structured packaging and pruning. The original MADDNESS formulation still produces a full output feature map and therefore still pays a cost proportional to the full matrix dimensions. AMU argues that this is wasteful for cascaded neural-network layers because subsequent layers only need the split-index values used for encoding. It therefore prunes away unused feature-map elements, preserves only the split-index values needed by the next stage, and reorganizes the output into packages so that the next AMU can consume them directly. The resulting compression ratios are written as
$$
\mathrm{Input\ Compression\ Ratio}_i = \frac{I_i N_i}{U_{i-1}}, \qquad
\mathrm{Output\ Compression\ Ratio}_i = \frac{O_i M_i}{U_i}, \qquad
\mathrm{LUTsize}_i = N_i 2^{I_i},
$$
with the main argument being that once $I,N,O,M$ are fixed, the AMU sees a fixed amount of useful input and produces a fixed amount of useful output, independent of the original large matrix dimensions [2407.02362].

Hierarchical nested-lattice quantization approaches batching from the memory side. A standard single-layer Voronoi code requires a pairwise inner-product LUT of size $2^{2dR}$, which is practical only when $dR$ is small, roughly $dR\le 8$–$9$ for modern CPUs. The hierarchical scheme splits the rate $R$ into $M$ layers so that the per-layer LUT shrinks to
$$
2^{2dR/M},
$$
while the full inner product is reconstructed from $M^2$ lookups. The paper identifies the tradeoff succinctly as “smaller LUT, more lookups,” and then connects this directly to batched queries by splitting an $n$-dimensional vector into $K$ blocks and reusing the same small LUT for every block product [2505.13164].

Other systems implement analogous reuse in different forms. LUT-GEMM stores LUTs in GPU shared memory and relies on shared LUT reuse inside thread blocks during single-batch autoregressive generation [2206.09557]. LUT Tensor Core uses operator fusion to reduce precompute overhead to about 2.5% of execution time in its reported setups, table symmetrization to halve table size from $2^K$ to $2^{K-1}$, and an elongated tile shape that maximizes table reuse along a large $N$ dimension; its chosen optimal configuration is $M2N64K4$ [2408.06003]. Platinum constructs one LUT per input chunk and reuses it across many weight rows and multiple output columns simultaneously, with the final design using $n_{\text{cols}}=8$ and $L=52$ processing elements [2511.21910].

## 4. Architectural realizations

The hardware embodiments of LUT-GEMV differ sharply in where the LUT resides and how the address-generation path is organized. AMU is organized into three pipelined blocks—Allocator, Encoder, and Aggregator. The allocator unpacks the pruned input package and distributes $N$ blocks to $N$ parallel encoders. Each encoder performs the tree-based prototype search using only the split-index values and corresponding split thresholds, with successive comparisons accumulated into the address bits. After all encoders finish, the aggregator reads one partial product from each relevant LUT column and sums them to generate the output package. The aggregation step is expressed as
$$
\text{partial result} = \sum_{c=1}^{N} \mathrm{LUT}_c[\mathrm{ID}_c],
$$
followed by bias addition if needed and quantized NN post-processing by thresholding [2407.02362].

SAIL places LUT-GEMV inside SRAM by using compute-capable SRAM blocks, or C-SRAMs, adjacent to LLC slices. Its execution flow brings DRAM data into the LLC, has the Data Feeding Module fetch tensor fragments from the data cache, builds LUTs locally from the weight rows in C-SRAM, broadcasts activation bits to the C-SRAMs, performs the LUT-based partial GEMV, aggregates partial sums, and sends the result to the CPU for dequantization or further conversion. The architecture requires a single new RISC-V instruction for LUT-based tiled GEMV and about 2% overall system hardware overhead, while maintaining dual functionality as both compute and storage units [2509.25853].

GPU realizations emphasize software-hardware co-design rather than standalone PIM structures. LUT-GEMM uses shared-memory LUT storage, assigns LUTs to thread blocks, and accumulates partial outputs across thread blocks via `atomicAdd`; the paper states that only 1 KB per 8 hidden dimensions is needed and that A100 shared memory is sufficient to store all LUTs for large hidden sizes [2206.09557]. LUT Tensor Core instead defines a custom LMMA instruction family,
$$
\text{lmma.}\{M\}\{N\}\{K\}.\{A_{\text{dtype}}\}\{W_{\text{dtype}}\}\{Accum_{\text{dtype}}\}\{O_{\text{dtype}}\},
$$
and builds a compilation stack on TVM, Roller, and Welder, with graph rewriting, operator fusion, LUT scheduling, and code generation targeted to LUT-based mpGEMM [2408.06003].

ASIC realizations push the same pattern toward specialized low-bit execution. Platinum consists of processing elements, aggregators, on-chip SRAM buffers, and special functional units. Runtime LUT construction follows a 4-stage pipeline—fetch build-path entry, read source LUT and input element, add or subtract, and write back the constructed entry—while the build path itself is generated offline as a minimum spanning tree over a construction graph. The paper presents this offline path generation as the key mechanism that removes runtime shortcut selection logic and increases the processing-element budget under the same chip area [2511.21910].

## 5. Quantitative behavior and performance regimes

Reported performance gains are highly dependent on workload structure, precision regime, and memory hierarchy. On a 100 MHz ZCU104, AMU reports that for a classification layer it can achieve about $4\times$ higher throughput and $4\times$ higher energy efficiency when problem size grows from MNIST-scale $(64,10)$ to CIFAR10-scale $(512,10)$, while the MVAU gains about $1.1\times$ throughput and loses about 40% efficiency. For the full SFC accelerator, the reported gains reach up to $9\times$ throughput and $112\times$ energy efficiency over FINN-generated MVAU-based baselines at comparable task complexity. The same work also reports that a fully partitioned AMU can reach $\mathrm{II}=1$ and that its energy efficiency is substantially better than grouped-memory versions, at the cost of increased LUT usage [2407.02362].

In SAIL, the Pattern-Aware LUT Access mechanism exploits the observation that about 17% of input activation patterns repeat within computation batches. Each Data Feeding Module contains a 32-entry fully-associative Pattern Reuse Table holding a 32-bit hash of the input pattern and the previous LUT result; on a hit, the design bypasses the C-SRAM access and directly reuses the stored result. The reported benefit is a 13.8% reduction in computation cycles. At the system level, evaluations on a modified gem5 simulator report up to 10.7x speedup and 19.9x higher tokens per dollar compared to ARM Neoverse-N1 CPU baselines, and up to 7.04x better cost efficiency than NVIDIA V100 GPUs. For the Q4 GEMV kernel, the full LUT plus in-memory type-conversion system reaches 3.81x speedup over the ARM baseline [2509.25853].

For nested-lattice quantization, the key quantitative result is not raw throughput but retention of quantization quality under smaller tables. With base lattice $D_4$, dimension $d=4$, and hierarchy depth $M=2$, the hierarchical scheme is reported to be strictly better than the lower-rate Voronoi bound and almost as good as the full Voronoi code with $r=q^M$, with distortion still within about $1/2$ bit of the Shannon limit. In the inner-product experiment with dimension $n=512$, chunk size $d=4$, and $K=128$ chunks, hierarchical quantization is reported to be very close to Voronoi coding at the same rate while reducing the LUT-size requirement from $2^{2dR}$ to $2^{2dR/M}$ [2505.13164].

LLM-oriented GPU and ASIC results show similarly strong but workload-specific improvements. LUT-GEMM states that MatMul accounts for at least 75% of processing time across model sizes and input lengths during generation, claims about 2.6x reduction in computation compared to previous GEMM kernels in the evaluated setting, and reports that for OPT-175B with 3-bit quantization it achieves 2.1x lower token generation latency than OPTQ on a single GPU [2206.09557]. LUT Tensor Core reports over an order-of-magnitude improvement in both compute density and energy efficiency, up to 8.2x end-to-end speedup in some LLM scenarios, up to 20.9x compute density improvement, and up to 11.2x energy-efficiency improvement; in a fair comparison table it reports up to 6.93x speedup while using only 38.3% of original Tensor Core area [2408.06003]. Platinum reports, on BitNet b1.58-3B, up to 73.6x, 4.09x, and 2.15x speedups over SpikingEyeriss, Prosperity, and 16-thread T-MAC, respectively, along with energy reductions of 32.4x, 3.23x, and 20.9x, all within a 0.96 mm$^2$ chip area [2511.21910].

These results do not support a single monotone conclusion such as “larger batch is always better.” LUT-GEMM explicitly states that it primarily focuses on single-batch inference and exhibits diminishing performance gains as the batch size increases, whereas SAIL and Platinum derive much of their advantage precisely from amortizing table construction across batched inputs or reused rows and columns [2206.09557, 2509.25853, 2511.21910].

## 6. Limits, misconceptions, and active design tensions

A common misconception is that LUT-GEMV is simply an exact GEMV engine with a different microarchitecture. The literature is more heterogeneous. AMU is explicitly approximate, depends on learned decision trees and codebooks that are assumed valid at inference time, and is not presented as a general-purpose numerical GEMM accelerator [2407.02362]. Hierarchical nested-lattice decoding and weight-only LUT kernels are exact only with respect to their quantized representations; the principal approximation is inherited from quantization rather than from the table lookup itself [2505.13164, 2206.09557].

A second misconception is that lookup tables remove the memory problem unconditionally. In fact, table size often grows exponentially with grouping parameters. Standard nested-lattice inner-product decoding requires $2^{2dR}$ entries, which is why the hierarchical construction splits the rate across layers [2505.13164]. LUT Tensor Core identifies the same pressure in the dependence on $K$ and therefore settles on a small group size, with its software and hardware optimizations organized around table symmetrization, table quantization, and elongated tiling [2408.06003]. Platinum’s design choices likewise reflect the fact that LUT-based methods remain memory-heavy and that larger $n_{\text{cols}}$ improves throughput only up to a point; beyond 8, area efficiency diminishes [2511.21910].

A third design tension concerns preprocessing and integration. AMU’s efficiency relies on structured layer packaging, pruning, and reorganization of feature maps for cascaded AMUs, so it is not a drop-in replacement for arbitrary dense matrices without preprocessing [2407.02362]. SAIL reduces arithmetic and data movement in CPU-based LLM inference, but it does so by adding a new instruction, C-SRAM structures, a Data Feeding Module, and an in-memory type-conversion pipeline [2509.25853]. LUT-GEMM, although highly relevant to GEMV-like autoregressive decoding, does not present an explicit batched GEMV algorithm or multi-query batch scheduling strategy and warns that gains diminish with larger batch sizes [2206.09557].

The broader trajectory of the field suggests two persistent directions. One is more aggressive co-design between quantization, compilation, and memory hierarchy, as seen in LMMA-based compilation, fused precompute, and pattern-aware table reuse [2408.06003, 2509.25853]. The other is sharper specialization by operand structure, exemplified by AMU’s codebook-addressed approximation and Platinum’s separate bit-serial and ternary execution paths with offline-generated construction schedules [2407.02362, 2511.21910]. This suggests that batched LUT-GEMV is unlikely to converge on a single canonical implementation; rather, it is emerging as a family of table-driven linear-algebra kernels whose efficiency depends on matching the lookup scheme to the numerical format, the batch-reuse opportunity, and the memory system in which the tables reside.

Source: https://www.emergentmind.com/topics/batched-lut-based-general-matrix-vector-multiplication-lut-gemv