TeLLMe: Edge LLM Ternary FPGA Accelerator

• TeLLMe is a ternary large language model accelerator that exploits 1.58-bit weights and 8-bit activations to achieve low-power, high-throughput inference on edge FPGAs.
• It utilizes a table-lookup matrix multiplication engine that precomputes activation group combinations, reducing resource usage and aligning with FPGA LUT fabric.
• By architecturally separating compute-heavy prefill from memory-bound decode, TeLLMe delivers up to 16.4× performance improvements over prior FPGA baselines while operating under sub-7W power.

Searching arXiv for TeLLMe and closely related edge-FPGA ternary LLM accelerator work. TeLLMe is a ternary LLM accelerator for low-power edge FPGAs that is designed to support both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations. It was introduced to address a specific deployment gap: recent low-bit quantization methods such as BitNet and DeepSeek compress model weights to ternary values with minimal accuracy loss, yet edge deployment remains constrained by limited on-chip resources, tight power budgets, and the often-neglected latency of the prefill phase. On platforms such as the AMD Kria KV260, TeLLMe is presented as an architecture specialized for FPGA-native execution of ternary LLM inference under a sub-7W envelope, rather than as an adaptation of conventional FP16 or INT8 datapaths (Qiao et al., 22 Apr 2025).

1. Design objectives and problem setting

TeLLMe targets edge-class inference for generative transformers, where matrix multiplication, attention, KV-cache access, normalization, softmax, and quantization-dequantization must all execute within severe LUT, BRAM, URAM, DSP, bandwidth, and power constraints. The central design claim is that low-bit compression by itself does not make edge deployment straightforward: even when weights are reduced to 1.58 bits, accelerator efficiency still depends on how ternary linear algebra, attention scheduling, and auxiliary functional units are realized in hardware.

A defining feature of the design is its explicit support for both prefill and decode. This distinction is not incidental. The prefill stage processes the whole prompt, is compute-heavy, and inherits the quadratic attention cost of causal transformers. The decode stage is memory-bound, dominated by matrix-vector-style operations and KV-cache traffic. TeLLMe therefore treats prefill and decode as architecturally distinct phases rather than forcing them through a uniform execution path.

The paper presents this as a corrective to prior FPGA work, which often focused only on decoding or did not fully exploit the computational implications of ternary weights. In that framing, TeLLMe is described as the first edge FPGA accelerator to fully support ternary LLM inference in both prefill and decode.

2. Ternary representation and computational model

TeLLMe operates in the BitNet-1.58 or DeepSeek-style quantization regime, with weights constrained to the ternary alphabet

$W \in \{-1,0,+1\}.$

This is termed “1.58-bit” because $\log_2 3 \approx 1.585$. Activations remain at 8-bit precision. The immediate arithmetic implication is that multiplication by weights reduces to add, subtract, or skip. In the paper’s formulation, ternary matrix multiplication is treated as a simplification of mixed-precision bitwise matmul:

${\bf A} \otimes {\bf W} = {\bf A} \otimes {\bf W}_{0}, \quad {\bf W}_{\text{ternary} \in \{-1,0,1\}.$

The architecture is built around this asymmetry between ultra-low-bit weights and higher-precision activations. Rather than preserving conventional multiplier-centric pipelines, TeLLMe restructures linear layers around lookup and accumulation. This is important because, on FPGAs, LUT fabric can be exploited directly as a computational resource, whereas a naïve ternary realization based on explicit add/subtract selection may consume more logic than a table-based implementation.

The paper also emphasizes that ternary deployment does not eliminate the need for full-model support functions. RMSNorm, Absmax quantization, dequantization, and SiLU remain integral to inference, and their hardware treatment strongly affects end-to-end throughput and bandwidth.

3. Table-lookup ternary matrix multiplication

The most distinctive component of TeLLMe is its table-lookup-based ternary matrix multiplication engine. The core idea is to group activations into blocks of size $G$ and precompute the results of all possible ternary combinations for each activation group. For the example highlighted in the paper, $G=3$, so a group produces

$3^G = 27$

possible combinations, addressable with about 5 bits. Weights are offline-preprocessed into packed index vectors, and online computation proceeds in two phases: construction of a lookup table from grouped activations, followed by indexed retrieval and accumulation.

The lookup procedure is intended to minimize resource use while preserving throughput. Instead of duplicating sign-selection logic for every output channel, TeLLMe reuses precomputed activation-group tables across many outputs. The paper attributes much of the design’s FPGA suitability to this choice: distributed RAM and LUT fabric can sustain heavily parallel table access, and the accumulation phase can be fully pipelined with an initiation interval of one cycle because the lookup iterations have no inter-iteration dependencies.

Parallelism is increased through multiple lookup tables and concurrent index-vector processing. Activation tiles are streamed into the engine, weights are stored in packed index form, and URAM is used to buffer matmul weights in a ping-pong arrangement. The result is a LUT-heavy but multiplier-light design. The reported LUT counts for three matmul implementations were 52,094 for the TeLLMe table-lookup design, 59,999 for a naïve implementation, and 61,303 for a partial-storage approach, corresponding to savings of 7,905 LUTs and 9,209 LUTs respectively relative to those alternatives (Qiao et al., 22 Apr 2025).

This behavior is also analytically significant. A common intuition is that ternary weights should be easiest to implement via explicit pass, negate, or zero logic. TeLLMe rejects that assumption for the edge-FPGA setting. Its results suggest that a full lookup-table formulation is better aligned with LUT-dominated fabrics, particularly when parallel access and grouped activation reuse are prioritized.

4. Attention scheduling and fused low-bit functional units

TeLLMe’s treatment of attention is motivated by the observation that prefill is frequently the neglected bottleneck in edge LLM accelerators. A naïve causal-attention schedule loads $Q$, $K$, and $V$ in straightforward order, but this wastes bandwidth and computes interactions that are later masked out. TeLLMe therefore introduces a reversed reorder schedule for prefill attention: instead of beginning from $\mathbf{q}_1$, it starts from $\log_2 3 \approx 1.585$0 and works backward so that computation aligns more closely with the causal mask.

The paper compares three scheduling regimes at parallelism level $\log_2 3 \approx 1.585$1. Reverse scheduling is reported to require data block load and iteration count of

$\log_2 3 \approx 1.585$2

whereas naïve scheduling requires data block load of $\log_2 3 \approx 1.585$3 and iteration count of $\log_2 3 \approx 1.585$4. Dense scheduling keeps bandwidth near constant but still performs unnecessary work for LLM causality. TeLLMe’s reverse schedule is therefore presented as a bandwidth-efficient method for practical prefill.

The attention module is further described as a fused FlashAttention-like computation, treated as a special case of FlashAttention v2 with block size 1. Instead of materializing the full attention matrix, the accelerator maintains online softmax state through the score $\log_2 3 \approx 1.585$5, running maximum $\log_2 3 \approx 1.585$6, denominator $\log_2 3 \approx 1.585$7, and output numerator $\log_2 3 \approx 1.585$8. This reduces memory traffic and limits buffering to the quantities needed for streaming accumulation.

Outside attention, TeLLMe includes tightly integrated support for RMSNorm, Absmax quantization, dequantization, and SiLU. RMSNorm is expressed as

$\log_2 3 \approx 1.585$9

after which the input is scaled by a learned parameter. Since both RMSNorm and Absmax quantization require two passes, TeLLMe fuses them into two optimized hardware passes rather than four logical traversals. Dequantization is fused into the linear-output pipeline, and SiLU is pipelined into the preceding linear unit. These choices reflect the paper’s broader argument that special function units must be fused wherever possible to control bandwidth and intermediate storage (Qiao et al., 22 Apr 2025).

5. Hardware organization, implementation, and reported performance

The implemented design combines four main subsystems: table-lookup ternary matmul for both prefill and decode, reverse-reorder fused attention for prefill, a unified decoding attention and LM-head reuse path, and fused normalization, quantization, and activation units. Weights are stored offline in packed index form, activation tiles are streamed from memory, URAM buffers matmul weights, BRAM supports attention and intermediate state, and the top-level AXI interface uses 256-bit transfers.

Decode is treated differently from prefill. Because it is memory-bound, TeLLMe reuses the decode attention unit for the LM head, on the grounds that both have matrix-vector characteristics with limited reuse. Prefill, by contrast, is provisioned with higher parallelism and a more deeply fused pipeline because it is compute-bound.

The reported implementation uses Vitis HLS and Vivado 2023.1 on an AMD Kria KV260 with Zynq UltraScale+ XCK26 MPSoC at 250 MHz. The paper evaluates a 0.7B BitNet-style model and reports up to 9.51 tokens/s, more than 9 tokens/s at 512-token context, about 8 tokens/s at 1024-token context, and prefill latency of 0.55–1.15 s for prompts of 64–128 tokens. Power is reported at 6.72 W, within the stated sub-7W budget.

A comparison against prior FPGA baselines situates TeLLMe as follows:

System	Throughput	Platform
SECDA	0.58 tokens/s	PYNQ
LlamaF	1.50 tokens/s	ZCU102
Li et al.	4.90 tokens/s	KV260
TeLLMe	9.51 tokens/s	KV260

The paper characterizes this as up to about 16.4× improvement over a prior FPGA baseline. Reported total resource utilization is 206 BRAM, 356 DSP, 155,838 FF, 108,994 LUT, and 48 URAM, corresponding to about 71%, 28%, 66%, 93%, and 75% utilization respectively. The design is therefore explicitly LUT-heavy, which is consistent with its reliance on table-lookup matmul rather than arithmetic multipliers (Qiao et al., 22 Apr 2025).

6. Limitations, interpretation, and subsequent development

TeLLMe’s advantages are tied to a specific deployment regime. The paper implicitly identifies several constraints: decode throughput remains limited by the external memory bandwidth of the KV260; the architecture is optimized for ternary weights and 8-bit activations rather than for higher-precision models; LUT utilization is already near fabric limits; and much of the accelerator’s advantage depends on the ternary model assumption. These are not incidental engineering details but part of the design’s scope. TeLLMe is not a general-purpose transformer accelerator; it is a specialized hardware realization of a particular low-bit inference strategy.

A plausible implication is that TeLLMe’s significance lies less in raw throughput than in architectural validation. It shows that practical edge LLM inference on a low-power FPGA becomes more plausible when prefill is treated as a first-class target, when decode and prefill are separated architecturally, and when ternary arithmetic is mapped onto FPGA-native lookup structures rather than emulated as reduced-precision conventional compute.

The subsequent “TeLLMe v2” work retains the same core direction while extending it into a more explicitly end-to-end accelerator. It adds a table-lookup-based ternary matrix multiplication engine labeled TLMM, a fine-grained analytic URAM-based weight buffer management scheme, a streaming dataflow architecture that fuses floating-point element-wise operations with linear computations, reversed-reordered prefill attention, and a specialized decoding-stage attention. On the same KV260-class target, TeLLMe v2 reports up to 25 tokens/s decoding throughput and 0.45–0.96 s time-to-first-token for 64–128 token prompts under a 5 W power budget, indicating that the original TeLLMe architecture became the basis for a continuing line of edge-FPGA ternary LLM research (Qiao et al., 3 Oct 2025).