VitaLLM: A Versatile and Tiny Accelerator for Mixed-Precision LLM Inference on Edge Devices

Abstract: We present VitaLLM, a mixed precision accelerator that enables ternary weight LLMs to run efficiently on edge devices. The design combines two compute cores, a multiplier free TINT core for ternary-INT projections and a BoothFlex core that reuses a radix-4 Booth datapath for both INT8$\times$INT8 attention and ternary-INT-sustaining utilization without duplicating arrays. A predictive sparse attention mechanism employs a leading-one (LO) surrogate with a comparison-free top-$K$ selector to prune key/value (KV) fetches by roughly $1-K/M$ for $M$ cached tokens, confining exact attention to $K$ candidates. System-level integration uses head-level pipelining and an absmax-based quantization barrier to standardize cross-core interfaces and overlap nonlinear reductions with linear tiles. A 16 nm silicon prototype at 1 GHz/0.8 V achieves 72.46 tokens/s in decode and 0.88 s prefill (64 tokens) within 0.214 mm^2 and 120 KB on-chip memory, while reducing KV traffic and improving utilization in ablations. These results demonstrate practical BitNet b1.58 (3B) inference on edge-class platforms and provide a compact blueprint for future mixed-precision LLM accelerators.

• The paper introduces VitaLLM, an edge accelerator that leverages mixed-precision computing and dual-core integration to efficiently support large LLM inference on constrained devices.
• The paper details a dual-core architecture combining the multiplier-free TINT core and the flexible BoothFlex core, achieving over 38% throughput gains with precise KV-cache reduction techniques.
• The paper demonstrates that employing predictive sparse attention and quantization barriers significantly reduces memory bottlenecks, evidencing a 54.86× reduction in KV-cache traffic.

VitaLLM: A Compact Mixed-Precision Accelerator for Efficient Edge LLM Inference

Introduction

VitaLLM introduces an edge-centric LLM accelerator optimized for mixed-precision inference, specifically targeting highly compressed models like BitNet b1.58, which utilizes ternary weights and INT8 activations. The architecture directly addresses bottlenecks in hardware utilization across varying datatypes, inefficient KV-cache bandwidth management, and heterogeneous computational interfaces encountered in previous ternary LLM ASIC/FPGA accelerators. VitaLLM combines a dual-core compute system with predictive sparse attention and seamless cross-core operation. The design yields competitive throughput/area metrics in silicon, confirmed by rigorous ablation studies.

BitNet b1.58 Model Support

VitaLLM’s compute fabric is designed to optimally support BitNet b1.58, enabling a memory/computation bottleneck up to $7\times$ smaller than full-precision models, yet maintaining competitive accuracy at the 3B-parameter scale. BitNet replaces every weight projection in Transformer layers with BitLinear, i.e., ternary-INT operations in attention and feedforward submodules.

Figure 1: BitNet b1.58 architecture with BitLinear replacing all weight projection layers.

Compute Core Microarchitecture

VitaLLM integrates two specialized compute cores sharing output-stationary dataflow and buffer infrastructure, enhancing hardware utilization and minimizing redundant traffic.

TINT Core: The TINT core is a multiplier-free, $8{\times}8$ PE array dedicated to all ternary-INT projections. Each PE performs a select-accumulate based on a ternary weight’s sign, eschewing general-purpose multipliers for a sign-controlled, mux-based selector. This architecture achieves high throughput for Q/K/V projections and all linear layers, emitting outputs in fixed tiles while maintaining local accumulator state.

Figure 2: TINT core: $8\times8$ multiplier-free PE array for ternary–INT with output-stationary mapping and tiled outputs.

BoothFlex Core: BoothFlex employs a shared radix-4 Booth pipeline, flexibly executing both INT8$\times$INT8 attention and ternary-INT multiplications using bit-serial, output-stationary scheduling. This design efficiently amortizes computation cycles, avoiding area overhead from duplicative arrays. BoothFlex enables single-cycle ternary-INT and five-cycle INT8 operations within the same PE.

Figure 3: BoothFlex core: shared radix-4 Booth array supporting ternary–INT (single iteration) and INT8×INT8 (five iterations).

Predictive Sparse Attention and Memory Reduction

To mitigate random-access pressure and arithmetic redundancy in conventional attention/KV caching, VitaLLM implements a LOP-driven predictive sparse attention mechanism. Instead of full dot products for all cached keys, a computationally efficient, multiplier-free surrogate (based on leading-one and sign information) is applied to filter candidate keys. A comparison-free top-$K$ selector outputs only the indices with highest correlations, restricting subsequent exact INT8 attention to this reduced set without retraining or altering model weights.

Figure 4: LOP array with a comparison-free top-K selector: streaming bucketized scores yield K indices without pairwise comparison.

This design reduces average KV fetches and compute by a factor proportional to $K/M$ (for $M$ cached tokens), as validated by substantial decreases in KV bandwidth and improved attention throughput in silicon measurements.

Head-Level Pipelining and System Scheduling

VitaLLM implements head-level pipelining, whereby each attention head is processed in a staggered fashion. As soon as TINT completes Q/K/V projections for head $h$, BoothFlex immediately computes MHA for that head, while TINT moves to $h+1$. This scheduling eliminates the need for large-scale intermediate buffering and enables BoothFlex to operate at high utilization each cycle.

Figure 5: Head-level pipelining: TINT finishes Q/K/V for head $h$ and BoothFlex computes attention; TINT advances to $8{\times}8$0.

After MHA, both cores jointly execute the final projections and FFN, alternating between ternary–INT and INT8$8{\times}8$1INT8 as required. This tight coupling, enabled by a unified OS buffer and dispatcher, unlocks several throughput and area advantages.

Quantization-Barrier Interface

To facilitate efficient interfacing between cores and support nonlinear operations (e.g., softmax, RMSNorm), VitaLLM employs an absmax-based quantization barrier. Reductions (such as sums, max, etc.) are overlapped with tile emission so that, upon vector completion, quantization is applied once, providing a standard integer-vector, single-scale interface. This decouples core scheduling from nonlinearity implementation and avoids complicated SRAM banking or feedback paths.

Figure 6: Absmax-based quantization barrier: reductions overlap with linear tile emission, and quantization is issued once per vector.

Overall Architectural Integration

The complete VitaLLM system comprises triple TINT cores, a single BoothFlex core, a lightweight LOP/filtering unit, and an on-chip buffer system orchestrated by a low-overhead dispatcher. Weights and KV cache data reside off-chip (DDR), but effective bandwidth is drastically reduced by selective candidate gating. Both compute and nonlinear units synchronize via standard quantized streams, maintaining a tightly coupled streaming architecture.

Figure 7: Overall architecture. TINT and BoothFlex arrays share on-chip buffers; LOP gates KV fetch with top-K indices, nonlinear unit implements softmax/RMSNorm.

Experimental Results and Ablations

Implemented in TSMC 16nm at 1 GHz/0.8V, VitaLLM’s silicon prototype achieves 72.46 tokens/s decoding speed and 0.88 s prefill time for 64 tokens using 0.214 mm² area and 120 KB on-chip memory. Static power consumption is 59.12 mW for full operation.

Ablations demonstrate:

• LOP-based Pruning: MHA throughput increases by 26.31%, KV-cache traffic reduces by $8{\times}8$2 (Figure 8).
• Head-Level Pipelining: Throughput improves by 54.31% (Figure 9).
• BoothFlex Dual-Mode: PE utilization increases from 0.51% to 69.20%; FFN throughput increases by 25.17% (Figure 9).
• Combined Impact: When all strategies are applied, overall system throughput increases by 38.17% (Figure 9).

Figure 8: Effect of LOP: normalized MHA throughput (higher is better) and KV-cache EMA (lower is better).

Figure 9: Throughput ablations: head-level pipelining (MHA), BoothFlex strategy (FFN), and overall with both enabled.

Implications and Future Directions

VitaLLM demonstrates that properly designed mixed-precision accelerators can achieve high throughput, low power, and minimal area for LLM inference on edge platforms, even at billion-parameter scales. By tightly integrating compute, predictive attention, and quantized interface scheduling, the architecture presents a viable path toward practical deployment of compact LLMs (such as BitNet) in mobile and always-on scenarios.

Future research may focus on:

• Extending the flexible BoothFlex datapath to additional quantization/activation bitwidths.
• Exploring more advanced, possibly learned, sparse attention heuristics for further KV bandwidth pruning.
• Generalizing the cross-core quantization interface for model-agnostic, runtime-adaptive operation.

Conclusion

VitaLLM establishes a solid blueprint for mixed-precision LLM acceleration on edge devices, substantiated by hardware-measured performance gains arising from dual-mode compute, predictive top-K gating, and seamless cross-core interfacing. The approach efficiently supports large ternary LLMs in extremely compact power and area configurations, emphasizing the tangible benefits of hardware-software co-design at the level of model arithmetic, attention scheduling, and system integration for edge AI deployments (2605.00320).