VitaLLM: A Versatile, Ultra-Compact Ternary LLM Accelerator with Dependency-Aware Scheduling

Abstract: Deploying LLMs on resource-constrained edge devices faces critical bottlenecks in memory bandwidth and power consumption. While ternary quantization (e.g., BitNet b1.58) significantly reduces model size, its direct deployment on general-purpose hardware is hindered by workload imbalance, bandwidth-bound decoding, and strict data dependencies. To address these challenges, we propose \textbf{VitaLLM}, a hardware-software co-designed accelerator tailored for efficient ternary LLM inference. We introduce a heterogeneous \textbf{Dual-Core Compute Strategy} that synergizes specialized TINT-Cores for massive ternary projections with a unified BoothFlex-Core for mixed-precision attention, ensuring high utilization across both compute-bound prefill and bandwidth-bound decode stages. Furthermore, we develop a \textbf{Leading One Prediction (LOP)} mechanism to prune redundant Key-Value (KV) cache fetches and a \textbf{Dependency-Aware Scheduling} framework to hide the latency of nonlinear operations. Implemented in TSMC 16nm technology, VitaLLM achieves a decoding throughput of 70.70 tokens/s within an ultra-compact area of 0.223 mm$^2$ and a power consumption of 65.97 mW. The design delivers a superior Figure of Merit (FOM) of 17.4 TOPS/mm$^2$/W, significantly outperforming state-of-the-art accelerators. Finally, we explore an extended bit-serial design (BoothFlex-BS) to demonstrate the architecture's adaptability for precision-agile inference.

• The paper presents VitaLLM, an accelerator that combines a heterogeneous dual-core design and dependency-aware scheduling to efficiently support ternary LLM inference.
• It employs a specialized dataflow with Leading One Prediction to prune redundant memory accesses and balance heterogeneous computational workloads.
• Experimental results demonstrate breakthrough efficiency with 17.4 TOPS/mm²/W, 70.70 tokens/sec decoding, and significantly reduced area and power consumption.

VitaLLM: Architecture and Innovations for Ultra-Compact Ternary LLM Inference

Introduction and Motivation

Deployment of billion-parameter LLMs on edge devices is fraught with resource limitations in memory, bandwidth, and power. Ternary quantization—specifically BitNet b1.58—enables aggressive compression by reducing weight precision to $\{-1, 0, +1\}$, yet this imposes nontrivial challenges for practical hardware, such as unbalanced computational workloads, bandwidth-bound decoding, and complex nonlinear dependencies. Existing commodity or even some custom hardware platforms are fundamentally mismatched to the ternary operation regime, resulting in underutilization and inefficient dataflow. The VitaLLM accelerator directly addresses these mismatches through an architecture tailored to ternary LLM inference, introducing specialized cores, bandwidth-sparing memory hierarchies, and dependency-aware scheduling.

Architectural Contributions and System Overview

VitaLLM's architectural design adopts three central strategies: (1) a Dual-Core Compute Strategy with heterogeneous cores to match computational heterogeneity; (2) Leading One Prediction (LOP) to prune redundant KV-cache memory accesses during decoding; and (3) Dependency-Aware Scheduling to mitigate stalls from nonlinear and quantization dependencies.

Heterogeneous Dual-Core Design

TINT-Cores perform the primary high-throughput ternary projections, exploiting multiplier-free circuits for maximal area and energy efficiency, while the BoothFlex-Core provides mixed-precision (ternary and INT8) support necessary for attention computation. By reconfiguring BoothFlex-Core to assist in ternary projections during FFN and output stages, hardware utilization remains consistently high across diverse compute phases, directly addressing the pronounced workload imbalance of BitNet b1.58.

Figure 1: Overview of the BitNet b1.58 architecture, where BitLinear modules operate on Ternary$\times$INT8 matrices replacing standard linear projection layers.

Figure 2: Block diagram of VitaLLM, showing integration of TINT-Cores, BoothFlex-Core, a LOP core, and a hierarchical memory system.

Specialized Dataflow and Memory Hierarchy

TINT-Cores employ an output-stationary dataflow for ternary matrix multiplications, caching partial sums locally, which reduces bandwidth and intermediate storage requirements.

Figure 3: TINT-Core architecture with an $8 \times 8$ PE array, employing selector logic instead of multipliers for local sum updates.

Figure 4: Output-stationary dataflow minimizes on-chip data movement within the TINT-Core.

BoothFlex-Core unifies INT8 and ternary projections via Radix-4 Booth encoding and padding, reusing the same logical datapath for both precision regimes.

Figure 5: BoothFlex-Core microarchitecture, supporting both high-precision INT8 and single-cycle ternary operations via shared Booth multipliers.

System Integration: Bandwidth and Dependency Mitigation

Leading One Prediction (LOP) for Sparse Attention

During bandwidth-bound decode stages, attention requires repeated access to the KV cache, often fetching much redundant data. The LOP mechanism predicts top-K relevant tokens for each attention step using a hardware-efficient approximation based on leading one positions, dramatically reducing external memory accesses without significant degradation in perplexity (empirically verified on WikiText-2 and C4).

Figure 6: LOP-Core with ExpAdd units and a Top-K Selector for memory-efficient sparse attention.

Dependency-Aware Scheduling

Nonlinear functions such as Softmax and RMSNorm create hard pipeline barriers due to their reliance on global statistics. VitaLLM resolves this with two complementary schemes:

• Two-stage Nonlinear Operations: First, tile-wise operations are processed immediately (e.g., elementwise exponentials or products); second, deferred scaling is applied post-global reduction, fusing with subsequent quantization when possible. For Softmax, the pipeline is further unblocked using a static upper bound for the global max ("unified max"), proven to preserve accuracy in BitNet scenarios.
• Q-Friendly Two-Level Scheduling: While quantization requires synchronization on the vector-level max, intra-vector operations remain fully pipelined, preventing tile-level stalls.

  Figure 7: Head-Level Pipelining: projections for head $h$ overlap with attention for head $h-1$, eliminating idle time for the mixed-precision engine and reducing on-chip memory demand.

  

Figure 8: Dependency barrier in conventional (naive) nonlinear scheduling.

Figure 9: Quantization barrier illustrating the global maximum dependency.

Hardware Realization and Experimental Results

VitaLLM is implemented in TSMC 16nm technology, achieving:

• Throughput: 70.70 tokens/sec decoding on BitNet b1.58 3B.
• Area: 0.223 $\text{mm}^2$ (SRAM dominates area at 69.2%; compute cores <2.3%).
• Power: 65.97 mW; memory (SRAM) is the principal source of energy consumption due to minimized compute path overhead.
• Figure of Merit: 17.4 TOPS/$\text{mm}^2$/W, exceeding prior state-of-the-art designs by a substantial margin.

  Figure 10: Physical layout in TSMC 16nm, demonstrating ultra-compact core implementation.

  

Figure 11: Area breakdown: on-chip SRAM buffers are the largest contributor.

Figure 12: Throughput plot, highlighting the system's performance across different scheduling and pruning strategies.

Bandwidth Adaptation: The peak bandwidth demand during decode remains within LPDDR5T system constraints (76.8 GB/s), with effective operation possible at much lower bandwidths for reduced throughput, enabling deployment even in low-end platforms.

Ablation and Comparative Studies:

• LOP pruning reduces redundant KV cache reads by nearly two orders of magnitude; throughput improves by 35.7%.
• Head-level pipelining and dual-core FFN scaling deliver 61.74% total throughput improvement over naive designs.
• Against Slim-Llama, TeLLMe, TerEffic, and other leading ASIC/FPGA ternary accelerators, VitaLLM has superior area, power, and FOM, without significant model quality loss.

Figure 13: Attention throughput comparing head-level pipelining and dependency-aware scheduling.

Precision-Agile Extension: BoothFlex-BS

VitaLLM extends adaptability via the BoothFlex-BS core, introducing true bit-serial mixed-precision operation. By decomposing activations into 4-bit nibbles processed in temporal and spatial shifts, arbitrary integer bitwidths (e.g., INT4, INT8, INT16) can be supported without substantial area overhead.

Figure 14: BoothFlex-BS microarchitecture integrates bit-serial MACs and shift-add logic for variable-precision inference.

Relative to BitMoD—a leading bit-serial LLM accelerator—BoothFlex-BS demonstrates >20× area and power advantage, albeit with a throughput penalty (50% reduction for full-system BoothFlex-BS replacement at otherwise negligible area/power increase), making it viable for scenarios where precision agility outweighs peak throughput requirements.

Theoretical and Practical Implications

Theoretical Implications:

VitaLLM’s design demonstrates the efficacy of tightly coupling ternary-quantized algorithmic innovations (e.g., BitNet b1.58) with hardware architectures exploiting the mathematical properties of low-precision weights and activations. The result is a departure from traditional homogeneous compute resources, towards heterogeneity, approximate pipeline-breaking, and aggressive memory bandwidth savings—frameworks now likely to inform broader edge-AI accelerator design.

Practical Implications:

The reduced area and energy requirements position VitaLLM for deployment in highly constrained on-device environments (e.g., smartphones, wearables), sharply lowering the bar for privacy-preserving and low-latency LLM inference outside the cloud. The system-level methodologies (e.g., LOP-based memory pruning, head-level pipelining) are orthogonal to model architecture and could be adopted in INT4, INT8, or sparsely quantized LLMs.

Future Directions

Further research could investigate generalized support for structured sparsity or layerwise adaptive quantization, hierarchical on-device storage with compression-aware dataflow, and dynamic neuro-adaptive scheduling for context-aware optimization. Additionally, holistic cross-stack co-design—spanning model training for hardware-friendliness to runtime adaptation—remains a promising avenue, particularly in scenarios demanding both agility and robustness under variable edge workloads.

Conclusion

VitaLLM delivers a rigorously evaluated, fully integrated hardware-software co-design for BitNet b1.58 inference on the edge. Its innovations—heterogeneous compute strategies, memory bandwidth pruning via LOP, and pipeline optimization for nonlinearity and quantization—yield state-of-the-art efficiency in area, power, and throughput. The extension to precision-agile operation via BoothFlex-BS further confirms the architectural adaptability. These advances collectively push the boundary for practical, high-performance LLM deployment in resource-limited environments.