BitROM: CiROM Accelerator for LLMs

• BitROM is a CiROM accelerator that fuses ternary weights with BitNet’s 1.58-bit quantization for reload-free, edge-friendly LLM inference.
• It employs a bidirectional ROM array and a Tri-Mode Local Accumulator to double effective storage and achieve a 10× area efficiency improvement over prior designs.
• The architecture integrates Decode-Refresh eDRAM and LoRA overlays to manage on-die KV-cache and enable adaptable, low-overhead task performance.

Searching arXiv for the BitROM paper and closely related work on BitNet/LLM accelerators. BitROM is a compute-in-read-only-memory (CiROM) accelerator co-designed with BitNet’s 1.58-bit quantization model for reload-free LLM inference at the edge. In CiROM, parameters are permanently fused as ROM cells, so inference does not reload model weights from external memory; only activations and KV-cache data move at runtime. BitROM is presented as the first CiROM-based accelerator aimed at overcoming the area and scalability constraints that have limited CiROM deployment for LLMs, particularly by combining ternary-weight model design, a bidirectional ROM macro, a ternary-specialized local accumulator, and an integrated Decode-Refresh eDRAM for on-die KV-cache management (Zhang et al., 10 Sep 2025).

1. Definition, motivation, and problem formulation

BitROM is motivated by a specific limitation of prior CiROM approaches: although CiROM can eliminate runtime weight updates and therefore avoid the weight-movement bottleneck that dominates many SRAM- and DRAM-based compute-in-memory systems, naïvely mapping LLMs into digital CiROM leads to prohibitive area. The cited example is LLaMA-7B, which in a naïve digital CiROM approach requires more than $1{,}000 \ \text{cm}^2$ of silicon area in 14 nm CMOS. The data attributes this to the combination of large parameter counts, multi-bit weight storage, and conventional adder-tree overhead (Zhang et al., 10 Sep 2025).

The architecture addresses this by co-design rather than by circuit optimization alone. BitROM targets BitNet-b1.58/a4.8 models, uses 1.58b weights and 4b/8b activations, and specializes the compute path for ternary weights $w \in \{-1, 0, +1\}$. This permits both denser weight storage and a simplified arithmetic path in which multiply-accumulate reduces to addition, subtraction, or no-op. The resulting design is fully digital, which the source contrasts with analog CiM and analog CiROM approaches that rely on multi-level storage and ADCs and can be fragile to noise in Transformer workloads. In this framing, “reload-free” means that weights are permanently on-chip and are never fetched from external memory during inference.

The area rationale is explicit. Using prior digital CiROM density normalized to 65 nm, approximately $487 \ \text{Kb/mm}^2$, and assuming 8-bit weights for 7B parameters, the area estimate becomes

$$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$

This is the baseline that BitROM seeks to avoid through ternary storage and shared accumulation structures. A plausible implication is that the central contribution is not merely a denser ROM cell, but a joint circuit–model reformulation of LLM inference for CiROM.

2. Quantization model and ternary computational basis

BitROM is tightly coupled to BitNet’s 1.58-bit representation. The numerical basis is information-theoretic: a three-level code carries $\log_2(3)$ bits of information, with

$\log_2(3) \approx 1.585.$

This is the origin of the “1.58-bit” terminology used for ternary weights (Zhang et al., 10 Sep 2025).

The arithmetic kernel is the ternary dot product

$y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$

Because the weight alphabet contains only sign and zero, multiplication can be replaced by conditional accumulation: $S = S_{+} - S_{-},$ where

$$S_{+} = \sum_{j: w_j = +1} x_j, \qquad S_{-} = \sum_{j: w_j = -1} x_j.$$

Zero weights are skipped. This decomposition is central to BitROM’s local accumulation scheme and to its attempt to exploit weight sparsity.

The activation format is 4-bit by default, with 8-bit activations supported via bit-serial processing in two cycles with shift-and-accumulate. The macro uses ternary decoding with two comparators per weight and reference levels at $1/8 \ V_{DD}$ and $w \in \{-1, 0, +1\}$0 to classify weights into $w \in \{-1, 0, +1\}$1. The data states that BitNet-b1.58 reports perplexity and downstream scores on par with full-precision LLMs, and that zero-shot task results in the BitROM study show that LoRA-adapted BitNet variants improve over base BitNet across tasks with small overhead. This suggests that BitROM’s architectural specialization depends on the empirical viability of extreme weight quantization rather than treating quantization as a secondary compression layer.

3. Bidirectional ROM Array and ternary readout mechanism

BitROM’s principal storage primitive is the Bidirectional ROM Array, or BiROMA. Each ROM cell is a single NMOS transistor whose source and drain are connected across two symmetric sides of the column, denoted even (“E”) and odd (“O”). Three signal rails encode ternary values: ‘0’, ‘+1’, and ‘−1’, driven respectively as $w \in \{-1, 0, +1\}$2, $w \in \{-1, 0, +1\}$3, and $w \in \{-1, 0, +1\}$4. These rails are placed in metal $w \in \{-1, 0, +1\}$5 at minimum pitch to maximize density (Zhang et al., 10 Sep 2025).

The key architectural idea is bidirectional readout. Because the E and O sides are symmetric and independently drivable, the same transistor can encode one ternary weight for the $w \in \{-1, 0, +1\}$6 read direction and another ternary weight for the $w \in \{-1, 0, +1\}$7 direction. In operation, one side is configured as source lines (SL) to supply the ternary-coded voltage, while the other side serves as bitlines (BL) to sense it. When the wordline (WL) is asserted, the BL discharges toward the SL-coded level through the conducting device; changing direction accesses the second stored weight. The effect is two ternary weights per transistor.

The read sequence is specified as follows: precharge and equalize BL with digit equalizers (DEQ), configure one side as SL and drive one of $w \in \{-1, 0, +1\}$8, assert WL, sense using two comparators referenced at $w \in \{-1, 0, +1\}$9 and $487 \ \text{Kb/mm}^2$0, then deassert WL and advance via the column selector (CS). The comparator outputs serve distinct roles. The MSB comparator, referenced against $487 \ \text{Kb/mm}^2$1, detects zero versus non-zero and directly gates the TriMLA enable signal, implementing zero-skipping. The LSB comparator, referenced against $487 \ \text{Kb/mm}^2$2, distinguishes addition from subtraction.

This mechanism yields the reported density gain. Storing two ternary weights per transistor doubles effective storage relative to a single-direction ROM cell, contributing to a measured bit density of $487 \ \text{Kb/mm}^2$3. Together with a shared global adder tree, this underpins the reported $487 \ \text{Kb/mm}^2$4 improvement in area efficiency over prior digital CiROM designs.

4. Tri-Mode Local Accumulator and array-level organization

Computation in BitROM is organized around the Tri-Mode Local Accumulator, or TriMLA, which is specialized for ternary-weight processing. TriMLA maintains separate running sums for positive and negative contributions, skips zeros, and computes the local result as $487 \ \text{Kb/mm}^2$5. The MSB comparator output disables accumulation when $487 \ \text{Kb/mm}^2$6, while the LSB selects add versus subtract (Zhang et al., 10 Sep 2025).

The attachment granularity is explicit: one TriMLA is connected to a group of eight BiROMA columns. TriMLA sequentially processes channels to build local partial sums. After all channels for a vector are processed locally, a single shared global adder tree aggregates TriMLA outputs in one shot. The dataflow therefore proceeds from WL/CS selection and ternary voltage driving, through BL sensing and local ternary accumulation, to one final reduction stage shared across the array.

This local-then-global accumulation strategy is presented as beneficial in three ways. First, it enables zero-skipping close to the bitcells. Second, it reduces switching activity and off-array traffic. Third, it avoids the area cost of many distributed adder trees. In the post-layout macro, TriMLA plus periphery plus the global adder tree occupy only $487 \ \text{Kb/mm}^2$7 of total macro area, and 8-bit output width is described as sufficient empirically to avoid overflow.

At the macro and system level, the reported circuit organization is a BiROMA of $487 \ \text{Kb/mm}^2$8 rows by $487 \ \text{Kb/mm}^2$9 columns, with one TriMLA per eight columns and a shared global adder tree across the entire array. The implementation is in TSMC 65 nm CMOS with $$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$0 supplies. For detailed system evaluation, the target model is Falcon3-1B with 18 Transformer layers and grouped-query attention using 4 KV heads. The design is partitioned into six independent macro partitions, each handling three layers and up to six input batches, and a six-stage batch-level pipeline is used to keep all partitions utilized. This suggests that throughput optimization is pursued through layer partitioning and pipeline occupancy rather than through weight streaming.

5. Decode-Refresh eDRAM and LoRA overlay for adaptability

BitROM integrates an on-die eDRAM for KV-cache management during decoding. The target access pattern is the autoregressive decode phase, in which each new token’s KV vectors are written once and then read at every subsequent step. The earliest tokens are therefore read most frequently and dominate DRAM bandwidth (Zhang et al., 10 Sep 2025).

The architecture exploits the fact that eDRAM naturally refreshes on read. The read/refresh sequence is WL open, sense amplifier activate, and WL close; any read refreshes the row. BitROM’s “Decode-Refresh” eDRAM leverages the observation that the Token-Between-Token interval is small and in practice much less than DRAM $$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$1, for example 64 ms, so entries read every step do not require explicit refresh scheduling. In the traffic model for sequence length $$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$2, total decode reads are proportional to

$$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$3

If $$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$4 early tokens are buffered on-die, the reduction fraction is approximated by

$$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$5

For the reported example $$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$6 and $$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$7, the model gives approximately $$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$8, and the measured reduction is $$56 \times 10^9 \text{ bits} \,/\, (0.487 \times 10^6 \text{ bits/mm}^2) \approx 114{,}989 \ \text{mm}^2 \approx 1{,}149.9 \ \text{cm}^2.$$9 external DRAM reads. For Falcon3-1B with grouped-query attention and sequence length 128, BitROM provisions $\log_2(3)$0 eDRAM to buffer the 32 earliest tokens.

Adaptability is provided not by reprogrammable ROM weights but by LoRA-based adapters. The inference-time weight is

$\log_2(3)$1

where $\log_2(3)$2, $\log_2(3)$3, $\log_2(3)$4 is rank, and $\log_2(3)$5 is a scaling factor. BitROM realizes the adapter path with small digital MAC units, described as 4-input multiplier-and-adder units, per relevant projection. The ROM path computes $\log_2(3)$6, the adapter path computes $\log_2(3)$7 in parallel, and the results are summed digitally.

The reported configuration uses LoRA rank $\log_2(3)$8, applied only to the Value and Output projections in attention and the Down projection in the MLP. LoRA weights are quantized to 6 bits, while activations remain 8 bits. The parameter overhead is approximately $\log_2(3)$9, depending on model size, and the additional operations are only approximately $\log_2(3) \approx 1.585.$0 of the corresponding projection layers. This is presented as enabling task flexibility without violating the reload-free CiROM design.

6. Quantitative results, scalability, caveats, and interpretation

In 65 nm CMOS, BitROM reports $\log_2(3) \approx 1.585.$1 in 1.58b/4b mode and $\log_2(3) \approx 1.585.$2 for 8-bit activation mode, with a bit density of $\log_2(3) \approx 1.585.$3. The architecture is described as offering a $\log_2(3) \approx 1.585.$4 improvement in area efficiency over prior digital CiROM and a $\log_2(3) \approx 1.585.$5 reduction in external DRAM access during decode when the DR eDRAM buffers 32 early tokens at sequence length 128 (Zhang et al., 10 Sep 2025).

On benchmark behavior, the study reports zero-shot tasks on WikiText-2, PTB, SQuAD, Gigaword, and DROP, with consistent improvements from LoRA adapters over base BitNet models. The explicit example provided is Falcon3-7B on SQuAD, where the gains relative to base are EM $\log_2(3) \approx 1.585.$6 and F1 $\log_2(3) \approx 1.585.$7. Perplexity improvements on WikiText-2 and PTB are also reported across sizes.

The reported comparisons require careful interpretation. Analog CiROMs may achieve higher raw TOPS/W in some cases, but the data states that they lack robustness for LLMs, whereas BitROM is digital and error-free for Transformer inference. Relative to prior digital CiROM, BitROM’s bit density is approximately $\log_2(3) \approx 1.585.$8 higher, $\log_2(3) \approx 1.585.$9 versus $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$0, and this density is a central enabler of LLM-scale mapping. Relative to BitNet-optimized digital ASICs such as Slim-Llama, the data notes that higher TOPS/W may be reported elsewhere, but BitROM eliminates weight-load overhead and reduces DRAM traffic through DR eDRAM, which improves system-level efficiency.

Scalability is presented in two distinct senses, and conflating them is a common source of misunderstanding. Using the reported density and $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$1 bits per parameter, the effective ROM bitcell area is approximated by

$y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$2

This yields about $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$3 for $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$4 parameters and about $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$5 for $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$6 parameters. However, the source explicitly cautions that these figures reflect ROM bitcell density only, not full-chip area. Actual accelerator area must include TriMLA, drivers, the global adder tree, control, interconnect, and eDRAM. In 14 nm CMOS for Falcon3-1B, the measured implementation occupies $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$7 for ROM macros and $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$8 for on-die DR eDRAM.

The limitations stated for BitROM follow directly from this specialization. The architecture assumes ternary weights and 4b/8b activations, so the accumulation path is specialized rather than general-purpose. The data notes that extreme quantization can be model-family dependent. The DR eDRAM strategy targets early-token buffering for short to moderate sequences, specified as 32–256, and very long sequences or batch-heavy server workloads may require larger eDRAM or hierarchical KV management. eDRAM retention also requires $y_i = \sum_j w_j x_j, \quad w_j \in \{-1, 0, +1\}.$9, so worst-case timing and temperature-dependent retention require guard-banding. Future work identified in the source includes broader adapter placement, structured sparsity in weights and activations, and expanded support for diverse Transformer variants.

Taken together, these details define BitROM as a digital CiROM architecture whose significance lies in the co-design of model quantization, ROM storage, local ternary accumulation, and decode-aware KV buffering. The design does not merely compress an LLM into ROM; it redefines the memory–compute interface around the constraints of ternary BitNet inference and the asymmetry between prefill and decode.