BitROM CiROM: Scalable AI Inference

• BitROM CiROM architectures are defined by immutable ROM cells that perform in-memory matrix operations, significantly reducing energy and area costs.
• They employ advanced designs such as BiROMA and TriMLA to enable efficient ternary quantization and scalable, high-density computation for AI workloads.
• Architectural extensions including ReBranch, on-die caching, and LoRA adapters further enhance system flexibility and performance for CNN and LLM inference.

BitROM CiROM (Compute-in-Read-Only-Memory) architectures refer to a category of specialized computing-in-memory (CiM) designs that exploit high-density, read-only memory arrays for matrix-vector multiplication (MVM), with the principal aim of reducing energy and area costs associated with the storage and movement of neural network weights. BitROM and its successors, including YOLoC and the advanced LLM-oriented BitROM, leverage both ROM device physics and circuit/algorithmic co-design to achieve scalable, energy-efficient, and high-capacity AI inference engines that address the core limitations of conventional SRAM-CiM and weight reload bottlenecks (Chen et al., 2022, Kaiser et al., 2023, Zhang et al., 10 Sep 2025).

1. Foundational BitROM Cell and Crossbar Macro

At the circuit level, the prototypical BitROM cell is a single NMOS transistor whose gate is irreversibly tied—at fabrication—to either the wordline (WL) or ground, representing a binary “1” or “0” weight respectively. No write capability is available post-fabrication; thus, data (weights) are immutable, a property that distinguishes BitROM from SRAM- or eDRAM-based CiM. In operation, computation is performed in situ: an activation bit $a_i \in \{0, 1\}$ is applied to the WL, and multiplication ($w_i a_i$) reduces to a logical AND; enabled cells discharge the bitline (BL), and the total BL voltage drop accumulates the columnwise dot-product. This analog sum is digitized by a column ADC, yielding multi-bit partial sums for further processing (Chen et al., 2022).

Macros are arranged in crossbar architectures (e.g., $128 \times 256$ arrays), with groupwise ADC sharing (e.g., 5-bit ADC per 16 BLs). Weight mapping for multi-bit quantization is performed on a bit-plane basis; for $b$-bit weights, $b$ physical ROM planes are required. Peripheral circuits manage pre-charging, input serialization, and accumulation, while post-ADC logic reconstructs the full-precision MAC result.

2. Advanced ROM Array Designs and Ternary Quantization

The BitROM concept was further extended to address the demands of billion-parameter LLMs by storing ternary weights ($w \in \{-1, 0, +1\}$) with maximized density. The Bidirectional ROM Array (BiROMA) implements two ternary weights per transistor via bidirectional access: each nMOS source/drain terminal is prewired to one of three voltage rails (0, $0.25V_{DD}$, $0.5V_{DD}$). Depending on read direction, each transistor emits a separate ternary value, and two independent weights are extracted per cell. Comparator thresholds are set at $1/8 V_{DD}$ and $3/8 V_{DD}$, discriminating between ternary states with $w_i a_i$0 effective bits per weight consistent with BitNet quantization. This results in raw bit densities up to $w_i a_i$1 kb/mm$w_i a_i$2 and area reductions of $w_i a_i$3 versus previous digital CiROMs (Zhang et al., 10 Sep 2025).

3. Peripheral Circuits: Accumulation, Sensing, and Dataflow

ROM-CiM systems require dedicated periphery for analog-to-digital conversion, serialization, partial summation, and global accumulation:

• Sense amplifiers and ADCs convert BL voltage to digital partial sums, with one ADC typically shared across multiple columns for area optimization.
• Local shift-and-add units, as in Tri-Mode Local Accumulators (TriMLA), implement ternary accumulation: zero-skip gating ($w_i a_i$4), addition ($w_i a_i$5), or subtraction ($w_i a_i$6) on a per-channel basis. Hardware comparators separate weights and select arithmetic operations in real time.
• For high-throughput scalability, local accumulation is further aggregated in multiway adder trees before final stage pooling or nonlinear activation.

Latency and energy models are explicitly characterized as functions of precharge, wordline drive, comparator operation, and accumulation time. Example per-MAC figures: $w_i a_i$7, $w_i a_i$8 (TriMLA, 65 nm) (Zhang et al., 10 Sep 2025).

4. Architectural Extensions: ReBranch, On-Die Caching, and LoRA Adapters

The immutability of ROM-based storage is fundamentally offset by three key mechanisms:

• Residual Branch (ReBranch): For transfer learning or adaptation, a lightweight SRAM-CiM "^^^^0^^^^" operates in parallel with the ROM-CiM "^^^^1^^^^" The ^^^^0^^^^ employs a compressed–residual–decompressed sequence using additional pointwise and small conv layers, requiring only $w_i a_i$9 the parameters of a full convolution. Incoming activations are broadcast to both engines, outputs are merged via on-chip adders, and a controller synchronizes sequencing and post-processing (Chen et al., 2022).
• Decode-Refresh eDRAM (KV-Cache Management): For LLM inference, BitROM integrates a small on-die eDRAM—organized as a cache for the key/value (KV) projections generated during decoding. Caching the first $128 \times 256$0 tokens on-die yields a $128 \times 256$1 reduction in external DRAM traffic, substantially lowering energy and DRAM bandwidth requirements (Zhang et al., 10 Sep 2025).
• LoRA Adapters: Hardware LoRA integration is achieved by augmenting the TriMLA infrastructure with small multiplier–adder units for low-rank projection matrices $128 \times 256$2 and $128 \times 256$3. LoRA updates, quantized to 6 bits, are co-processed with core inference, resulting in only $128 \times 256$4 area/power overhead and negligible latency increase ($128 \times 256$5), while enabling rapid domain transfer and state-of-the-art accuracy benchmarks.

5. Dual-Context ROM-Augmented SRAM Architectures

BitROM/CiROM architectures can also be realized as ROM-augmented SRAM (notably 8T-SRAM). By VT-programming the isolated read port transistors (LVT↔ROM "1", HVT↔ROM "0"), a single bank can act as pure ROM, pure RAM, or support true dual-context (simultaneous RAM/ROM read). Sense margin, delay, and energy are analytically modeled; area penalties are minimal, with 1.3× density gain over separate 8T-SRAM + ROM, and only $128 \times 256$6–$128 \times 256$7 extra metal routing. Monte-Carlo simulations demonstrate $128 \times 256$8 yield and robust sense separation. Typical read delays are $128 \times 256$9–$b$0 baseline 8T SRAM delays, with $b$1–$b$2 dynamic energy overhead (Kaiser et al., 2023).

Potential use cases include on-chip LUTs for mathematical functions, in-memory neural inference (weights in ROM, activations in RAM), and context-switching in security/IoT accelerators without reconfiguration overhead.

6. Area, Energy, and System-Level Trade-Offs

The primary system-level advantage of BitROM/CiROM lies in memory density and weight reload elimination:

• Cell/array density: BitROM achieves $b$3 higher density than 6T-SRAM-CiM ($b$4/bit vs. $b$5/bit at 28 nm); BiROMA at 65 nm achieves $b$6 kb/mm$b$7, a $b$8 improvement over prior architectures (Chen et al., 2022, Zhang et al., 10 Sep 2025).
• Throughput/energy: YOLoC demonstrates $b$9 TOPS/W ($b$0 GOPS/mm$b$1); BitROM achieves $b$2 TOPS/W with system energy savings enhanced by local cache and no runtime weight reloading.
• Application-level performance: On object detection (YOLO/ResNet-18), YOLoC attains $b$3–$b$4 energy savings versus SRAM-CiM with $b$5 latency overhead and $b$6 mean average precision (mAP) difference (Chen et al., 2022).
• LLM scalability: BitROM enables practical LLM inference in advanced nodes; e.g., LLaMA-7B fits in $b$7 cm$b$8 (vs. $b$9 cm$w \in \{-1, 0, +1\}$0 without density enhancement).

Process variation and ADC sharing present ongoing design challenges, with mitigation schemes including calibration, ECC, and controlled ADC column multiplexing.

7. Applications and Future Directions

BitROM/CiROM architectures support a broad range of workloads:

• Large-scale CNNs and LLMs where weight reload elimination and density are critical.
• Embedded and edge AI where area and power budgets preclude DRAM-intensive architectures.
• Multi-context compute engines (e.g., shared RAM/ROM compute for on-chip functions, security, and IoT).

Emerging work in BiROMA and TriMLA demonstrates further integration of algorithmic–circuit co-design (e.g., BitNet quantization, LoRA hardware), suggesting that BitROM/CiROM frameworks will remain key to scaling efficient, flexible inference processors for contemporary and future machine learning workloads (Chen et al., 2022, Kaiser et al., 2023, Zhang et al., 10 Sep 2025).