TrilinearCIM: Transformer Accelerator

• TrilinearCIM is a Transformer accelerator architecture that extends conventional bilinear NVM-CIM primitives to an approximately trilinear computation using DG-FeFET technology.
• It remaps attention into three stages that eliminate runtime ferroelectric reprogramming, yielding up to 46.6% energy reduction and 20.4% latency improvement in NLP tasks.
• The design leverages a fixed top-gate weight and dynamic back-gate modulation to perform three-operand MAC operations in-memory, while incurring a 37.3% area overhead and sensitivity in vision workloads.

TrilinearCIM is a Transformer accelerator architecture that extends conventional non-volatile-memory compute-in-memory from a bilinear primitive to an approximately trilinear one by using a Double-Gate FeFET (DG-FeFET). Its stated purpose is to eliminate the runtime ferroelectric reprogramming cycles that conventional FeFET-CIM incurs when evaluating self-attention, where $Q$, $K$, and $V$ are all dynamically generated from the input sequence. The architecture is presented as performing complete Transformer attention computation exclusively in NVM cores without runtime reprogramming, while reporting up to $46.6\%$ energy reduction and $20.4\%$ latency improvement over conventional FeFET CIM at $37.3\%$ area overhead under the reported BERT-base configuration (Mia et al., 8 Apr 2026).

1. Problem formulation and architectural motivation

Conventional NVM-CIM arrays are naturally bilinear. Their basic operation is of the form

$I_j=\sum_i G_{ij}V_i,$

where $G_{ij}$ is a stored conductance and $V_i$ is a runtime input. This is well matched to CNN and MLP inference, where trained weights are static and activations are dynamic. Transformer self-attention is structurally different because its central operands are generated from the input itself: $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$ and attention is

$K$0

Since both $K$1 and $K$2 are dynamic, and the value aggregation phase also uses dynamic $K$3, conventional FeFET-CIM is forced into a compute–write–compute loop in which dynamic intermediates must be reprogrammed into NVM arrays before later matrix multiplications can proceed (Mia et al., 8 Apr 2026).

The paper quantifies this burden for BERT-Base with sequence length $K$4, key dimension $K$5, $K$6 heads, and $K$7 layers, giving an aggregate runtime programming volume of approximately $K$8. This write traffic is compounded by FeFET read/write asymmetry: reads are around $K$9 ns and femtojoules per cell, while writes are around $V$0 ns and sub-pJ per cell. The same runtime reprogramming also stresses finite FeFET endurance. The paper positions prior responses to this problem in four classes: write-reduction approaches such as ReTransformer, hybrid NVM-CIM/CMOS accelerators such as X-Former and iMTransformer, DRAM-PIM approaches such as TransPIM, and analog transient-storage approaches based on capacitors rather than NVM (Mia et al., 8 Apr 2026).

2. DG-FeFET device principle and the trilinear primitive

The hardware mechanism is a DG-FeFET with a ferroelectric top gate and a non-ferroelectric back gate. The top gate stores the non-volatile state and therefore the nominal channel conductance $V$1. The back gate does not store data; it dynamically modulates the channel during inference. In the paper’s linearized model, the conductance becomes

$V$2

with extracted parameters

$V$3

and operation restricted to a conductance band

$V$4

within which a band-averaged constant

$V$5

is used (Mia et al., 8 Apr 2026).

With a drain-source input voltage $V$6, the cell current is modeled as

$V$7

The first term is an unwanted DC offset. The second term is the useful three-factor product of row input $V$8, stored weight $V$9, and back-gate modulator $46.6\%$0. The implementation removes the offset by a reference read at $46.6\%$1 followed by subtraction. In abstract form, the array therefore realizes a three-operand MAC $46.6\%$2, whereas standard analog CIM realizes only a bilinear $46.6\%$3 primitive (Mia et al., 8 Apr 2026).

This device-level reformulation is the defining feature of TrilinearCIM. Its trilinearity is not a tensor-algebra abstraction but a circuit-level multiplication in which a dynamic back-gate operand modulates the stored conductance of an NVM cell during inference.

3. Attention remapping into three trilinear stages

TrilinearCIM remaps Transformer attention into three trilinear stages: $46.6\%$4

$46.6\%$5

$46.6\%$6

These identities are the core algebraic transformation of the architecture. Since $46.6\%$7, one has $46.6\%$8, so $46.6\%$9 is algebraically equivalent to $20.4\%$0. Likewise, since $20.4\%$1, the third stage is algebraically equivalent to $20.4\%$2 (Mia et al., 8 Apr 2026).

The operand-to-terminal mapping is explicit. In Stage 1, the row input is $20.4\%$3, the stored weight is $20.4\%$4, and the back-gate input is the scalar $20.4\%$5. In Stage 2, the row input is $20.4\%$6, the stored weight is $20.4\%$7, and the back-gate input is $20.4\%$8. In Stage 3, the row input is $20.4\%$9, the stored weight is $37.3\%$0, and the back-gate input is $37.3\%$1. This is the mechanism by which the architecture avoids reprogramming $37.3\%$2, $37.3\%$3, or attention scores into ferroelectric states during inference (Mia et al., 8 Apr 2026).

The paper argues that this remapping eliminates explicit storage of $37.3\%$4, $37.3\%$5, and $37.3\%$6 and keeps only the input sequence $37.3\%$7 for reuse and residual paths, lowering buffer capacity requirements by about $37.3\%$8. A plausible implication is that the principal benefit is not only reduced write energy but also a tighter alignment between attention algebra and the physical operand structure of the array.

4. Array organization, peripheral pipeline, and execution model

At array level, the DG-FeFET crossbar is selector-less and relies on FeFET’s high on/off ratio $37.3\%$9 to suppress sneak paths. The architecture supports two trilinear accumulation organizations. Configuration (a), used for Stage 2, realizes

$I_j=\sum_i G_{ij}V_i,$0

with $I_j=\sum_i G_{ij}V_i,$1 stored in conductance, $I_j=\sum_i G_{ij}V_i,$2 applied on rows, and $I_j=\sum_i G_{ij}V_i,$3 applied on columns through the back gate. Configuration (b), used for Stage 3, realizes

$I_j=\sum_i G_{ij}V_i,$4

with $I_j=\sum_i G_{ij}V_i,$5 stored in conductance, $I_j=\sum_i G_{ij}V_i,$6 as row input, and $I_j=\sum_i G_{ij}V_i,$7 broadcast as a back-gate scalar across columns. Stage 1 can use either configuration because its back-gate operand is fixed (Mia et al., 8 Apr 2026).

The hardware hierarchy is chip $I_j=\sum_i G_{ij}V_i,$8 tile $I_j=\sum_i G_{ij}V_i,$9 processing element $G_{ij}$0 subarray. Tiles are arranged in a scalable 2D mesh with H-tree interconnect. There is a global SRAM buffer for $G_{ij}$1, a chip-level accumulation unit, and a Special Function Unit for operations that the NVM cores do not perform efficiently: softmax, layer normalization, and activation. Each PE contains a $G_{ij}$2 array grid. At subarray level, the default simulated configuration uses $G_{ij}$3 DG-FeFET arrays, 8-bit inputs and weights, 2-bit cells, 8-bit ADCs, and 8:1 column muxing, with $G_{ij}$4 nm CMOS peripherals and $G_{ij}$5 nm FeFET memory in BEOL-style integration. Inputs are applied bit-serially from LSB to MSB, and multibit weights are recombined digitally by shift-add (Mia et al., 8 Apr 2026).

The non-linear functions remain digital. Softmax is implemented as a four-stage pipeline consisting of max-reduction, LUT-based exponential, adder-tree sum, and reciprocal-LUT-plus-multiply normalization, with 256-entry LUTs at 8-bit precision. LayerNorm is a two-pass digital pipeline computing mean and variance followed by inverse-square-root LUT and affine transform. GELU uses the approximation

$G_{ij}$6

Accordingly, the claim of “complete attention computation exclusively in NVM cores without runtime reprogramming” refers to the linear algebra of attention, not to an all-analog implementation of every Transformer sub-operation (Mia et al., 8 Apr 2026).

5. Empirical evaluation

The evaluation is performed in TransCIM, built on NeuroSim, with both a quantized digital baseline and a CIM emulation mode. Both use INT8 post-training quantization for inputs and weights. The digital baseline accumulates in FP32 and has no ADC or output quantization, while the CIM mode includes ADC clipping and quantization, hierarchical accumulation, and the DG-FeFET operating-band constraints (Mia et al., 8 Apr 2026).

On BERT-base-uncased, the paper evaluates all nine GLUE tasks and reports that TrilinearCIM exceeds conventional bilinear FeFET-CIM on seven of nine tasks, is identical on one, and underperforms on one.

GLUE task	CIM-Bilinear	CIM-Trilinear
CoLA	42.17	43.70
SST-2	89.72	91.32
MRPC	84.24	85.54
RTE	68.11	66.78
STS-B	82.02	83.76
WNLI	56.34	56.34
QNLI	82.04	85.78
QQP	82.61	83.45
MNLI	72.44	75.58

The paper attributes the GLUE advantage to reduced error accumulation, because bilinear CIM repeatedly digitizes, requantizes, remaps, and rewrites intermediate tensors, whereas the trilinear design keeps weights static and uses the back gate for dynamic operands. It also reports much lower run-to-run standard deviation for the trilinear architecture (Mia et al., 8 Apr 2026).

On ViT-base, the behavior reverses. Bilinear CIM is closer to the digital baseline on all reported vision datasets: CIFAR-10 $G_{ij}$7 vs $G_{ij}$8, CIFAR-100 $G_{ij}$9 vs $V_i$0, and ImageNet-1K $V_i$1 vs $V_i$2. The paper attributes this to sparse outlier attention scores in ViT, which are more sensitive to the uniform back-gate DAC quantization path than the NLP workloads are (Mia et al., 8 Apr 2026).

For area, latency, and energy, the headline BERT-base results are:

Sequence length	CIM-Bilinear	CIM-Trilinear
64	$V_i$3	$V_i$4
128	$V_i$5	$V_i$6

At sequence length $V_i$7, this corresponds to $V_i$8 area overhead, $V_i$9 lower latency, $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$0 lower energy, throughput improvement from $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$1 to $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$2 inf/s, and TOPS/W improvement from $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$3 to $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$4. At sequence length $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$5, it corresponds to $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$6 area overhead, $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$7 lower latency, $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$8 lower energy, throughput improvement from $Q = X W_Q^T,\qquad K = X W_K^T,\qquad V = X W_V^T,$9 to $K$00 inf/s, and TOPS/W improvement from $K$01 to $K$02. The paper further notes that trilinear execution still has zero runtime cell writes, whereas bilinear CIM incurs millions of cell writes per inference (Mia et al., 8 Apr 2026).

6. Position within trilinear hardware research and reported limitations

Within the CIM literature, TrilinearCIM is distinguished by converting the in-array primitive itself, rather than by merely reducing writes or offloading dynamic attention. ReTransformer reduces writes but does not eliminate them; X-Former and iMTransformer keep static projections in NVM-CIM but move dynamic attention to digital CMOS; TransPIM avoids NVM rewrites by using DRAM-PIM, with the corresponding DRAM trade-offs. TrilinearCIM instead keeps the complete linear attention path—query scaling, score synthesis, and value aggregation—inside NVM cores without runtime reprogramming (Mia et al., 8 Apr 2026).

Its reported limitations are equally central to its definition. The architecture incurs $K$03 area overhead from additional DACs and back-gate drivers. Its trilinear behavior depends on operating within a restricted conductance band and approximating $K$04 by a constant, which the paper notes still requires experimental hardware validation. Softmax, LayerNorm, GELU, and final distributed accumulation remain digital. Accuracy degrades on ViT workloads because of back-gate quantization sensitivity. Reliability questions such as read disturb and broader device-level validation remain open. The design also presumes DG-FeFET devices and selector-less array behavior with high on/off ratio, and decoder-style causal attention with KV-cache management is left to future work (Mia et al., 8 Apr 2026).

In the broader landscape of specialized trilinear hardware, TrilinearCIM should be distinguished from architectures that are trilinear or memory-centric without being direct CIM. TriADA, for example, is a specialized trilinear tensor accelerator with an outer-product, output-stationary, active-memory-streamed dataflow and in-place local accumulation for 3D matrix-by-tensor multiply-add, but it is not a direct compute-in-memory mechanism in the conventional sense (Sedukhin et al., 28 Jun 2025). This distinction is important because TrilinearCIM’s novelty lies specifically in realizing a three-operand MAC inside NVM cells, rather than only in mapping trilinear computation onto a memory-centric accelerator.

Taken together, these features define TrilinearCIM as a DG-FeFET-based NVM-CIM architecture whose essential contribution is the replacement of the conventional bilinear $K$05 primitive by an approximately trilinear $K$06 computation, enabling attention-specific dataflow that eliminates runtime ferroelectric rewriting in the linear attention path (Mia et al., 8 Apr 2026).