Immutable Tensor Architecture

• Immutable Tensor Architecture is a hardware paradigm that encodes fixed neural weights as static logic within ASIC circuits, eliminating dynamic memory access.
• It employs a split-brain system design where stateless linear computations run on the ITA ASIC while dynamic tasks are handled by host processors.
• The approach delivers significant energy savings (up to 50× reduction) and enhanced security, making it ideal for stable, long-term AI inference in edge applications.

The Immutable Tensor Architecture (ITA) is a hardware paradigm for deploying large-scale deep learning models, notably LLMs, that reconceptualizes immutable model weights not as data to be fetched but as fixed physical circuit topology. Instead of storing model parameters in DRAM or SRAM and loading them on demand, ITA physically encodes all neural weights directly into the silicon of a purpose-built ASIC at synthesis time. This “One Model, One Chip” approach eliminates instruction fetch/decode logic and obviates the energy and latency bottlenecks of memory hierarchies, enabling secure, energy-efficient AI inference tailored to edge and embedded devices (Li, 28 Nov 2025).

1. Guiding Principles: Weights as Immutable Circuit Topology

Traditional AI accelerators treat network parameters $\theta$ as mutable software data, repeatedly fetching model weights from memory at every inference step. ITA asserts that since inference requires fixed parameters, weights should be hardwired into circuitry as static logic. This yields several primary features:

• The entire neural computation graph—matrices $W_q, W_k, W_v, W_1, W_2, W_3$—is instantiated as a spatial arrangement of gates and metal interconnects, resulting in a pure dataflow engine.
• The instruction-fetch, decode, and storage hierarchy for weights are fully eliminated.
• Flexibility in model updating and in-field training is sacrificed for significant gains in silicon area efficiency, energy consumption, and attack surface reduction.
• The architecture mirrors legacy “cartridge” systems where logic was directly fabricated per application.

By design, the ITA is best matched to stable, long-lived models, where immutable deployment and hardware-level security are desired over frequent model iteration or retraining.

2. Split-Brain System Design

ITA divides system responsibilities between a host processor (CPU/GPU) and the ITA ASIC device. This “split-brain” structure leverages the host for dynamic, stateful workloads, while relegating fixed, stateless linear projections to the ITA engine.

Component	Host CPU/GPU Responsibilities	ITA ASIC Responsibilities
Tokenization	Yes	No
KV-cache	Yes (DRAM-backed)	No
Attention Softmax	Yes	No
Linear layers	No	Yes (ROM-embedded logic)
FFN computation	No	Yes
DRAM/SRAM	Yes (KV-cache only)	No

A canonical inference flow per token comprises tokenization, host-device activation transfer (8 KB), QKV computation on device (16 KB return), host-side attention and sampling, and further host-device transfer for FFN computation. All weight-dependent computation occurs in the stateless, hardwired ITA block.

3. Pure Dataflow Execution Model

ITA implements a spatially unrolled, statically routed dataflow architecture:

• Every transformer layer corresponds to a fixed physical pipeline of constant-weight multiply–accumulate (MAC) units.
• There is no program counter, instruction flow, or dynamic memory address generation for weights.
• Activations propagate linearly through QKV and FFN logic, leveraging deeply pipelined, shift-add trees for constant-coefficient multipliers.
• Removal of DRAM/SRAM for weights entirely eliminates $E_{\text{DRAM}}$ terms in the energy per inference.

The total inference energy is formally expressed as:

$$E_{\mathrm{total}} = N_{\mathrm{ops}} \cdot E_{\mathrm{MAC}} + N_{\mathrm{params}} \cdot E_{\mathrm{wire}}$$

where $E_{\mathrm{MAC}}$ is the dynamic MAC operation energy, and $E_{\mathrm{wire}}$ is the energy incurred by routing through the chip’s metal layers.

By contrast, traditional DRAM-based acceleration includes an obligatory $|\theta| \cdot E_{\mathrm{DRAM}}$ term for each token generated.

4. Hardware Microarchitecture and Synthesis

ITA targets mature process nodes—TSMC 28HPC+ or 40 nm planar—with the following characteristic workflow:

• Model parameters are quantized to INT4, and activations to INT8.
• Weights, at a density of ≈0.12 μm²/bit (hardwired logic), are synthesized into metal interconnect and gate arrays with global routing overhead between 1.4× and 3×.
• Additional area for control, SerDes interfaces, and power management adds an estimated 15% overhead.

Exemplar deployments:

• TinyLlama-1.1B (1.1G parameters): 4.4G raw bits occupy 528 mm², increasing to ≈850 mm² after routing/control; physical design optimization has achieved 520 mm² mono-die implementation.
• Llama-2-7B: 28G raw bits necessitate ~5410 mm², implemented as eight chiplets of ~460 mm² each, on a 2.5D silicon interposer.

Pipeline folding and mesh layout strategies minimize wirelength (mean: 5 mm/interconnect, Metal-3, ~0.2 fF/μm), optimizing both latency and power.

5. Performance Analysis and Formal Characterization

Formal device-level and system-level metrics for ITA include:

• Energy per MAC: $E_{\mathrm{MAC}} \approx 0.05$ pJ, $E_{\mathrm{interconnect}} \approx 4.0$ pJ. Compared to GPU INT8 operation (201 pJ), ITA provides a 49.6× reduction.
• Bandwidth per token: $$B_{\text{token}} = (16\,\mathrm{KB} + 8\,\mathrm{KB}) \times 32 + 64\,\mathrm{KB} = 832\,\mathrm{KB/token}$$
• Sustained bandwidth: $B = 832\,\mathrm{KB/token} \times T$, where $T$ is tokens per second.
• Pipeline latency per linear layer: $L_{\text{layer}} = \frac{\text{pipeline_depth}}{f_{\text{clk}}}$; total device compute $\approx 64\,\mu\mathrm{s}$
• System power and throughput: Device (1.13 W @ 20 tok/s), SerDes (0.5 W), and host CPU (5–10 W), for total 7–12 W system power, a 10–15× gain relative to 200–300 W GPUs.
• Gate count reduction: ITA’s constant-coefficient INT8 multiplier uses 243 gates (vs. 1180 for generic), yielding 4.85× reduction theoretically; empirical FPGA LUT reduction is 1.81× (system estimate: 1.62×).

Device cost is estimated at $52 for TinyLlama-1.1B and$165 for Llama-2-7B (in 10K-volume); at large scale, NRE amortization yields $25/unit and retail in the$200–500 range.

6. Comparative Evaluation

Accelerator	Throughput (tok/s)	Power	Unit Cost	Notes
Qualcomm Hexagon	≈20	1.5 W	n/a	Edge NPU
Google Coral TPU	Low	2 W	$60	Low throughput
ITA (TinyLlama)	10–20	1.1 W	$165	Measured @1.13 W, 20 tok/s

PCIe 3.0×4 yields up to 188 tokens/s interface-limited; Thunderbolt 4, up to 192 tokens/s; USB 3.0, 126 tokens/s. Host-limited throughput falls in the 10–20 tokens/s range for real-world scenarios.

7. Limitations, Security Properties, and Prospective Enhancements

ITA’s immutability entails a rigid lack of in-field updates or fine-tuning capability, circumscribing its use to stable, long-term support models rather than rapid research prototyping. Side-channel vulnerabilities persist: deterministic power traces can potentially leak hardwired weights, motivating countermeasures such as clock randomization or noise injection (with a reported 10–20% area/power overhead).

Security is augmented by raising the minimum cost of model extraction: casual attacks (software dump, $0–2k) are replaced by the need for physical reverse-engineering ($50k+). Like classical ROM-embedded hardware, ITA’s model secrecy is both a strength (attack resistance) and, via possible side-channels, a challenge for cryptographic-grade deployments.

Proposed research directions include:

• Hybrid ITA: Hardwire 70% of parameters while keeping QKV in SRAM for limited in-field updates.
• On-device KV-cache: Incorporation of 256MB eDRAM absorbs attention computation and reduces host-induced latency.
• Hardwiring sparse/approximate attention: Targets very long-context models via pre-synthesized context handling.
• Extended quantization and empirical benchmarking: INT3 logic-aware quantization, and accuracy validation against MMLU, HellaSwag, etc.

In summary, ITA demonstrates that weight-as-circuit logic yields approximately 50× energy savings and strong security properties for fixed-model AI inference, reaffirming a domain-specific ASIC approach in edge and embedded AI applications (Li, 28 Nov 2025).