Low-Cost Multi-Precision Systolic Arrays for Accelerating FHE NTTs on AI ASICs

Abstract: Fully Homomorphic Encryption (FHE) ensures robust data privacy but suffers from prohibitive computational overhead. Accelerating FHE on AI hardware like Tensor Processing Units (TPUs) is promising, yet fundamentally limited by a precision mismatch: TPUs are optimized for 8-bit arithmetic, whereas FHE and its critical parts such as the Number Theoretic Transform (NTT), demand high precision. Current approaches bridge this gap using matrix decomposition to execute NTT computations on low-precision matrix engines. However, reconstructing the full-precision results requires shift-and-add accumulation that does not match the dataflow of matrix multiplication. This forces offloading full-precision reconstruction from matrix engines to vector processors that disrupts the matrix multiplication dataflow, creating significant performance bottleneck. To resolve this limitation, we propose a minimally modified multi-precision systolic array that performs full-precision output reconstruction natively within the array in sync with low-precision matrix multiplication under a uniform dataflow. Synthesized at 7nm with OpenRoad, our design incurs negligible hardware overhead. Cycle-accurate simulations using SCALE-Sim demonstrate that natively executing NTTs on the proposed architecture achieves at least 1.33x speedup, for transform sizes 2^12 to 2^16 on 128x128 matrix engines, successfully enabling standard AI hardware to support high-precision FHE acceleration.

• The paper demonstrates that embedding full-precision reconstruction within the systolic array eliminates external carry propagation and dataflow bottlenecks.
• It proposes a minimally modified design with Reconstruction PEs that synchronizes 8-bit matrix multiplication and full-precision digit recomposition internally.
• Experimental results show speedups from 1.33x to 4.49x for FHE NTTs with less than 1% hardware overhead, enabling efficient encrypted computation.

Low-Cost Multi-Precision Systolic Arrays for Accelerating FHE NTTs on AI ASICs

Introduction and Motivation

Fully Homomorphic Encryption (FHE) represents a class of cryptosystems enabling computation over encrypted data, but suffers from severe computational overhead primarily due to modular arithmetic over large integer moduli. The Number Theoretic Transform (NTT), fundamental for polynomial multiplication in FHE schemes, demands high-precision operations (32–64 bits), which are mismatched with the low-precision (8-bit) architectures used in current AI ASICs (e.g., TPUs). Existing acceleration approaches utilize matrix decomposition, executing decomposed high-precision matrix multiplications using low-precision engines, but these require reconstruction steps (shift-and-accumulate with carry propagation) that disrupt the native matrix multiplication dataflow and introduce performance bottlenecks.

This paper proposes a minimally modified multi-precision systolic array architecture capable of performing both low-precision matrix multiplication and full-precision reconstruction internally and synchronously within the matrix engine. This eliminates dataflow interruptions, offloading, and reduces overall cycles required for FHE NTT computation, with negligible hardware overhead.

Architectural Design

The architecture builds on a standard weight-stationary systolic array where each processing element (PE) executes 8-bit products and vertical accumulation. The innovation lies in the last row of the array, named Reconstruction PEs (RPEs), which are augmented with an additional adder to support digit-level horizontal carry propagation. Intermediate results produced by matrix multiplication are streamed least-significant-digit first to synchronize carry propagation across columns, reconstructing full-precision outputs as a seamless extension of the matrix multiplication.

Figure 1: The proposed systolic array with last-row RPEs performing simultaneous vertical addition and horizontal carry propagation.

This design ensures that typical multiply-add throughput is preserved throughout the array while enabling digit recomposition internally, thus avoiding the external vector processing unit (VPU) bottleneck found in previous approaches.

Dataflow and Reconstruction Mechanism

The high-precision matrix multiplication process is separated into decomposition (expanding full-precision values into digit matrices), intermediate low-precision matrix multiplications, and recomposition (shift-and-add with carry propagation). Digit decomposition for the left matrix forms expanded row vectors, while the right matrix uses a Toeplitz structure to maintain required cross-digit products for schoolbook multiplication. Carry propagation is restricted within each digit group to localize the reconstruction, preventing expansion beyond required columns.

Figure 2: In-sync vertical reduction and horizontal carry propagation for single-digit and full-precision matrix reconstruction within the systolic array.

This mechanism is particularly tailored for the 4-step NTT algorithm used in FHE, with offline digit-weight reduction for twiddle factors via Basis-Aligned Transformation (BAT), minimizing required matrix dimensions and further optimizing execution.

Experimental Evaluation and Numerical Results

Cycle-accurate simulation using SCALE-Sim (v3) models a 128×128 systolic array and a 128-lane VPU, representative of industrial AI ASICs. Comparative assessment against TensorFHE ("TensorFHE: Achieving Practical Computation on Encrypted Data Using GPGPU" [10071017]) and CROSS ("Leveraging ASIC AI Chips for Homomorphic Encryption" [11408507]) demonstrates consistent runtime reduction:

• For NTT sizes $2^{12}$ to $2^{16}$ and precision from 32 to 64 bits, full-precision reconstruction overhead is almost entirely removed.
• Speedup ranges from $1.33\times$ (large NTTs) to $4.49\times$ (smaller NTTs), mainly attributed to elimination of external accumulation via the VPU.
• Hardware synthesis at 7nm using OpenROAD shows less than 1% area and power overhead for the proposed RPE augmentation, with the relative penalty decreasing at larger array sizes.

  Figure 3: Physical layout of an 8×8 systolic array instance with the proposed reconstruction-enabled last row.

These results directly showcase the architectural efficiency and scalability with negligible resource consumption.

Practical and Theoretical Implications

The integration of full-precision reconstruction inside systolic arrays closes the longstanding precision mismatch barrier between FHE workloads and AI ASICs, transforming standard low-precision matrix engines into practical accelerators for high-precision modular arithmetic. This dual-mode operation ensures compatibility with low-precision AI workloads and enables real-world adoption of privacy-preserving computation without specialized hardware. The architectural paradigm supports sustained accelerator utilization, minimizes data movement, and drastically improves energy efficiency for cryptographic workloads.

Theoretically, this design eliminates serialization bottlenecks during multi-precision matrix operation, opening avenues for more complex homomorphic operations and broader cryptographic primitives on commodity AI hardware. Future work may investigate integration of adaptive-precision schemes, further digit-level optimizations, and deployment for post-quantum secure computing frameworks.

Conclusion

The paper presents a systolic array architecture with minimally invasive modifications—specifically, enhanced last-row processing elements capable of digit-level carry propagation—to synchronize low-precision matrix multiplication with high-precision reconstruction entirely within the array. This enables FHE NTT acceleration on standard AI ASICs, achieving strong numerical speedups and negligible hardware overhead. The design fundamentally bridges the gap in computational precision and efficiency needed for scalable encrypted computation in practical AI deployments (2606.19866).