GPU Acceleration of TFHE-Based High-Precision Nonlinear Layers for Encrypted LLM Inference

Abstract: Deploying LLMs as cloud services raises privacy concerns as inference may leak sensitive data. Fully Homomorphic Encryption (FHE) allows computation on encrypted data, but current FHE methods struggle with efficient and precise nonlinear function evaluation. Specifically, CKKS-based approaches require high-degree polynomial approximations, which are costly when target precision increases. Alternatively, TFHE's Programmable Bootstrapping (PBS) outperforms CKKS by offering exact lookup-table evaluation. But it lacks high-precision implementations of LLM nonlinear layers and underutilizes GPU resources. We propose \emph{TIGER}, the first GPU-accelerated framework for high-precision TFHE-based nonlinear LLM layer evaluation. TIGER offers: (1) GPU-optimized WoP-PBS method combined with numerical algorithms to surpass native lookup-table precision limits on nonlinear functions; (2) high-precision and efficient implementations of key nonlinear layers, enabling practical encrypted inference; (3) batch-driven design exploiting inter-input parallelism to boost GPU efficiency. TIGER achieves 7.17$\times$, 16.68$\times$, and 17.05$\times$ speedups over a CPU baseline for GELU, Softmax, and LayerNorm, respectively.

• The paper introduces TIGER, the first GPU-accelerated framework for high-precision nonlinear LLM inference using TFHE.
• It employs GPU-native WoP-PBS, numerical refinement, and hardware-aware batching to overcome precision bottlenecks in encrypted nonlinear operations.
• Empirical results demonstrate up to 17× speedups and a 15.5× reduction in inference time, enabling scalable privacy-preserving machine learning.

GPU Acceleration of TFHE-Based High-Precision Nonlinear Layers for Encrypted LLM Inference

Introduction

Deploying LLMs on cloud infrastructure introduces serious privacy risks, motivating research into Fully Homomorphic Encryption (FHE) solutions for encrypted inference. Classical FHE protocols like CKKS are well-suited for the linear operations in Transformer architectures, but their inability to efficiently and precisely handle critical nonlinear layers (e.g., Softmax, LayerNorm, GELU) remains a primary barrier for high-fidelity encrypted inference. In contrast, TFHE supports exact programmable bootstrapping (PBS) via lookup tables (LUTs), though its GPU implementations and support for high-precision, complex nonlinearities are immature and highly resource constrained.

This paper introduces TIGER, the first GPU-accelerated framework targeting TFHE-based, high-precision, nonlinear LLM layer evaluation. TIGER integrates GPU-optimized WoP-PBS algorithms, numerical refinement, and batch-centric parallelization to overcome key algorithmic and systems obstacles and provides practical TFHE substrate for LLM model-level operators.

Challenges in Encrypted Nonlinear Layer Evaluation

LLMs' nonlinear layers are demonstrably sensitive to quantization and precision loss—aggressive approximation in these layers results in catastrophic degradation of perplexity and model utility, as confirmed by empirical studies on GPT-2. While linear layers can generally be compressed to 4–8 bit precision with minimal quality loss, reducing nonlinear layers to even 4-bit precision inflates perplexity by orders of magnitude, rendering the model nonviable for real-world use.

CKKS-based schemes lack native support for nonlinear computation, depending on high-degree polynomial approximations that are both computation- and memory-intensive, especially as required precision increases. Advanced TFHE techniques, notably WoP-PBS, slightly relax these constraints by structuring LUT-based nonlinear evaluation without the exponential overhead of tree-structured approaches like FBT-TFHE. However, their practical precision is capped around 20 bits, and existing open-source implementations are CPU-based, severely underutilizing contemporary GPU hardware.

TIGER Framework Architecture

TIGER introduces a modular architecture (Figure 1): at the top level, composite nonlinear LLM operations (Softmax, LayerNorm, GELU) are implemented as compositions of high-precision function evaluators and fixed-point arithmetic operators, built atop a common set of optimized TFHE primitives.

Figure 1: Architecture overview of TIGER, depicting hierarchical modularization, refinement-based nonlinear evaluation, and core GPU modules for TFHE computations.

Central innovations in TIGER include:

• GPU-native WoP-PBS: A highly-parallelized, GPU-optimized WoP-PBS execution path, sidestepping both the exponential noise accumulation and computation cost of alternative bootstrapping methods.
• Numerical refinement augmentation: LUT approximations are paired with Taylor expansion-based local refinement, enabling effective precision expansion without a proportional increase in memory or LUT size.
• Hardware-aware batching and scheduling: PBS batch sizes are auto-tuned to GPU memory and cache characteristics, leveraging both intra- and inter-input parallelism for resource maximization and throughput.

High-Precision Nonlinear Operator Construction

Key nonlinearities are constructed using LUT–numerical co-design:

• Exp(-x), central in Softmax, is decomposed such that high-bits are handled with logical short-circuiting, middle bits are mapped by WoP-PBS, and low bits refined via first-order Taylor expansion. This minimizes LUT size while covering the input's dynamic range at high fidelity.
• GELU(x) is efficiently handled by sign extraction, region partitioning, and a single-lookup plus low-rank correction scheme, with piecewise refinements based on the normal CDF and PDF.
• InvSqrt(x), required for LayerNorm, is tackled by multi-interval LUT partitioning and local Taylor approximations, covering a wide value range with region-specific parameterizations.

These decompositions, combined with batch-scheduled and memory-aware multiply/divide primitives at the fixed-point operator level, compose the nonlinearities as needed for LLM inference under TFHE.

Systems Optimizations and Efficient Execution

TIGER’s performance gains are not limited to algorithmic improvements but are deeply tied to co-optimization with GPU hardware:

• Optimized FFT kernels employ 4-radix expansion and Karatsuba multiplication to directly accelerate underlying polynomial arithmetic for bootstrapping.
• Multiply scheduler (Figure 2) prunes unnecessary partial products and exploits execution plan memoization, minimizing computation and memory access overhead across batched operations.

  Figure 2: The multiply scheduler prunes, groups, and schedules fixed-point multiplications to maximize GPU parallelism and minimize unnecessary computation.

• Batch splitting addresses memory bandwidth and cache utilization challenges observed in naive PBS batch scheduling. Empirical results (Figure 3) show that throughput scales sublinearly with large batch sizes due to cache saturation and DRAM contention.

  Figure 3: PBS time (red) and efficiency (blue) as a function of batch size, with linear scaling (dashed red) for reference; batch size must be tuned for peak throughput.

Empirical Evaluation

TIGER is benchmarked on an RTX 6000 Ada Generation GPU. The GPU-accelerated implementation delivers speedups over multithreaded CPU baselines of 7.17$\times$ (GELU), 16.68$\times$ (Softmax), and 17.05$\times$ (LayerNorm) (Figure 4). End-to-end, a GPT-2 Transformer block with GPU-based nonlinear TFHE evaluation achieves an overall wall-time reduction from 4.25 hours to roughly 16.4 minutes—an overall speedup of 15.54$\times$.

Figure 4: Layer-wise execution time for GELU, Softmax, LayerNorm comparing TIGER to CPU and GPU-FBT alternatives.

Ablation studies (Figure 5) confirm the orthogonal contributions of batch-splitting, Karatsuba FFT, and 4-radix decomposition, with batch-splitting providing the largest incremental gain.

Figure 5: TIGER ablation—relative run-time impact of key system- and algorithm-level optimizations.

Scalability experiments (Figure 6) further demonstrate the necessity of the batch split tactic, with inefficient batching causing up to 4.35$\times$ slowdowns at large LLM dimensions.

Figure 6: Execution time scaling with input dimension for GELU, Softmax, LayerNorm across batching strategies; optimal batch splitting maintains scalability at all sizes.

Implications and Outlook

TIGER’s architecture validates that FHE-based LLM inference is feasible and efficient for practical, high-precision use cases, with the nonlinear operators remaining the key latency bottleneck. By addressing precision-scaling challenges via hybrid LUT-analytic strategies and matching batching to hardware, TIGER closes the fundamental throughput and precision gaps encountered by prior approaches.

The modular design of TIGER generalizes seamlessly beyond LLMs to privacy-preserving machine learning tasks in probabilistic prediction, risk-scoring, and biomedical analytics, particularly where numerical errors can carry critical real-world consequences. Its device ownership model and memory footprint are amenable to multi-GPU and distributed scaling with minimal synchronization cost, suggesting promising avenues for further research on cross-device protocol conversion and larger architectures.

Further developments could include adaptive batch-size selection, more hardware-specific kernel fusion, and expanded support for additional activation functions and LLM variants. Integration with OpenFHE and similar frameworks can further streamline linear-nonlinear operator interoperation and protocol transitions.

Conclusion

TIGER is the first practical implementation of high-precision, GPU-accelerated TFHE-based nonlinear layer evaluation for encrypted LLM inference. It leverages GPU-native WoP-PBS, numerical refinement, and sophisticated batching to address precision limitations, resource underutilization, and algorithmic bottlenecks. Substantial empirical speedups and scalable design demonstrate a viable path to efficient, high-precision privacy-preserving LLM inference. TIGER lays the groundwork for future encrypted learning systems leveraging high-performance heterogeneous hardware.