Safhire: Hybrid FHE ML Inference

• Safhire is a hybrid private ML inference framework that combines server-side FHE-based linear operations with client-side plaintext non-linear activations.
• Its design eliminates bootstrapping overhead by resetting ciphertext noise at each non-linearity, significantly reducing latency compared to fully encrypted methods.
• The protocol employs randomized shuffling and fast ciphertext packing to enhance model confidentiality and optimize communication efficiency.

Safhire is a hybrid private ML inference framework based on fully homomorphic encryption (FHE) in which the server evaluates linear, parameterized layers on encrypted inputs, while the client decrypts intermediate results, applies non-linear activations in plaintext, and re-encrypts for the next round. Implemented over a TFHE-based RLWE scheme, it is designed to remove two central bottlenecks of fully encrypted inference: expensive bootstrapping and inefficient approximation of non-linear activations. Its distinguishing features are elimination of bootstrapping, support for exact activations such as ReLU, and randomized shuffling of intermediate outputs as a model-confidentiality mechanism (Biswas et al., 1 Sep 2025).

1. Problem setting and design motivation

Safhire addresses the practical limits of end-to-end FHE inference. In prior fully encrypted systems, ciphertext noise accumulates across layers, so inference must periodically invoke bootstrapping to refresh ciphertexts. The paper identifies this as the dominant runtime cost, reporting that in Orion bootstrapping accounts for 62%–85% of total inference time across tested ResNet workloads. A second obstacle is the treatment of non-linear activations: functions such as ReLU are not natively supported and are typically replaced by high-degree polynomial approximations or alternative activations, increasing multiplicative depth, noise growth, and inference latency (Biswas et al., 1 Sep 2025).

Safhire changes the computational split rather than attempting to optimize these two costs within a fully encrypted forward pass. Linear layers remain on the server under encryption, but non-linearities are evaluated exactly by the client after decryption. This architecture removes the need to preserve ciphertext validity across deep networks and therefore removes bootstrapping from the online inference path. It also avoids approximation error from encrypted activation surrogates, since the client computes the activation directly in plaintext (Biswas et al., 1 Sep 2025).

This places Safhire in a distinct position relative to prior systems. It differs from fully encrypted frameworks such as Orion, HyPHEN, CHET, EVA, CryptoNets, CryptoDL, and LoLa because those systems keep the full forward pass encrypted on the server. It also differs from MPC/HE hybrids such as Gazelle and Delphi, because Safhire does not invoke online MPC for non-linear layers; instead, the client decrypts, evaluates the non-linearity locally, and re-encrypts (Biswas et al., 1 Sep 2025).

2. Protocol architecture and round structure

Safhire executes inference in $L$ rounds, each corresponding to a linear block followed by a client-side non-linearity. The server computes linear maps of the form

$f(x)=Ax+b$

with $A \in \mathbb{R}^{m \times d}$ and $b \in \mathbb{R}^m$. Convolution is treated equivalently, since the paper notes that a convolution can be represented as multiplication by a Toeplitz matrix (Biswas et al., 1 Sep 2025).

Stage	Server	Client
Input preparation	Receives ciphertexts	Flattens, chunks, encrypts
Linear block	Evaluates encrypted convolution / fully connected layer	Waits
Intermediate handling	Unshuffles previous output, shuffles current output, packs ciphertexts	Decrypts packed outputs
Non-linearity	None	Applies exact ReLU and requantization
Next round	Receives fresh ciphertexts	Re-encrypts activated output

For each round, the client first flattens its input tensor into a vector $V \in \mathbb{Z}^E$, partitions it into chunks $c_0,\ldots,c_{k-1}$, and embeds each chunk into a polynomial

$m_j(X)=\sum_{i=0}^{N-1} c_{j,i}X^i.$

These polynomials are encrypted as RLWE ciphertexts and sent to the server. The server then performs coefficient extraction, reconstructs the encrypted tensor in the layout required by the current layer, and homomorphically evaluates the linear computation. After that, it applies a fresh secret permutation to the output rows, fast-packs the resulting ciphertexts, and returns them to the client (Biswas et al., 1 Sep 2025).

The client decrypts the packed ciphertexts, reconstructs the intermediate vector, applies the activation exactly in plaintext, and requantizes using the per-layer scale $\eta$. The requantization step is described as

$$y \mapsto \mathrm{clip}\!\big(\lfloor y/\eta \rceil\big).$$

The client then re-encrypts the activated output and starts the next round. Because each round begins from fresh ciphertexts supplied by the client, ciphertext noise is reset at every non-linearity boundary, which is the mechanism by which Safhire avoids bootstrapping (Biswas et al., 1 Sep 2025).

3. Cryptographic machinery and protocol-level optimizations

The framework uses a TFHE-based RLWE scheme. In setup, the client generates an RLWE secret key $s(X)$ and key-switching keys. For an automorphism $f(x)=Ax+b$0, the paper defines the key-switching key as

$f(x)=Ax+b$1

where $f(x)=Ax+b$2 is the decomposition base and $f(x)=Ax+b$3 is the decomposition depth (Biswas et al., 1 Sep 2025).

A central operation is partial trace extraction. The server uses encrypted polynomial representations to recover coefficient information through trace identities such as

$f(x)=Ax+b$4

The paper then defines extraction procedures for different levels $f(x)=Ax+b$5, including

$f(x)=Ax+b$6

and

$f(x)=Ax+b$7

These identities are used to reduce server work when the client precomputes encrypted powers of the message polynomial (Biswas et al., 1 Sep 2025).

Safhire’s other major systems optimization is fast ciphertext packing. The server takes $f(x)=Ax+b$8 RLWE ciphertexts and packs them into one RLWE ciphertext encrypting

$f(x)=Ax+b$9

The paper states that this reduces ciphertext count by a factor of $A \in \mathbb{R}^{m \times d}$0. Together with partial extraction, this optimization targets the communication and server-side overheads that would otherwise offset the latency gains from removing bootstrapping (Biswas et al., 1 Sep 2025).

4. Security model and confidentiality mechanisms

Safhire pursues two privacy objectives: client data privacy and server model confidentiality. The server is assumed to be semi-honest (honest-but-curious), meaning that it follows the protocol but tries to infer information about the client input from the messages it receives. The client, by contrast, may be adversarial with respect to model extraction and may send arbitrary inputs in every layer. Network traffic is assumed to be protected by authenticated and encrypted channels such as TLS, while side channels such as timing and power leakage are explicitly outside scope (Biswas et al., 1 Sep 2025).

Client data privacy follows the standard FHE model: the server processes only ciphertexts and never sees plaintext inputs or plaintext activations. The more distinctive issue is model confidentiality. Because the client decrypts intermediate outputs in a hybrid protocol, a naive design would reveal much more about the model than standard black-box inference. Safhire addresses this with randomized shuffling: after each encrypted linear block, the server applies a secret random permutation $A \in \mathbb{R}^{m \times d}$1 to the output rows, and in the next round it privately undoes the previous permutation using $A \in \mathbb{R}^{m \times d}$2. The permutation is derived from a secret seed based on the session identifier and round number, and is unique per client and per round (Biswas et al., 1 Sep 2025).

The paper does not claim perfect cryptographic hiding of all intermediate information. Rather, it argues that randomized shuffling obfuscates intermediate values and makes exact model reconstruction practically impossible. It also states that explicit leakage is limited to setup metadata, message sizes, and the number of exchanged messages. This places Safhire’s confidentiality claim in a pragmatic category: the system is designed to resist practical model extraction despite the client’s access to shuffled plaintext intermediates, but it is not presented as a zero-leakage protocol in the formal sense (Biswas et al., 1 Sep 2025).

5. Empirical performance and comparison with Orion

The evaluation reports results on multiple standard models and datasets, with Safhire achieving 1.5X–10.5X lower inference latency than Orion, a state-of-the-art baseline, while maintaining manageable communication overhead and comparable accuracy (Biswas et al., 1 Sep 2025).

The main performance explanation is architectural rather than micro-optimizing. Orion keeps the full forward pass encrypted, so its runtime is heavily dominated by bootstrapping; Safhire removes that cost entirely by decrypting at non-linearity boundaries. The second source of improvement is semantic rather than purely cryptographic: Safhire uses exact client-side activations instead of encrypted polynomial approximations. This removes both approximation overhead and the multiplicative-depth burden that would otherwise accelerate noise growth (Biswas et al., 1 Sep 2025).

The paper’s protocol optimizations are also intended to keep the hybrid design practical at the systems level. Fast ciphertext packing reduces the number of ciphertexts transmitted back to the client, while partial extraction reduces server-side work in reconstructing linear-layer inputs. The resulting design attempts to balance computation and communication rather than optimizing either in isolation. The reported outcome is a latency profile that is substantially lower than a fully encrypted baseline without sacrificing model architecture to FHE-friendly activation substitutions (Biswas et al., 1 Sep 2025).

6. Position within private inference research

Safhire belongs to a broader line of work on privacy-preserving inference, but its most important conceptual contribution is the placement of the privacy boundary exactly at the interface between linear and non-linear computation. Fully encrypted systems retain a simpler trust split but pay for bootstrapping and activation approximation. MPC/HE hybrids move non-linearities into secure two-party protocols. Safhire instead assigns them to the client directly, turning the client into the refresh point for both ciphertext noise and activation semantics (Biswas et al., 1 Sep 2025).

This yields a specific tradeoff structure. The framework preserves encrypted server-side evaluation for model-parameterized linear layers, but accepts client access to shuffled plaintext intermediates. The paper’s security contribution is therefore inseparable from its systems contribution: randomized shuffling is not an auxiliary enhancement but the mechanism that makes the hybrid split compatible with model confidentiality claims. A plausible implication is that Safhire is best understood not as a general replacement for end-to-end FHE inference, but as a distinct design point that prioritizes practical latency reduction while retaining encrypted handling of client data and a pragmatic, obfuscation-based defense for server weights (Biswas et al., 1 Sep 2025).

In that sense, Safhire reframes the practicality question in FHE inference. Instead of asking how far a fully encrypted forward pass can be optimized before bootstrapping and polynomial activations become prohibitive, it asks which parts of inference must remain encrypted at all. The answer proposed by the paper is sharply asymmetric: keep linear layers encrypted, move non-linear layers to the client, and use shuffling plus packing and extraction optimizations to make the resulting protocol computationally and communicatively viable (Biswas et al., 1 Sep 2025).