CryptPEFT: Efficient Private Inference

• CryptPEFT is a framework for private neural network inference that uses parameter-efficient fine-tuning by combining a public pretrained backbone with lightweight, encrypted adapters.
• It implements a one-way communication architecture that confines encrypted computation to the adapter, achieving speedups from 20.62× to 291.48× while maintaining high accuracy.
• The design incorporates cost-aware neural architecture search and MPC-friendly modifications, optimizing latency and privacy in encrypted computation without compromising utility.

CryptPEFT is a framework for efficient and privacy-preserving neural network inference in parameter-efficient fine-tuning (PEFT) settings, where a public pretrained backbone is combined with lightweight task-specific adapters. It is introduced as the first PEFT solution specifically designed for private inference scenarios, and its central architectural idea is a one-way communication (OWC) structure that confines encrypted computation to the adapter rather than the full model. In evaluations with Vision Transformer backbones on standard image-classification benchmarks, it reports speedups ranging from $20.62\times$ to $291.48\times$ and attains 85.47% accuracy on CIFAR-100 with 2.26 seconds of inference latency (Xia et al., 17 Aug 2025).

1. Problem setting and motivation

CryptPEFT is situated at the intersection of PEFT and cryptographic private inference. In contemporary PEFT pipelines, methods such as LoRA and AdaptFormer insert small trainable adapter modules into a large frozen backbone, enabling rapid task adaptation with a small per-task memory footprint while preserving strong utility. Private inference introduces a dual confidentiality requirement: user inputs should remain hidden from the service provider, and the fine-tuned adapters—often trained on sensitive or proprietary data—should remain hidden from the client. Cryptographic frameworks, especially secure multi-party computation (MPC), provide a natural mechanism for this setting, but their direct application to PEFT remains expensive (Xia et al., 17 Aug 2025).

The core obstacle is that classical PEFT uses bidirectional interaction between backbone and adapter through intermediate activations. A common misconception is that PEFT is automatically inexpensive to secure because only a small fraction of parameters is trainable. CryptPEFT is motivated by the opposite observation: naïvely combining PEFT with MPC still incurs substantial encrypted computation across both backbone and adapter, precisely because their intermediate representations must be exchanged securely. Existing approaches either encrypt the entire model computation, which is slow and communication-heavy, or allow partial plaintext computation via intermediate decryption, which introduces leakage risk (Xia et al., 17 Aug 2025).

2. One-way communication architecture

The defining mechanism of CryptPEFT is its OWC architecture. OWC enforces strictly forward information flow: the client runs the public backbone locally in plaintext up to a designated adapter interface, encrypts the resulting intermediate representation, and sends it to the service provider. The provider executes the adapter under secure computation and returns an encrypted prediction, which the client decrypts. Information is therefore allowed to move from the plaintext backbone into the encrypted adapter, but not back into earlier backbone layers before the final output is revealed (Xia et al., 17 Aug 2025).

This restriction is not merely an implementation detail; it is the architectural condition that confines encrypted computation to the smallest proprietary component. In standard PEFT, residual or feedback-style interactions from adapter outputs into backbone processing would require decrypted intermediate values and thus either enlarge the encrypted computation graph or create privacy leakage. OWC disallows such feedback paths. The result is a private inference pipeline in which the backbone remains local and plaintext, while the adapter alone is evaluated under MPC. Within the paper’s security model, this yields privacy for both user inputs and adapter parameters under the semi-honest setting and simulation-based (UC) frameworks, with no leakage beyond what is inferable from query/output pairs (Xia et al., 17 Aug 2025).

3. Adapter co-design for encrypted execution

OWC imposes a severe functional constraint on adapter design. Conventional PEFT adapters depend on two-way interaction with the backbone and often rely on residual pathways to propagate task-specific features. When those pathways are removed, utility can drop significantly. CryptPEFT therefore treats the adapter not as a direct transplant of a standard PEFT block, but as a component co-designed for encrypted execution under OWC (Xia et al., 17 Aug 2025).

Its principal design innovation is the inclusion of an internal attention mechanism inside the adapter itself. Rather than relying on the backbone’s attention to mediate inter-token structure, the adapter refines the critical $[\mathrm{CLS}]$ token and token interactions within the encrypted domain. Because conventional attention mechanisms are expensive under MPC, especially due to Softmax and GELU, CryptPEFT replaces them with a Learnable Linear Attention mechanism using only addition and multiplication:

$\text{LLA}(Q, K, V) = L(QK^T)V.$

This makes the attention block more MPC-friendly while preserving a mechanism for modeling inter-token dependencies (Xia et al., 17 Aug 2025).

The remaining adapter stack is likewise shaped by cryptographic cost. Weight matrices use a low-rank decomposition in a LoRA-style form with rank $r$, making parameter count tunable. GELU is replaced with ReLU to improve MPC efficiency. Adapter placement is also restricted: instead of inserting adapters at every layer, CryptPEFT places them only in deeper layers of the backbone, which the experiments report as a favorable compromise between encrypted compute and predictive performance. Taken together, these modifications define an adapter family that is OWC-compatible rather than merely PEFT-compatible (Xia et al., 17 Aug 2025).

4. Cost-aware neural architecture search

Because OWC-compatible adapters still present a nontrivial utility–efficiency trade-off, CryptPEFT introduces an automated neural architecture search (NAS) procedure. The search space is parameterized by the number of attention heads $h$, the low-rank approximation rank $r$, the number of inserted adapters $s$, and a fixed scaling factor. Unlike generic NAS formulations that optimize proxy FLOPs or parameter count, CryptPEFT models latency directly in terms of encrypted computation and communication costs, including data volume, bandwidth, number of protocol rounds, and per-round latency (Xia et al., 17 Aug 2025).

The search procedure profiles candidate architectures under specific network conditions such as WAN and LAN settings, then selects the lowest-latency design that satisfies a target utility constraint. For WAN, the paper gives an example communication-time model:

$$\text{comm\_time} = (0.02117\,h + 0.00344\,r + 0.35828)\,s + 0.15541.$$

This operationalizes the trade-off between adapter capacity and private inference latency. Smaller adapters reduce encrypted computation, but overly aggressive compression harms accuracy; larger or more numerous adapters recover utility but increase communication and MPC overhead. The reported search results identify “sweet spots” on this frontier and further show that, under resource constraints, deeper-layer placement is more effective than shallow-layer placement (Xia et al., 17 Aug 2025).

5. Experimental evaluation and comparative performance

The empirical study uses a frozen ViT-B backbone and evaluates on CIFAR-10, CIFAR-100, Food-101, SVHN, and Flowers-102. Baselines comprise traditional PEFT methods such as LoRA and AdaptFormer with full MPC inference, a Simple Fine-Tuning (SFT) regime in which only the last few layers are fine-tuned and computed under MPC while the backbone remains in plaintext, and MPCViT as a state-of-the-art MPC-tailored ViT baseline. The implementation is end-to-end in CrypTen and is evaluated under simulated network conditions of WAN (400 Mbps/4 ms) and LAN (1 Gbps/0.5 ms) (Xia et al., 17 Aug 2025).

Across these settings, CryptPEFT reports large latency reductions. Relative to PEFT baselines, the speedup ranges from $20.62\times$ in WAN to $291.48\times$0 in LAN. Relative to SFT, it also reports a $291.48\times$1 speedup in WAN, with CIFAR-100 inference latency reduced from approximately 45 seconds to 2.26 seconds. Accuracy is maintained despite this reduction in encrypted computation: on CIFAR-100, the reported 85.47% accuracy matches or surpasses classical PEFT and SFT baselines while using far less encrypted work. On average, the framework reduces the number of parameters involved in encryption by nearly 89% compared with SFT baselines. Against MPCViT, the paper reports up to 8–10 percentage points higher accuracy and 9–13$291.48\times$2 faster latency, attributing this to the joint use of a public backbone and OWC-compatible adapters (Xia et al., 17 Aug 2025).

These results characterize CryptPEFT as a systems co-design rather than a purely cryptographic optimization. The performance gains arise not only from “encrypting less,” but from modifying model topology so that only a structurally isolated adapter remains inside the encrypted domain. In that sense, the framework couples architectural constraints, secure protocol cost modeling, and automated design search into a single private-inference pipeline (Xia et al., 17 Aug 2025).

6. Scope, security model, and relation to broader PEFT privacy research

CryptPEFT is evaluated primarily on vision tasks, but its OWC principle, adapter-design strategy, and cost-aware search procedure are described as applicable to LLMs and, with minor changes, to other cryptographic backends such as TEE-based execution. Its security target is private inference in the semi-honest setting; it does not claim to resolve the broader space of PEFT vulnerabilities arising during collaborative or federated training (Xia et al., 17 Aug 2025).

That distinction matters because adjacent PEFT research has identified substantial training-time risks. In federated PEFT, “PEFT-as-an-Attack” shows that PEFT modules themselves can be used as a jailbreak vector: with less than 1% of parameters trainable, and with 1–5 malicious clients in a 15-client federation over 25 communication rounds, LoRA-based attacks raise attack success rate from below 4% before attack to 70–80% after 20 rounds with 5 malicious clients, while legitimate-task accuracy is preserved or even improved by 28–67%. The same study finds that robust aggregation schemes such as DnC and ClippedClustering can reduce ASR to below 2% only in single-domain IID settings, whereas in multi-domain non-IID settings all tested schemes fail and ASR often remains above 70%; post-PEFT safety alignment can reduce ASR to below 10% but incurs an alignment tax, including a 3% accuracy drop on MedQA and up to 20% on MetaMathQA (Li et al., 2024).

A separate line of work, ReCIT, studies privacy leakage from PEFT gradients rather than inference confidentiality. Under a threat model in which the attacker controls the pretrained model and observes PEFT gradients, ReCIT combines malicious memorization through Personal Notes with filter-based token extraction and token pairing to reconstruct both contextual prefixes and PII. It reports up to 10$291.48\times$3 higher PII recovery rates than prior attacks and remains effective at batch sizes up to 128 across LoRA, FedAdapter, and Offsite-Tuning settings (Xie et al., 29 Apr 2025).

Taken together, these results suggest that PEFT should not be treated as a monolithic privacy technology. CryptPEFT addresses confidential inference with encrypted adapters; PaaA concerns integrity and safety alignment in federated adaptation; ReCIT concerns gradient leakage and data reconstruction. The broader implication is that efficient PEFT deployment requires differentiated defenses for inference-time confidentiality, training-time privacy, and federated robustness rather than a single generic notion of “secure PEFT” (Xia et al., 17 Aug 2025).