Revisiting ML Training under Fully Homomorphic Encryption: Convergence Guarantees, Differential Privacy, and Efficient Algorithms

Abstract: We present the first theoretical convergence analysis of machine learning training under fully homomorphic encryption (FHE), combined with a differentially private (DP) training algorithm tailored to encrypted computation. Our approach improves computational efficiency over standard differentially private gradient descent (DP-GD) while achieving comparable utility. In particular, we prove convergence of approximate gradient descent using polynomial approximations of activation and loss functions, which are required for FHE compatibility. To preserve privacy in downstream tasks, we integrate differential privacy without relying on costly per-sample gradient clipping, enabling scalable encrypted learning. We also provide data-independent hyperparameter selection and theoretically grounded strategies for polynomial approximation which can be of independent interest. Together, these contributions advance the feasibility of efficient, private, and secure machine learning on sensitive data.

• The paper establishes theoretical convergence guarantees for polynomial gradient descent under FHE, bounding both suboptimality and parameter error.
• It introduces a novel differential privacy algorithm that removes costly per-sample gradient clipping while retaining strong (ε, δ)-DP guarantees and high model utility.
• The method leverages barrier-augmented objectives and offline hyperparameter selection to achieve up to 5× speedup and reduced circuit depth with minimal accuracy loss.

Revisiting ML Training under Fully Homomorphic Encryption: Convergence Guarantees, Differential Privacy, and Efficient Algorithms

Introduction and Problem Setting

This paper (2605.27782) systematically addresses the challenge of training ML models on sensitive data outsourced to untrusted infrastructure, using a unified framework that leverages Fully Homomorphic Encryption (FHE) and Differential Privacy (DP). Prior FHE-ML methods supported secure outsourcing but lacked rigorous convergence guarantees; while DP approaches for ML faced severe complexity bottlenecks when adapted to the encrypted domain. The authors present a new protocol that not only provably guarantees security and privacy during and after training but also achieves strong computational efficiency by introducing new algorithmic and analytical techniques.

Figure 1: End-to-end workflow for secure and differentially private training over FHE-encrypted datasets.

Contributions

The core contributions are:

1. Theoretical Convergence Guarantees under FHE with Polynomial Approximations:

A formal analysis of approximate (polynomial) GD for FHE compatibility is established, with clear bounds for both the suboptimality and the parameter distance with respect to the true loss, despite the surrogate loss potentially losing convexity.

2. Efficient DP-ML without Gradient Clipping under FHE:

A novel DP algorithm is introduced that eliminates per-sample gradient clipping, typically a major bottleneck under FHE, while retaining provable $(\epsilon, \delta)$-DP guarantees and high utility. The approach incorporates a barrier-function-augmented objective, yielding tight, data-independent sensitivity and bound control.

3. Rigorous, Data-Independent Hyperparameter and Approximation Selection:

The method enables fully offline selection of key hyperparameters and polynomial approximations, with error and monotonicity guarantees over all encountered input domains, removing heuristic interval selection endemic to previous FHE-ML work.

4. Advanced Implementation and Empirical Validation:

Experiments demonstrate $\geq 3\times$–$5\times$ computational speedup and matching (or smaller) multiplicative circuit depth compared to standard FHE DP-GD with negligible reduction in model accuracy or AUC—at most 1.82% in accuracy and 0.78% in AUC—all while ensuring strong privacy and security.

Methodology and Technical Innovations

Secure and Private Training Architecture

Data owners encrypt their datasets (and pre-sampled DP noise vectors) with FHE. The untrusted server executes homomorphic DP gradient descent, adding encrypted noise per iteration. The model remains encrypted until final decryption by the client. The framework generalizes naturally to multiple clients in a non-interactive, single round setting.

Polynomial Approximation and Convergence Analysis

Due to FHE restrictiveness, all nonlinearities and losses (including activation functions) are replaced with polynomial approximations. The analysis proves:

• For strongly convex losses: Given gradient error $\zeta$, after $T$ steps, the suboptimality and parameter error scales with $\zeta$ and the smoothness and strong convexity constants of the surrogate polynomial loss.
• For DP-noisy training (expected curvature): The expected squared distance to the true optimum is upper-bounded by a term scaling with $\sigma^2$ (noise), $\zeta^2$ (approximation error), and the expected curvature parameter.

Elimination of Clipping: Barrier-Augmented Objectives

Standard DP-GD depends on per-sample clipping to bound sensitivity, a process requiring expensive non-polynomial (e.g., norm, division, comparison) operations that explode circuit depth under FHE. The paper demonstrates that, via a strictly convex barrier-augmented objective, the iterates ${w}_i$ are confined to a prescribable region, ensuring all inputs for polynomial approximations remain in a bounded interval, enabling precise sensitivity calibration and DP analysis without the need for encrypted clipping.

Numerical and Empirical Results

The approach is benchmarked across several datasets (MNIST, Adult, Credit, Compas) in both plaintext and FHE settings. The findings include:

• Model utility: The new protocol matches or slightly surpasses clipping-based FHE DP-GD in both accuracy and AUC, with maximum observed drop $<$2% in accuracy.
• Runtime and depth: For 128-bit security (RingDim 131072), per-iteration runtime is decreased $\geq 3\times$0 compared to standard FHE DP-GD. Circuit depth per iteration is reduced from 24 (traditional) to 9 (no-clipping).
• Noise distribution: The output perturbation baseline, in contrast, shows dramatically worse accuracy and inconsistency across runs.

Figure 2: Convergence profile (distance to optimum) for two polynomial approximation strategies (MM/LS) for MNIST Pair-1.

Figure 3: Distribution of $\geq 3\times$1-values at iteration 1000; shows initial concentration within the target interval.

Figure 4: Distribution of $\geq 3\times$2-values at iteration 1080; standard approach results in out-of-range divergence, while the barrier-augmented model remains strictly bounded.

Limitations and Theoretical Scope

The results rigorously cover convex (or barrier-modified strongly convex) objective functions and their surrogates. Analysis of deep, non-convex networks is left to future investigation, although the core mechanism of norm control by objective augmentation may extend more broadly, offering a pathway to scalable privacy-preserving non-convex optimization under FHE.

Implications and Future Directions

From a privacy engineering perspective, the methodology bridges the deployment gap for end-to-end encrypted and DP-ML in resource-constrained, untrusted settings such as finance and healthcare. The removal of heuristic “safety margins” and roundtrips for parameter tuning is particularly relevant for real-world scalability in non-interactive settings. Theoretically, the techniques for controlling polynomial approximation errors, parameter bounding, and non-clipping DP analysis strengthen the underpinnings of FHE-ML and may generalize to related privacy-preserving computational paradigms.

Potential future avenues include adaptation and analysis for non-convex frameworks (deep learning), extending multi-party scenarios, and further reducing polynomial and computational overhead without sacrificing privacy or statistical power.

Conclusion

This work provides a comprehensive theoretical and practical framework for efficient, privacy-preserving, and secure ML training under full data encryption, leveraging polynomial surrogates and eliminating gradient clipping. It delivers the first formal convergence and privacy guarantees for this setting, achieves strong empirical performance, and enables implementation of DP-ML over encrypted data in a manner practical for multi-party, non-interactive outsourced environments.

Reference:

"Revisiting ML Training under Fully Homomorphic Encryption: Convergence Guarantees, Differential Privacy, and Efficient Algorithms" (2605.27782)