CryptoDL: Deep Neural Networks over Encrypted Data

Abstract: Machine learning algorithms based on deep neural networks have achieved remarkable results and are being extensively used in different domains. However, the machine learning algorithms requires access to raw data which is often privacy sensitive. To address this issue, we develop new techniques to provide solutions for running deep neural networks over encrypted data. In this paper, we develop new techniques to adopt deep neural networks within the practical limitation of current homomorphic encryption schemes. More specifically, we focus on classification of the well-known convolutional neural networks (CNN). First, we design methods for approximation of the activation functions commonly used in CNNs (i.e. ReLU, Sigmoid, and Tanh) with low degree polynomials which is essential for efficient homomorphic encryption schemes. Then, we train convolutional neural networks with the approximation polynomials instead of original activation functions and analyze the performance of the models. Finally, we implement convolutional neural networks over encrypted data and measure performance of the models. Our experimental results validate the soundness of our approach with several convolutional neural networks with varying number of layers and structures. When applied to the MNIST optical character recognition tasks, our approach achieves 99.52\% accuracy which significantly outperforms the state-of-the-art solutions and is very close to the accuracy of the best non-private version, 99.77\%. Also, it can make close to 164000 predictions per hour. We also applied our approach to CIFAR-10, which is much more complex compared to MNIST, and were able to achieve 91.5\% accuracy with approximation polynomials used as activation functions. These results show that CryptoDL provides efficient, accurate and scalable privacy-preserving predictions.

• The paper introduces a novel method to approximate CNN activation functions using low-degree polynomials, enabling efficient computations on encrypted data.
• It implements the framework on standard datasets like MNIST and CIFAR-10, achieving 99.52% accuracy and processing 164,000 predictions per hour.
• The work paves the way for privacy-preserving AI in sensitive fields such as healthcare and finance by maintaining data confidentiality during inference.

Overview of CryptoDL: Deep Neural Networks over Encrypted Data

The paper presents CryptoDL, a framework designed to facilitate the deployment of convolutional neural networks (CNNs) on encrypted data via homomorphic encryption (HE). This approach addresses the growing concern of privacy in machine learning applications, where access to raw data may pose security threats. Central to this research is the use of practical HE schemes that allow computations on encrypted data without the need to decrypt it, thereby preserving data privacy.

Methodology and Contributions

CryptoDL's primary innovation lies in approximating the activation functions (ReLU, Sigmoid, Tanh) used in CNNs with low-degree polynomials. These polynomial approximations enable the computation within HE constraints, which traditionally limit operations to addition and multiplication. The authors propose a robust method for finding polynomial approximations that minimizes the error introduced by replacing standard activation functions in neural networks with polynomial functions.

Key contributions of CryptoDL include:

• Theoretical foundation for polynomial approximation and demonstration of methods to approximate activation functions with low-degree polynomials.
• Implementation and training of CNNs using these polynomial approximations, with experimental validation on standard datasets including MNIST and CIFAR-10.
• Validation of the scalability and accuracy of CryptoDL, achieving 99.52% accuracy on MNIST while enabling efficient encrypted data predictions at 164,000 predictions per hour.

Results and Implications

The experimental results underscore the feasibility of using CryptoDL for accurate and efficient privacy-preserving machine learning tasks. The paper reports that CryptoDL achieves competitive accuracy rates, comparable with non-private implementations, demonstrating the potential of HE to support machine learning applications while ensuring data privacy.

This work has significant implications for both theoretical and practical applications in AI:

• Theoretical Implications: The research advances the understanding of how polynomial approximations can be utilized within deep learning architectures, especially regarding how neural networks can be designed to operate seamlessly on encrypted data.
• Practical Implications: Practically, this framework lays the groundwork for deploying machine learning models in sensitive domains, such as healthcare and finance, where data privacy is paramount. The ability to perform predictions on encrypted data without compromising privacy is a significant stride toward integrating AI into privacy-sensitive applications.

Future Directions

The study opens several avenues for future research developments. It suggests potential explorations into other types of neural network architectures beyond CNNs that could benefit from similar techniques in HE. Additionally, the possibility of developing more sophisticated polynomial approximation techniques that reduce computation overhead while maintaining high model accuracy remains a topic of interest. Further development could involve optimizing the CNN architectures to reduce computational cost while using polynomial approximations within HE schemes more effectively.

In conclusion, CryptoDL represents a significant step towards privacy-preserving AI, offering valuable insights and practical solutions for implementing machine learning over encrypted datasets. Its successful application to standard datasets supports the case for widespread adoption in an increasingly data-driven yet privacy-conscious world.