CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU (2104.10949v1)

Abstract: We introduce CryptGPU, a system for privacy-preserving machine learning that implements all operations on the GPU (graphics processing unit). Just as GPUs played a pivotal role in the success of modern deep learning, they are also essential for realizing scalable privacy-preserving deep learning. In this work, we start by introducing a new interface to losslessly embed cryptographic operations over secret-shared values (in a discrete domain) into floating-point operations that can be processed by highly-optimized CUDA kernels for linear algebra. We then identify a sequence of "GPU-friendly" cryptographic protocols to enable privacy-preserving evaluation of both linear and non-linear operations on the GPU. Our microbenchmarks indicate that our private GPU-based convolution protocol is over 150x faster than the analogous CPU-based protocol; for non-linear operations like the ReLU activation function, our GPU-based protocol is around 10x faster than its CPU analog. With CryptGPU, we support private inference and private training on convolutional neural networks with over 60 million parameters as well as handle large datasets like ImageNet. Compared to the previous state-of-the-art, when considering large models and datasets, our protocols achieve a 2x to 8x improvement in private inference and a 6x to 36x improvement for private training. Our work not only showcases the viability of performing secure multiparty computation (MPC) entirely on the GPU to enable fast privacy-preserving machine learning, but also highlights the importance of designing new MPC primitives that can take full advantage of the GPU's computing capabilities.

• The paper introduces a GPU-based cryptographic framework that leverages CUDA-optimized secret sharing to efficiently perform MPC protocols for privacy-preserving machine learning.
• Performance benchmarks demonstrate up to a 150× speed-up in private GPU-based convolutions and notable gains for non-linear operations like ReLU activation.
• The system scales to large CNNs with over 60 million parameters and datasets such as ImageNet, paving the way for more adaptive, GPU-accelerated privacy-preserving solutions.

Fast Privacy-Preserving Machine Learning on the GPU

The paper "Fast Privacy-Preserving Machine Learning on the GPU" presents a system designed to perform privacy-preserving machine learning entirely on graphics processing units (GPUs). The system addresses the significant computational overhead of secure multiparty computation (MPC) protocols used in privacy-preserving machine learning, especially when applied to large-scale deep learning models and datasets. By leveraging the parallel processing capabilities of GPUs, the authors propose an approach that aims to accelerate the execution of cryptographic protocols needed for privacy-preserving machine learning.

Core Contributions

1. GPU-based Cryptographic Framework: The system introduces a novel interface enabling cryptographic operations over secret-shared values to be conducted on GPUs using floating-point operations. This results in efficient execution leveraging highly-optimized CUDA kernels designed for linear algebra.
2. Performance Improvements: The private GPU-based convolution protocol demonstrates a substantial performance gain, exhibiting over a 150× speed-up compared to CPU-based approaches. Similarly, GPU-optimized protocols for non-linear operations like ReLU activation function show a 10× performance improvement over their CPU counterparts.
3. Scalable Privacy-preserving Computation: The system supports private inference and private training on convolutional neural networks (CNNs) with over 60 million parameters, handling large-scale datasets such as ImageNet. The protocols achieve a 2× to 8× improvement in private inference and a 6× to 36× improvement in private training over previous state-of-the-art methods.
4. Microbenchmark Analysis: The paper provides systematic evaluations, showcasing that using GPUs can significantly reduce computation time for privacy-preserving deep learning. The results indicate that even computationally intensive non-linear functions can benefit from GPU acceleration through appropriately designed cryptographic protocols.

Technical Approach

• Secret Sharing Scheme: Utilizes replicated secret sharing, a common technique in MPC where inputs are divided into multiple secret shares distributed among several parties. This ensures that inputs remain private throughout the computation process.
• Fixed-point Arithmetic on the GPU: Addresses how cryptographic operations, typically performed over discrete fields or rings, can be efficiently mapped onto floating-point arithmetic compatible with existing GPU libraries.
• Cryptographic Protocols: The authors outline protocols for numerous foundational operations within deep learning models, including linear calculations, convolutions, matrix multiplications, and activation functions, all optimized for execution on GPUs.

Implications and Future Directions

The presented system has significant implications for the scalability and efficiency of privacy-preserving machine learning. By demonstrating the substantial performance benefits of GPU utilization, the work lays the groundwork for future privacy-preserving systems that can support increasingly complex models and larger datasets. Critical avenues for further research include:

• Enhancing Protocol Efficiency: Developing even more GPU-friendly cryptographic protocols to reduce the remaining performance gap between secure and plaintext computations.
• Extending Features: Adding support for additional deep learning features often used in practice, such as batch normalization, which remains challenging to perform securely and efficiently.
• Adaptive Systems: Exploring dynamic approaches that balance computational loads across multiple GPUs or hybrid CPU-GPU systems to further increase scalability and efficiency.

Overall, the paper's contributions represent a significant step toward practical, scalable privacy-preserving machine learning solutions capable of handling the demands of modern deep learning applications.