QUOTIENT: Two-Party Secure Neural Network Training and Prediction (1907.03372v1)

Abstract: Recently, there has been a wealth of effort devoted to the design of secure protocols for machine learning tasks. Much of this is aimed at enabling secure prediction from highly-accurate Deep Neural Networks (DNNs). However, as DNNs are trained on data, a key question is how such models can be also trained securely. The few prior works on secure DNN training have focused either on designing custom protocols for existing training algorithms, or on developing tailored training algorithms and then applying generic secure protocols. In this work, we investigate the advantages of designing training algorithms alongside a novel secure protocol, incorporating optimizations on both fronts. We present QUOTIENT, a new method for discretized training of DNNs, along with a customized secure two-party protocol for it. QUOTIENT incorporates key components of state-of-the-art DNN training such as layer normalization and adaptive gradient methods, and improves upon the state-of-the-art in DNN training in two-party computation. Compared to prior work, we obtain an improvement of 50X in WAN time and 6% in absolute accuracy.

• The paper introduces QUOTIENT, a method enabling secure two-party neural network training and prediction using cryptographic techniques, ternary weights, and fixed-point arithmetic.
• QUOTIENT achieves significant efficiency gains, reducing WAN time by up to 50x and improving accuracy by ~6% over prior secure methods like SecureML.
• This work demonstrates the feasibility of scalable, privacy-preserving DNN training and prediction for sensitive applications like healthcare and finance.

Summary of QUOTIENT: Two-Party Secure Neural Network Training and Prediction

The paper presents QUOTIENT, a methodology designed to enhance the security and efficiency of training deep neural networks (DNNs) in a two-party computational framework. The core focus of the research is to address privacy concerns during neural network training and prediction by employing cryptographic techniques, specifically two-party computation (2PC). This paper circumvents the traditional performance limitations of secure computation protocols when applied to real-world machine learning tasks and proposes a novel integrated approach.

Methodological Innovations

1. Two-Party Secure Computation Framework: The research leverages recent advancements in secure multi-party computation (MPC) to create protocols that allow for the secure evaluation of neural network functions without compromising data privacy. The use of techniques such as Oblivious Transfer (OT) and its optimized version, Correlated Oblivious Transfer (COT), is central to this approach.
2. Ternary Weights and Fixed-Point Arithmetic: One of the unique aspects of QUOTIENT is the use of ternary weights $[-1, 0, 1]$, which simplifies computation and reduces the communication burden. The paper also employs fixed-point arithmetic to stabilize numerical operations and ensure precision, which reduces the computational overhead typically associated with floating-point arithmetic in secure settings.
3. Optimization Algorithms: The authors developed customized optimization procedures, including an adaptation of AMSgrad, to improve convergence rates and accuracy within secure computation constraints. Their secure AMSgrad optimizer outperforms standard stochastic gradient descent (SGD) methods in terms of convergence speed.

Key Numerical Results

The proposed QUOTIENT framework demonstrates substantial efficiency gains, achieving up to a 50x reduction in WAN time required for computation compared to previous approaches. It also secures an absolute accuracy improvement of approximately 6% over state-of-the-art methods such as SecureML when training on dataset examples like MNIST.

Theoretical and Practical Implications

The paper provides a significant step forward in bridging cryptographic techniques with machine learning applications. It indicates that secure training of complex DNN architectures, including convolutional and residual networks, is achievable without significant compromises in performance or accuracy. In practical terms, this enables scalable deployment of machine learning algorithms in environments where data confidentiality is critical, such as healthcare and finance.

Future Directions

The research opens avenues for further explorations into more specialized cryptographic constructs that could further optimize secure DNN operations. Potential enhancements could include reducing the communication overhead in WAN settings and expanding the framework to multi-party scenarios beyond two parties.

In conclusion, QUOTIENT illustrates a sophisticated blending of cryptographic security measures within machine learning model development, promising enhanced privacy without forfeiting computational efficiency. The integration of specialized algorithmic adjustments with secure computation protocols marks a significant advance in privacy-preserving AI applications.