MPCFormer: fast, performant and private Transformer inference with MPC (2211.01452v2)

Abstract: Enabling private inference is crucial for many cloud inference services that are based on Transformer models. However, existing private inference solutions can increase the inference latency by more than 60x or significantly compromise the inference quality. In this paper, we design the framework MPCFORMER as a practical solution, using Secure Multi-Party Computation (MPC) and Knowledge Distillation (KD). Through extensive evaluations, we show that MPCFORMER significantly speeds up Transformer inference in MPC settings while achieving similar ML performance to the input model. On the IMDb dataset, it achieves similar performance to BERTBASE, while being 5.3x faster. On the GLUE benchmark, it achieves 97% performance of BERTBASE with a 2.2x speedup. MPCFORMER remains effective with different trained Transformer weights such as ROBERTABASE and larger models including BERTLarge. Code is available at https://github.com/MccRee177/MPCFormer.

An Analysis of MPCFormer: Enhancing Transformer Inference with Secure Multi-Party Computation

The paper presents MPCFormer, a framework aimed at ensuring efficient, performant, and private inference of Transformer models utilizing Secure Multi-Party Computation (MPC). The motivation stems from the need for privacy-preserving inference in cloud-based services, which often compromises on either latency or accuracy. The authors propose MPCFormer as a solution that leverages both MPC and Knowledge Distillation (KD) to achieve substantial reductions in latency while maintaining performance levels akin to the original Transformer models used in inference tasks.

Core Contributions and Methodology

1. Framework Design: MPCFormer is designed to convert pre-trained Transformers into a form that facilitates fast inference under MPC settings. This transformation is achieved by replacing computational bottlenecks within Transformer models with MPC-friendly approximations, thereby reducing the computing load imposed by secure computations.
2. Use of MPC-friendly Approximations: The authors emphasize the replacement of complex operations, such as the GeLU activation and Softmax functions, which significantly decelerate inference under MPC conditions. Through approximations like quadratic substitutes, MPCFormer ensures that critical operations are executed faster without fully sacrificing accuracy.
3. Implementation of Knowledge Distillation: KD plays a crucial role in ensuring the MPC-friendly model retains high accuracy. By aligning intermediate representations between the original and approximated models, MPCFormer mitigates the loss of performance that typically follows aggressive approximation techniques.
4. Empirical Validation: Evaluations are conducted using benchmarks such as IMDb and GLUE, where MPCFormer exhibits considerable speedup (up to 5.3×) with marginal compromise on accuracy relative to standard Transformer models.
5. Compatibility and Portability: The framework is explicitly designed to be adaptable to various Transformer architectures and to support different MPC systems, showcased by its effectiveness across BERT variants and distinct datasets.

Results and Implications

• Performance Metrics: On the IMDb dataset, for example, MPCFormer achieves performance comparable to BERT_BASE with a notable 5.3× increase in processing speed. Similarly, on the GLUE benchmark, it maintains 97% of BERT_BASE's performance level while improving speed by 2.2×.
• Broader Applicability: Tests reveal MPCFormer’s adaptability to models beyond BERT_BASE, such as RoBERTa_BASE and larger constructs like BERT_LARGE, highlighting its utility in diverse operational contexts.
• Potential Impact: By effectively balancing the trade-offs between inference latency and model accuracy, MPCFormer can significantly enhance the practicality of deploying Transformer models in privacy-sensitive applications across various domains, including finance, healthcare, and defense.

Prospective Developments

While MPCFormer brings substantial progress to private inference technologies, future work could delve into adapting the framework for different MPC systems through more extensive empirical and theoretical analyses. Furthermore, exploring the construction of smaller and more efficient student models beyond mere architectural alterations could provide additional avenues for optimization.

In conclusion, MPCFormer stands as a significant advancement in the field of secure deep learning, offering a sophisticated balance of speed, privacy, and accuracy that has vital implications for both current and future applications of AI in privacy-oriented environments.