A Fast, Performant, Secure Distributed Training Framework For Large Language Model (2401.09796v2)
Abstract: The distributed (federated) LLM is an important method for co-training the domain-specific LLM using siloed data. However, maliciously stealing model parameters and data from the server or client side has become an urgent problem to be solved. In this paper, we propose a secure distributed LLM based on model slicing. In this case, we deploy the Trusted Execution Environment (TEE) on both the client and server side, and put the fine-tuned structure (LoRA or embedding of P-tuning v2) into the TEE. Then, secure communication is executed in the TEE and general environments through lightweight encryption. In order to further reduce the equipment cost as well as increase the model performance and accuracy, we propose a split fine-tuning scheme. In particular, we split the LLM by layers and place the latter layers in a server-side TEE (the client does not need a TEE). We then combine the proposed Sparsification Parameter Fine-tuning (SPF) with the LoRA part to improve the accuracy of the downstream task. Numerous experiments have shown that our method guarantees accuracy while maintaining security.
- “Peft: State-of-the-art parameter-efficient fine-tuning methods,” https://github.com/huggingface/peft, 2022.
- “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations, 2022.
- “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks,” arXiv preprint arXiv:2110.07602, 2021.
- “Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,” 2023.
- “Towards building the federated gpt: Federated instruction tuning,” arXiv preprint arXiv:2305.05644, 2023.
- “A survey on security and privacy of federated learning,” Future Generation Computer Systems, vol. 115, pp. 619–640, 2021.
- “Beyond inferring class representatives: User-level privacy leakage from federated learning,” in IEEE INFOCOM 2019-IEEE conference on computer communications. IEEE, 2019, pp. 2512–2520.
- “Privacy-preserving deep learning via additively homomorphic encryption,” IEEE Transactions on Information Forensics and Security, vol. 13, no. 5, pp. 1333–1345, 2018.
- “Slalom: Fast, verifiable and private execution of neural networks in trusted hardware,” arXiv preprint arXiv:1806.03287, 2018.
- “Intel® software guard extensions (intel® sgx) support for dynamic memory management inside an enclave,” in Proceedings of the Hardware and Architectural Support for Security and Privacy 2016, pp. 1–9. 2016.
- “Intel. intel trust domain extensions,” https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/documentation.html.
- “cpsgd: Communication-efficient and differentially-private distributed sgd,” Advances in Neural Information Processing Systems, vol. 31, 2018.
- “Securenn: 3-party secure computation for neural network training.,” Proc. Priv. Enhancing Technol., vol. 2019, no. 3, pp. 26–49, 2019.
- “Amd memory encryption,” White paper, p. 13, 2016.
- Tiago Alves, “Trustzone: Integrated hardware and software security,” Information Quarterly, vol. 3, pp. 18–24, 2004.
- “Model protection: Real-time privacy-preserving inference service for model privacy at the edge,” IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 6, pp. 4270–4284, 2021.
- “Model stealing attacks against inductive graph neural networks,” in 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022, pp. 1175–1192.
- “Cblue: A chinese biomedical language understanding evaluation benchmark,” arXiv preprint arXiv:2106.08087, 2021.
- “Promptcblue,” https://github.com/michael-wzhu/PromptCBLUE, 2023.
- “GLM-130b: An open bilingual pre-trained model,” in The Eleventh International Conference on Learning Representations (ICLR), 2023.
Sponsored by Paperpile, the PDF & BibTeX manager trusted by top AI labs.
Get 30 days freePaper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.