Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DP-NMT: Scalable Differentially-Private Machine Translation (2311.14465v2)

Published 24 Nov 2023 in cs.CL

Abstract: Neural machine translation (NMT) is a widely popular text generation task, yet there is a considerable research gap in the development of privacy-preserving NMT models, despite significant data privacy concerns for NMT systems. Differentially private stochastic gradient descent (DP-SGD) is a popular method for training machine learning models with concrete privacy guarantees; however, the implementation specifics of training a model with DP-SGD are not always clarified in existing models, with differing software libraries used and code bases not always being public, leading to reproducibility issues. To tackle this, we introduce DP-NMT, an open-source framework for carrying out research on privacy-preserving NMT with DP-SGD, bringing together numerous models, datasets, and evaluation metrics in one systematic software package. Our goal is to provide a platform for researchers to advance the development of privacy-preserving NMT systems, keeping the specific details of the DP-SGD algorithm transparent and intuitive to implement. We run a set of experiments on datasets from both general and privacy-related domains to demonstrate our framework in use. We make our framework publicly available and welcome feedback from the community.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI’16, page 265–283, USA. USENIX Association.
  2. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318.
  3. Large-scale differentially private BERT. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6481–6491, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  4. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in neural information processing systems, 31.
  5. Privacy enabled financial text classification using differential privacy and federated learning. In Proceedings of the Third Workshop on Economics and Natural Language Processing, pages 50–55, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  6. Bounds on the sample complexity for private learning and private data release. Machine learning, 94:401–437.
  7. Findings of the 2016 conference on machine translation (wmt16). In First conference on machine translation, pages 131–198. Association for Computational Linguistics.
  8. JAX: composable transformations of Python+NumPy programs.
  9. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 2280–2292.
  10. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650.
  11. Federated nearest neighbor machine translation. In The Eleventh International Conference on Learning Representations.
  12. Cynthia Dwork and Aaron Roth. 2013. The Algorithmic Foundations of Differential Privacy. Foundations and Trends® in Theoretical Computer Science, 9(3-4):211–407.
  13. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pages 51–60. IEEE.
  14. Amplification by shuffling: From local to central differential privacy via anonymity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 2468–2479. SIAM.
  15. Hiding among the clones: A simple and nearly optimal analysis of privacy amplification by shuffling. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS), pages 954–964. IEEE.
  16. Ivan Habernal. 2022. How reparametrization trick broke differentially-private text representation learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 771–777, Dublin, Ireland. Association for Computational Linguistics.
  17. The impact of differential privacy on group disparity mitigation. In Proceedings of the Fourth Workshop on Privacy in Natural Language Processing, pages 12–12.
  18. Flax: A neural network library and ecosystem for JAX.
  19. Membership inference attacks on sequence-to-sequence models: Is my data in your machine translation system? Transactions of the Association for Computational Linguistics, 8:49–63.
  20. Learning and Evaluating a Differentially Private Pre-trained Language Model. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1178–1189, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  21. Differential privacy: An economic method for choosing epsilon. In 2014 IEEE 27th Computer Security Foundations Symposium, pages 398–410. IEEE.
  22. Timour Igamberdiev and Ivan Habernal. 2022. Privacy-Preserving Graph Convolutional Networks for Text Classification. In Proceedings of the Language Resources and Evaluation Conference, pages 338–350, Marseille, France. European Language Resources Association.
  23. Timour Igamberdiev and Ivan Habernal. 2023. DP-BART for privatized text rewriting under local differential privacy. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13914–13934, Toronto, Canada. Association for Computational Linguistics.
  24. Paweł Kamocki and Jim O’Regan. 2016. Privacy issues in online machine translation services-european perspective. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4458–4462.
  25. What can we learn privately? SIAM Journal on Computing, 40(3):793–826.
  26. Differential privacy in natural language processing the story so far. In Proceedings of the Fourth Workshop on Privacy in Natural Language Processing, pages 1–11, Seattle, United States. Association for Computational Linguistics.
  27. Jaewoo Lee and Chris Clifton. 2011. How much is enough? choosing ε𝜀\varepsilonitalic_ε for differential privacy. In International Conference on Information Security, pages 325–340. Springer.
  28. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  29. Large language models can be strong differentially private learners. In International Conference on Learning Representations.
  30. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.
  31. Differentially private language models for secure data sharing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4860–4873, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  32. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR.
  33. Findings of the wmt 2022 biomedical translation shared task: Monolingual clinical case reports. In WMT22-Seventh Conference on Machine Translation, pages 694–723.
  34. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  35. Training mixed-domain translation models via federated learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2576–2586.
  36. Training text-to-text transformers with privacy guarantees. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2182–2193, Dublin, Ireland. Association for Computational Linguistics.
  37. How to dp-fy ml: A practical guide to machine learning with differential privacy. Journal of Artificial Intelligence Research, 77:1113–1201.
  38. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  39. Designing the business conversation corpus. In Proceedings of the 6th Workshop on Asian Translation, pages 54–61, Hong Kong, China. Association for Computational Linguistics.
  40. Communication-efficient federated learning for neural machine translation. arXiv preprint arXiv:2112.06135.
  41. One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7340–7353, Abu Dhabi, UAE.
  42. Selective differential privacy for language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2848–2859, Seattle, United States. Association for Computational Linguistics.
  43. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE.
  44. Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization. In Advances in Neural Information Processing Systems, volume 34, pages 26409–26421. Curran Associates, Inc.
  45. Attention is all you need. Advances in neural information processing systems, 30.
  46. Modeling without sharing privacy: Federated neural machine translation. In Web Information Systems Engineering–WISE 2021: 22nd International Conference on Web Information Systems Engineering, WISE 2021, Melbourne, VIC, Australia, October 26–29, 2021, Proceedings, Part I 22, pages 216–223. Springer.
  47. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1226–1235. PMLR.
  48. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
  49. To share or not to share: What risks would laypeople accept to give sensitive data to differentially-private nlp systems? arXiv preprint arXiv:2307.06708.
  50. Adaptive differential privacy for language model training. In Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022), pages 21–26, Dublin, Ireland. Association for Computational Linguistics.
  51. Mitigating data poisoning in text classification with differential privacy. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4348–4356, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  52. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
  53. Ying Yin and Ivan Habernal. 2022. Privacy-preserving models for legal natural language processing. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 172–183, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  54. Opacus: User-Friendly Differential Privacy Library in PyTorch. arXiv preprint.
  55. Differentially private fine-tuning of language models. In International Conference on Learning Representations.
  56. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now