LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training (2401.04348v3)
Abstract: Paraphrases are texts that convey the same meaning while using different words or sentence structures. It can be used as an automatic data augmentation tool for many Natural Language Processing tasks, especially when dealing with low-resource languages, where data shortage is a significant problem. To generate a paraphrase in multilingual settings, previous studies have leveraged the knowledge from the machine translation field, i.e., forming a paraphrase through zero-shot machine translation in the same language. Despite good performance on human evaluation, those methods still require parallel translation datasets, thus making them inapplicable to languages that do not have parallel corpora. To mitigate that problem, we proposed the first unsupervised multilingual paraphrasing model, LAMPAT ($\textbf{L}$ow-rank $\textbf{A}$daptation for $\textbf{M}$ultilingual $\textbf{P}$araphrasing using $\textbf{A}$dversarial $\textbf{T}$raining), by which monolingual dataset is sufficient enough to generate a human-like and diverse sentence. Throughout the experiments, we found out that our method not only works well for English but can generalize on unseen languages as well. Data and code are available at https://github.com/VinAIResearch/LAMPAT.
- Bertsekas, D. P. 1982. Projected Newton Methods for Optimization Problems with Simple Constraints. SIAM Journal on Control and Optimization, 20(2): 221–246.
- Joint Copying and Restricted Generation for Paraphrase. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, 3152–3158. AAAI Press.
- Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuning. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10): 10535–10544.
- Cross-lingual Language Model Pretraining. In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
- Creutz, M. 2019. Open subtitles paraphrase corpus for six languages. LREC 2018 - 11th International Conference on Language Resources and Evaluation, (2005): 1364–1369.
- Towards robustness against natural language word substitutions. arXiv preprint arXiv:2107.13541.
- How should pre-trained language models be fine-tuned towards adversarial robustness? Advances in Neural Information Processing Systems, 34: 4356–4369.
- Duolingo. 2020. Data for the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE).
- Multilingual Whispers: Generating Paraphrases with Translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), 17–26. Hong Kong, China: Association for Computational Linguistics.
- Foundation, W. 2019. ACL 2019 Fourth Conference on Machine Translation (WMT19), Shared Task: Machine Translation of News.
- Human-Paraphrased References Improve Neural Machine Translation. In Proceedings of the Fifth Conference on Machine Translation, 1183–1192. Online: Association for Computational Linguistics.
- Improving the Robustness of Question Answering Systems to Question Paraphrasing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 6065–6075. Florence, Italy: Association for Computational Linguistics.
- Explaining and Harnessing Adversarial Examples. arXiv:1412.6572.
- Zero-Shot Paraphrase Generation with Multilingual Language Models. arXiv:1911.03597.
- Factorising Meaning and Form for Intent-Preserving Paraphrasing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 1405–1418. Online: Association for Computational Linguistics.
- Parameter-Efficient Transfer Learning for NLP. In Chaudhuri, K.; and Salakhutdinov, R., eds., Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, 2790–2799. PMLR.
- LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
- Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping. arXiv:2102.11343.
- Krippendorff, K. 1970. Estimating the Reliability, Systematic Error and Random Error of Interval Data. Educational and Psychological Measurement, 30: 61 – 70.
- The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045–3059. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
- Pre-training via Paraphrasing. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems, volume 33, 18470–18481. Curran Associates, Inc.
- Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 4582–4597. Online: Association for Computational Linguistics.
- P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 61–68. Dublin, Ireland: Association for Computational Linguistics.
- Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. arXiv:1704.03976.
- Unsupervised Paraphrasing with Pretrained Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 5136–5150. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
- Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311–318. Philadelphia, Pennsylvania, USA: Association for Computational Linguistics.
- Adversarial Training for Commonsense Inference. In Proceedings of the 5th Workshop on Representation Learning for NLP, 55–60. Online: Association for Computational Linguistics.
- Adversarial training for free! In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
- On the Evaluation Metrics for Paraphrase Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 3178–3190. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
- A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, 223–231. Cambridge, Massachusetts, USA: Association for Machine Translation in the Americas.
- Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 90–121. Online: Association for Computational Linguistics.
- Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity. In Proceedings of the Fifth Conference on Machine Translation, 561–570. Online: Association for Computational Linguistics.
- PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3687–3692. Hong Kong, China: Association for Computational Linguistics.
- You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle. In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d'Alché-Buc, F.; Fox, E.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
- BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations.