Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions (2402.13647v1)

Published 21 Feb 2024 in cs.CL and cs.AI

Abstract: Unsupervised Text Style Transfer (UTST) has emerged as a critical task within the domain of NLP, aiming to transfer one stylistic aspect of a sentence into another style without changing its semantics, syntax, or other attributes. This task is especially challenging given the intrinsic lack of parallel text pairings. Among existing methods for UTST tasks, attention masking approach and LLMs are deemed as two pioneering methods. However, they have shortcomings in generating unsmooth sentences and changing the original contents, respectively. In this paper, we investigate if we can combine these two methods effectively. We propose four ways of interactions, that are pipeline framework with tuned orders; knowledge distillation from LLMs to attention masking model; in-context learning with constructed parallel examples. We empirically show these multi-way interactions can improve the baselines in certain perspective of style strength, content preservation and text fluency. Experiments also demonstrate that simply conducting prompting followed by attention masking-based revision can consistently surpass the other systems, including supervised text style transfer systems. On Yelp-clean and Amazon-clean datasets, it improves the previously best mean metric by 0.5 and 3.0 absolute percentages respectively, and achieves new SOTA results.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Style transformer: Unpaired text style transfer without disentangled latent representation. arXiv preprint arXiv:1905.05621.
  2. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335.
  3. Style transfer in text: Exploration and evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
  4. Multi-style transfer with discriminative feedback on disjoint corpus. arXiv preprint arXiv:2010.11578.
  5. Teacher-student architecture for knowledge learning: A survey. arXiv preprint arXiv:2210.17332.
  6. Toward controlled generation of text. In International conference on machine learning, pages 1587–1596. PMLR.
  7. Disentangled representation learning for non-parallel text style transfer. arXiv preprint arXiv:1808.04339.
  8. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
  9. Armanda Lewis. 2022. Multimodal large language models for inclusive collaboration learning tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 202–210, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
  10. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
  11. Delete, retrieve, generate: a simple approach to sentiment and style transfer. arXiv preprint arXiv:1804.06437.
  12. Dgst: a dual-generator network for text style transfer. arXiv preprint arXiv:2010.14557.
  13. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  14. A dual reinforcement learning framework for unsupervised text style transfer. arXiv preprint arXiv:1905.10060.
  15. Prompt-based editing for text style transfer. arXiv preprint arXiv:2301.11997.
  16. Politeness transfer: A tag and generate approach. arXiv preprint arXiv:2004.14257.
  17. Edit5: Semi-autoregressive text-editing with t5 warm-start. arXiv preprint arXiv:2205.12209.
  18. Text generation with text-editing models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts, pages 1–7, Seattle, United States. Association for Computational Linguistics.
  19. On text style transfer via style-aware masked language models. In Proceedings of the 16th International Natural Language Generation Conference, pages 362–374.
  20. Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771.
  21. Style transfer through back-translation. arXiv preprint arXiv:1804.09000.
  22. Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. arXiv preprint arXiv:1803.06535.
  23. Machel Reid and Victor Zhong. 2021. Lewis: Levenshtein editing for unsupervised text style transfer. arXiv preprint arXiv:2105.08206.
  24. A recipe for arbitrary text style transfer with large language models. arXiv preprint arXiv:2109.03910.
  25. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 837–848.
  26. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States. Association for Computational Linguistics.
  27. Large language models are not zero-shot communicators. arXiv preprint arXiv:2210.14986.
  28. Semi-supervised text style transfer: Cross projection in latent space. arXiv preprint arXiv:1909.11493.
  29. Style transfer from non-parallel text by cross-alignment. Advances in neural information processing systems, 30.
  30. Selective annotation makes language models better few-shot learners. arXiv pr arXiv:2209.01975.
  31. Transforming delete, retrieve, generate approach for controlled text style transfer. arXiv preprint arXiv:1908.09368.
  32. Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with small language models. arXiv preprint arXiv:2205.11503.
  33. Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with small language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2195–2222, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  34. Cat-llm: Prompting large language models with text style definition for chinese article-style transfer. arXiv preprint arXiv:2401.05707.
  35. R 33{}^{3}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT prompting: Review, rephrase and resolve for chain-of-thought reasoning in large language models under noisy context. arXiv preprint arXiv:2310.16535.
  36. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096–1103.
  37. Text style transferring via adversarial masking and styled filling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7654–7663.
  38. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597.
  39. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. arXiv preprint arXiv:1805.05181.
  40. Ni Xuanfan and Li Piji. 2023. A systematic evaluation of large language models for natural. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), pages 40–56.
  41. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
  42. Parallel data augmentation for formality style transfer. arXiv preprint arXiv:2005.07522.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Lei Pan (76 papers)
  2. Yunshi Lan (30 papers)
  3. Yang Li (1142 papers)
  4. Weining Qian (21 papers)
Citations (4)
View on Semantic Scholar