Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unsupervised Text Embedding Space Generation Using Generative Adversarial Networks for Text Synthesis (2306.17181v4)

Published 19 Jun 2023 in cs.CL and cs.LG

Abstract: Generative Adversarial Networks (GAN) is a model for data synthesis, which creates plausible data through the competition of generator and discriminator. Although GAN application to image synthesis is extensively studied, it has inherent limitations to natural language generation. Because natural language is composed of discrete tokens, a generator has difficulty updating its gradient through backpropagation; therefore, most text-GAN studies generate sentences starting with a random token based on a reward system. Thus, the generators of previous studies are pre-trained in an autoregressive way before adversarial training, causing data memorization that synthesized sentences reproduce the training data. In this paper, we synthesize sentences using a framework similar to the original GAN. More specifically, we propose Text Embedding Space Generative Adversarial Networks (TESGAN) which generate continuous text embedding spaces instead of discrete tokens to solve the gradient backpropagation problem. Furthermore, TESGAN conducts unsupervised learning which does not directly refer to the text of the training data to overcome the data memorization issue. By adopting this novel method, TESGAN can synthesize new sentences, showing the potential of unsupervised learning for text synthesis. We expect to see extended research combining LLMs with a new perspective of viewing text as an continuous space.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Flamingo: a visual language model for few-shot learning.
  2. Jointly measuring diversity and quality in text generation models. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 90–98, Minneapolis, Minnesota. Association for Computational Linguistics.
  3. Data augmentation generative adversarial networks.
  4. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214–223. PMLR.
  5. Gan augmentation: Augmenting training data using generative adversarial networks.
  6. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10–21, Berlin, Germany. Association for Computational Linguistics.
  7. Language gans falling short. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  8. Maximum-likelihood augmented discrete generative adversarial networks.
  9. Adversarial text generation via feature-mover’s distance. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 4671–4682, Red Hook, NY, USA. Curran Associates Inc.
  10. Scaling instruction-finetuned language models.
  11. Instructblip: Towards general-purpose vision-language models with instruction tuning.
  12. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  13. TILGAN: Transformer-based implicit latent GAN for diverse and coherent text generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4844–4858, Online. Association for Computational Linguistics.
  14. Maskgan: Better text generation via filling in the _______. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
  15. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
  16. Graves, Alex. 2013. Generating sequences with recurrent neural networks. CoRR, abs/1308.0850.
  17. Long text generation via adversarial training with leaked information. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  18. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  19. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
  20. Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735–1780.
  21. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
  22. Alias-free generative adversarial networks.
  23. A style-based generator architecture for generative adversarial networks.
  24. Kingma, Diederik P. and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  25. Kingma, Diederik P. and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
  26. Kullback, S. and R. A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 – 86.
  27. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  28. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
  29. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986–995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  30. Lin, Chin-Yew. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  31. Adversarial ranking for language generation. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 3158–3168, Red Hook, NY, USA. Curran Associates Inc.
  32. Dying relu and initialization: Theory and numerical examples. ArXiv, abs/1903.06733.
  33. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  34. Rectifier nonlinearities improve neural network acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing.
  35. Training language gans from scratch. In Neural Information Processing Systems.
  36. Relgan: Relational generative adversarial networks for text generation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
  37. Training language models to follow instructions with human feedback.
  38. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  39. Language generation with recurrent generative adversarial networks without pre-training. CoRR, abs/1706.01399.
  40. Unsupervised representation learning with deep convolutional generative adversarial networks.
  41. Language models are unsupervised multitask learners.
  42. Data augmentation using generative adversarial networks (cyclegan) to improve generalizability in ct segmentation tasks. Scientific Reports, 9.
  43. Relational recurrent neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 7310–7321, Red Hook, NY, USA. Curran Associates Inc.
  44. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  45. Llama: Open and efficient foundation language models.
  46. On data augmentation for gan training. Trans. Img. Proc., 30:1882–1897.
  47. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  48. Reltanh: An activation function with vanishing gradient resistance for sae-based dnns and its application to rotating machinery fault diagnosis. Neurocomputing, 363:88–98.
  49. Finetuned language models are zero-shot learners. CoRR, abs/2109.01652.
  50. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721, Lisbon, Portugal. Association for Computational Linguistics.
  51. Williams, Ronald J. and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270–280.
  52. Seqgan: Sequence generative adversarial nets with policy gradient. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1).
  53. Adversarial feature matching for text generation. In International Conference on Machine Learning, pages 4006–4015. PMLR.
  54. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 1097–1100, New York, NY, USA. Association for Computing Machinery.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Jun-Min Lee (2 papers)
  2. Tae-Bin Ha (1 paper)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now