Text Generation by a GAN-based Ensemble Approach
Subject Areas : electrical and computer engineering
Ehsan
Montahaie
1
Mahdieh
Soleymani Baghshah
2
Keywords: Text generationgenerative modelGANsensemble learning,
Abstract :
Text generation is one of the important problems in Natural Language Processing field. The former methods for text generation that are based on language modeling by the teacher forcing approach encounter the problem of discrepancy between the training and test phases and also employing an inappropriate objective (i.e., Maximum Likelihood estimation) for generation. In the past years, Generative Adversarial Networks (GANs) have achieved much popularity due to their capabilities in image generation. These networks have also attracted attention for sequence generation in the last few years. However, since text sequences are discrete, GANs cannot be easily employed for text generation, and new approaches like Reinforcement Learning and approximation have been utilized for this purpose. Furthermore, the instability problem of GANs training causes new challenges. In this paper, a new GAN-based ensemble method is proposed for sequence generation problem. The idea of the proposed method is based on the ratio estimation which enables the model to overcome the problem of discreteness in data. Also, the proposed method is more stable than the other GAN-based methods. It also should be noted that the exposure bias problem of teacher forcing approach does not exist in the proposed method. Experiments show the superiority of the proposed method to previous GAN-based methods for text generation.
[1] F. Huszar, How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? Computing Research Repository (CoRR), 2015. abs/1511.05101.
[2] I. J. Goodfellow, et al., "Generative adversarial nets," in Proc. Advances in Neural Information Processing Systems 27: Annual Conf. on Neural Information Processing Systems, vol. 2, pp. 2672-2680, Montreal, Canada, Dec. 2014.
[3] I. J. Goodfellow, NIPS 2016 Tutorial: Generative Adversarial Networks, Computing Research Repository (CoRR), 2017. abs/1701.00160.
[4] L. Yu, W. Zhang, J. Wang, and Y. Yu, "SeqGAN: sequence generative adversarial nets with policy gradient," in Proc. of the 31st AAAI Conf. on Artificial Intelligence, pp. 2852-2858, San Francisco, CA, USA, Feb. 2017.
[5] S. Bengio, O. Vinyals, N. Jaitly, N. Shazeer, "Scheduled sampling for sequence prediction with recurrent neural networks," Advances in Neural Information Processing Systems, vol. 1, pp. 1171-1179, Montreal, Canada, 7-12 Dec. 2015.
[6] M. J. Kusner and J. M. Hernandez-Lobato, GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution, arXiv e-prints, 2016: p. arXiv:1611.04051-arXiv:1611.04051.
[7] E. Jang, S. Gu, and B. Poole, "Categorical reparameterization with gumbel-softmax," in Proc. Int. Conf. on Learning Representations, ICLR’17, 12 pp., Toulon, France, 24-26 Apr. 2017.
[8] C. J. Maddison, A. Mnih, and Y. W. The, "The concrete distribution: a continuous relaxation of discrete random variables," in Proc. 5th Int. Conf. on Learning Representations, ICLR’17, 20 pp., Toulon, France, 24-26 Apr. 2017.
[9] A. M. Lamb, et al., Professor Forcing: A New Algorithm for Training Recurrent Networks, in Advances in Neural Information Processing Systems 29, D. D. Lee, et al., Editors. 2016, Curran Associates, Inc. pp. 4601-4609.
[10] Y. Zhang, et al., "Adversarial feature matching for text generation," in Proc. of the 34th Int. Conf. on Machine Learning, ICML’17, pp. 4006-4015, Sydney, Australia, Aug. 2017.
[11] G. L. Guimaraes, et al., Objective-Reinforced Generative Adversarial Networks (ORGAN) for Sequence Generation Models. CoRR, 2017. abs/1705.10843.
[12] K. Lin, et al., "Adversarial ranking for language generation," Advances in Neural Information Processing Systems, pp. 3158-3168, Long Beach, CA, USA, 4-9 Dec. 17.
[13] J. Guo, et al., "Long text generation via adversarial training with leaked information," in Proc. of the 32nd AAAI Conf. on Artificial Intelligence, AAAI’18, the 30th Innovative Applications of Artificial Intelligence, IAAI’18, and the 8th AAAI Symp. on Educational Advances in Artificial Intelligence, EAAI’18, pp. 5141-5148, New Orleans, Louisiana, USA, Feb. 2018.
[14] T. Che, et al., Maximum-Likelihood Augmented Discrete Generative Adversarial Networks, arXiv preprint arXiv:1702.07983, 2017.
[15] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville., "Improved training of wasserstein GANs," in Advances in Neural Information Processing Systems 30, I. Guyon, et al., Editors. 2017, Curran Associates, Inc. p. 5767-5777.
[16] O. Press, A. Bar, B. Bogin, J. Berant, and L. Wolf., Language Generation with Recurrent Generative Adversarial Networks without Pre-Training, arXiv preprint arXiv:1706.01399, 2017.
[17] S. Subramanian, S. Rajeswar, F. Dutil, C. Pal, and A. Courville, "Adversarial generation of natural language," in Proc. of the 2nd Workshop on Representation Learning for NLP, pp. 241-251, Vancouver, Canada, 3-3 Aug. 2017.
[18] A. S. Vezhnevets, et al., "FeUdal networks for hierarchical reinforcement learning," in Proc. of the 34th Int’ Conf. on Machine Learning, ICML 2017, vol. 70, pp. 3540-3549, Sydney, Australia, Aug. 2017.
[19] R. D. Hjelm and A. Jacob, "Boundary-seeking generative adversarial networks," in Proc. Int. Conf. on Learning Representations, 17 pp., Apr. 2018.
[20] M. H. Moghadam and B. Panahbehagh, Creating a New Persian Poet Based on Machine Learning, Computing Research Repository (CoRR), 2018. abs/1810.06898.
[21] S. H. Hosseini Saravani, M. Bahrani, H, Veisi, and S. Besharati, "Persian language modeling using recurrent neural networks," in Proc. 9th Int. Symp. on Telecommunications, IST’18, pp. 207-210, Tehran, Iran, 17-19 Dec. 2018.
[22] M. Sugiyama, T. Suzuki, and T. Kanamori, "Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation," Annals of the Institute of Statistical Mathematics, vol. 64, no. 5, pp. 1009-1044, 2012.
[23] X. Zhang and M. Lapata, "Chinese poetry generation with recurrent neural networks," in Proc. of the Conf. on Empirical Methods in Natural Language Processing, EMNLP’14, pp. 670-680, Doha, Qatar, 25-29 Oct. 2014.
[24] K. Papineni, S. Roukos, T. Ward, and W. –J. Zhu, "Bleu: a method for automatic evaluation of machine translation," in Proc. of the 40th Annual Meeting of the Association for Computational Linguistics, ACL’02, pp. 311-318, Philadelphia, PA, USA, Jul. 2002.
[25] Y. Zhu, et al., "Texygen: a benchmarking platform for text generation models," in Proc. 41st Int. ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR’18, pp. 1097-1100, Jun. 2018.
[26] D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, CoRR, 2014. abs/1412.6980.