Rhyme-Aware Chinese Lyric Generation Using GPT-2
This paper presents a novel approach to the task of Chinese lyric generation by integrating rhyme awareness into a pre-trained LLM, specifically enhancing GPT-2. The primary objective is to improve the generation of Chinese lyrics which traditionally require rhyme—a feature not adequately addressed by standard pre-trained LLMs.
Methodology
The authors modify the GPT-2 model to include a rhyme-aware mechanism. This is achieved by embedding rhyme information within the network architecture. The model is structured with two stacked modules: an underlying textual encoder and an upper encoder that integrates rhyme information with lexical and syntactic details from the textual encoder. This setup facilitates the representation of heterogeneous information in a unified feature space.
A rhyme vocabulary is constructed by classifying the pinyin of Chinese characters into 13 classes, enabling rhyme embedding. The rhyme-aware encoder employs multi-head self-attention to process token and rhyme embeddings, enhancing the model's ability to learn and generate rhyming lyrics. Additionally, the system incorporates layer normalization and residual connections to stabilize training and improve convergence.
Experimental Results and Ablation Studies
The model exhibits significant improvements in rhyming capability. In experiments conducted on the Chinese-Lyric-Corpus dataset, the proposed model achieves an impressive 82.2% rhyme rate, in contrast to the 30.9% rate with a standard pre-trained model. These results underscore the effectiveness of incorporating rhyme embeddings. Human evaluations further affirm the model's ability to generate meaningful, fluent, and consistent lyrics comparable to those of traditional methods, but with substantially improved rhyming qualities.
Ablation studies demonstrate that the absence of rhyme input markedly diminishes performance. Models leveraging rhyme embeddings significantly outperform their non-rhyming counterparts, particularly when using processed datasets that enhance rhyme consistency.
Implications and Future Work
The inclusion of rhyme embeddings in neural lyric generation models offers substantial improvements, especially in languages where rhyme is a critical aesthetic feature, such as Chinese. This advancement potentially extends beyond lyrics to other creative contexts requiring linguistic rhythm, such as poetry.
The paper acknowledges limitations in the current model, notably its reliance on a dataset that may not comprehensively cover diverse themes beyond love—reflecting the nature of the training data. Future work could enhance model efficacy by expanding artistic datasets and considering additional prosodic features like intonation.
Moreover, the model's approach could inspire further exploration into integrating other linguistic stylistic elements in natural language generation, thus broadening the practical applications of augmented LLMs in creative domains.
Overall, this work contributes a significant methodological enhancement in the domain of natural language generation for creative texts, offering a pathway to richer, more culturally resonant AI-generated content.