Listen and Chant Before You Read: The Ladder of Beauty in LM Pre-Training

Abstract: We show that pre-training a Transformer on music before language significantly accelerates language acquisition. Using piano performances (MAESTRO dataset), a developmental pipeline -- music $\to$ poetry $\to$ prose -- yields a $17.5\%$ perplexity improvement over random initialization ($p < 0.001$, 5 seeds), with music and poetry improving orthogonal model components (internal computation and embeddings, respectively). Convergence tests confirm that this is not a transient head start: at $d!=!64$, multi-seed validation (5 seeds) shows a persistent 5.5\% gap at plateau ($p = 0.017$), with the pipeline converging faster and to a lower loss in every run. Real music matches the transfer ceiling of synthetic patterns with one-third the data, and scaling experiments reveal that optimal pre-training data volume shifts with model capacity ($-3\% \to +3\% \to +6\%$ advantage of larger datasets from $d!=!16$ to $d!=!64$). Across the scales we study ($d!\in!{16,32,64}$, up to ${\sim}400$K parameters), these results suggest a capacity-dependent data curation principle and indicate that structured human creative outputs can provide an efficient pre-training substrate for small LLMs; stronger conclusions at modern pre-training scale will require substantially larger experiments.

• The paper demonstrates that pre-training on expert-curated music yields significant language model efficiency gains, with notable perplexity reductions compared to random initialization.
• The paper shows that incorporating an intermediate poetry phase further refines embeddings and adds measurable performance improvements over baseline models.
• The paper reveals capacity-dependent scaling effects, where larger models optimally benefit from increased structured music data, achieving up to a 6.1% improvement.

Developmentally-Staged LLM Pre-Training: Empirical Evidence from Music, Poetry, and Prose

Introduction and Motivation

This paper systematically investigates the hypothesis that structured, sequential, and hierarchically organized non-linguistic data—specifically music—can be leveraged for pre-pre-training Transformer LMs, enhancing their efficiency in subsequent language domain learning. Inspired by findings in cognitive science and previous work using synthetic data for “foundation warming” [lee2026nca], the authors construct a multi-phase developmental pipeline: music $\to$ poetry $\to$ prose. This pipeline is intended to reflect the trajectory of human language acquisition, where pre-linguistic exposure to regularities such as rhythm and prosody precedes lexical and syntactic mastery [kuhl2004early, patel2003language].

Key contributions include demonstrating that pre-training on real music (MAESTRO dataset) produces greater and more data-efficient transfer than synthetic patterns; that introducing an intermediate poetry stage leads to additive improvements; and that scaling model size increases the potential for pre-training data to yield transfer benefits. Rigorous ablations and multi-seed analyses corroborate the core findings.

Figure 1: The developmental pipeline: music pre-training establishes attention structures for dependency tracking, poetry calibrates embeddings toward language, and prose training evaluates generalization; vocabulary transitions between phases require selective weight transfer.

Experimental Protocols and Methodology

Model Architecture and Training

The experiments center on decoder-only Transformers (GPT-2 style), at three principal scales ($d=16$, $d=32$, $d=64$; up to $\sim$400K parameters): models are deliberately capacity-constrained to magnify the importance of efficient, transferable structural learning.

• Music phase: Models are trained on either real music (MAESTRO) or synthetic sequences with a controlled MIDI-like tokenization (160 tokens, REMI-inspired).
• Poetry and Prose phases: The vocabulary shifts to GPT-2 subwords (50,257 tokens). Only the internal computation layers (attention, FFN, LayerNorm) are transferred; token embeddings and output heads are reinitialized, due to lexical non-overlap.

Data and Controls

• Music: MAESTRO (36k chunks, professional Western piano repertoire) vs. structurally simplistic synthetic data, at matched volumes (3k, 12k, 36k chunks).
• Language: Gutenberg Poetry (36k chunks) forms the intermediate poetry phase; WikiText-103 ($\sim$10% subset) is used for final prose evaluation.

Hyperparameters and evaluation splits are tightly controlled across settings to isolate the effects of pre-training phase and data source.

Empirical Results

Music Pre-Training Yields Robust and Efficient Transfer

Music-trained models consistently and significantly outperform random initialization baselines on subsequent language modeling (WikiText-103), with perplexity (PPL) reductions persisting across data volumes and random seeds.

• Data quality: At $d=16$, real music (MAESTRO-12k) achieves equal or better transfer than synthetic music even when given only one-third the data volume, establishing that structured, expert-curated data enables more efficient acquisition of generalizable sequence modeling skills.

Additive Impact of a Developmental Pipeline

Inserting an intermediate poetry phase produces strong, additive gains, with the full pipeline (MAESTRO $\to$ poetry $\to$ prose) achieving a 17.5% PPL reduction at epoch 2 versus the random baseline ($\to$0, five seeds).

Notably, the improvements from music and poetry are orthogonal: analysis of model parameter transfer reveals that music pre-training enhances internal computation (attention/FFN), while the poetry phase tunes embeddings. The aggregate effect is both accelerated convergence and lower final loss.

Capacity-Dependent Scaling Principles

Scaling experiments address how optimal pre-training data volume interacts with model size. At $\to$1, increasing music data beyond 12k chunks saturates or marginally hurts performance (indicative of overfitting or redundancy); at $\to$2, the best results arise from the largest dataset (36k), and the margin of improvement widens (up to 6.1%).

Figure 2: Validation perplexity at epoch 2 across scales: music pre-training boosts performance at all scales, with larger models benefiting more from increased music data.

Figure 3: Learning curves show the optimal data condition shifts from 12k chunks (small models) to 36k (larger models); the best music condition consistently outperforms the random baseline.

Persistent and Irreducible Transfer Gains

Convergence tests training both the baseline and the full pipeline to plateau confirm that the pre-training advantage is not merely an ephemeral head start. At $\to$3, the pipeline retains a statistically significant 5.5% lower PPL at convergence ($\to$4, five seeds), with this advantage observed in every trial.

Mechanistic Analysis

• Transfer mechanism: Attention and FFN layers, trained on music, encode generic hierarchical and long-range pattern processing structures; these are directly re-usable for natural language, aligning with findings on cross-modal transfer [lee2026nca].
• Efficiency of real music: MAESTRO-trained models achieve equivalent or better language transfer with fewer examples, attributable to the richer statistical and hierarchical structure of expert musical performance.
• Developmental analogy: The pipeline closely parallels human developmental progressions, suggesting curriculum ordering from general sequence regularity to linguistic granularity may underlie efficient learning [kuhl2004early].

Implications and Future Directions

The results imply that foundation warming of LMs with structured, non-linguistic sequence data (particularly music with hierarchical organization) substantially improves learning efficiency and performance. This challenges the assumption that LMs should be trained on language from scratch and highlights that the nature (quality/structure) of pre-training data can matter as much as, or more than, its volume.

Practically, these findings suggest that for small to medium-sized LMs, careful curation of high-quality, structured pre-training corpora may be essential for maximizing learning efficiency. The scaling trends indicate that even at larger scales, there is no one-size-fits-all approach: both the quantity and composition of pre-training data require capacity-aware design.

The evidence for orthogonal contributions from the music and poetry phases suggests future work could explore more granular or modular developmentally-inspired curricula, or even adaptive curricula that monitor and respond to the saturation of structural patterns.

Limitations

The experiments, while robust in methodology, are conducted with models orders of magnitude smaller than current SOTA LMs and on a limited linguistic and musical typology (English and Western classical piano). The transfer findings await validation at larger parameter counts, with more diverse data, and on a wider range of practical downstream tasks.

Conclusion

This study empirically validates that a staged, developmentally-motivated pre-training pipeline—music $\to$5 poetry $\to$6 prose—provides strong and persistent learning advantages for Transformer LLMs, via efficient acquisition and transfer of general sequence modeling capabilities followed by progressively language-centric fine-tuning. The results contribute novel empirical support to the use of creative, hierarchically-structured non-linguistic domains as high-leverage substrates for neural network pre-training and invite further scaling and curricular exploration.