SongPrepE2E: End-to-End Lyrics Structuring

• SongPrepE2E is an end-to-end system that transforms raw full-song audio into a unified, structured lyrics format combining segment labels, timestamps, and lyrical content.
• It bypasses traditional multi-stage pipelines by leveraging MuCodec for audio tokenization and Qwen2-7B for semantic sequence generation, ensuring efficient processing.
• The approach achieves competitive WER and generation quality, serving as a scalable preprocessor for downstream song generation models like LeVo.

SongPrepE2E is the end-to-end component of the SongPrep framework that takes raw, full-song audio and directly outputs a fully structured lyrics representation: segment-level song structure, precise timestamps for each segment, and the associated lyrics text (Tan et al., 22 Sep 2025). Unlike the multi-stage SongPrep pipeline—Demucs → structure model → ASR → alignment—SongPrepE2E bypasses explicit source separation and ASR chunking, operates fully end-to-end over whole songs, and generates structure labels, segment boundaries, and lyrics in a single textual sequence (Tan et al., 22 Sep 2025). Within the broader SongPrep system, it is positioned as a preprocessing model for song generation data preparation, where automated structure analysis and lyrics transcription are required at scale.

1. Task definition and representational scope

SongPrepE2E operates on full-song audio from in-the-wild songs, including YouTube and internal copyright data, and processes up to 4 minutes per song in a single pass (Tan et al., 22 Sep 2025). Audio is discretized into MuCodec tokens at 25 Hz, with a codebook size of 16,384; for a 4-minute song, this yields approximately 6000 time-step tokens. The output is a textual structured representation for the entire song in the format [structure] [start:end] lyric, where structure is one of seven labels—intro, outro, inst, verse, [chorus](https://www.emergentmind.com/topics/chorus-0816c914-858d-49b9-9c0a-6e7b72a3a191), [bridge](https://www.emergentmind.com/topics/bridge-5c9eba77-1fd8-40e8-9860-1c25c7352f4e), silence—and start:end specifies segment boundaries relative to song start (Tan et al., 22 Sep 2025).

The representation is simultaneously a segmentation of the song into structural sections, a lyrics transcript aligned to each section, and a timestamped timeline suitable for downstream generation, alignment, and evaluation (Tan et al., 22 Sep 2025). Instrumental and silence segments can have empty or omitted lyrics. Because structural labels are text tokens such as [verse] and [chorus], rather than separate classifier outputs, SongPrepE2E frames structure analysis as sequence generation rather than as a distinct classification subtask.

This design makes the model a structured lyrics recognizer rather than a conventional ASR system. A plausible implication is that the target space is closer to a song-level symbolic description than to utterance-level transcription. That distinction matters because the model is asked not only to recover lexical content, but also to infer section boundaries and section type from raw mixed audio.

2. Architecture and joint modeling strategy

SongPrepE2E is a two-stage end-to-end model: an acoustic front-end, MuCodec, and a semantic back-end, Qwen2-7B (Tan et al., 22 Sep 2025). MuCodec converts continuous music audio into discrete tokens suitable as input to a LLM, while Qwen2-7B is fine-tuned to read those audio tokens and generate structured lyrics.

MuCodec comprises three principal components. First, the MuEncoder, frozen at SongPrepE2E training time, uses 13 stacked Conformer blocks. Its pretraining objectives are Masked Language Modeling on latent representations, reconstruction loss, and lyric recognition loss. The resulting latent representations are described as carrying musical structure cues, phonetic content, and higher-level semantics. Second, Residual Vector Quantization quantizes those latent features via multiple codebooks in a residual fashion, with codebook size 16,384. Third, a flow matching module is trained jointly with RVQ after the MuEncoder is frozen; its objectives are reconstruction loss, codebook loss, and representation alignment loss to HuBERT features. The stated purpose of the HuBERT alignment is to encourage MuCodec tokens to encode phonetics, timbre, and musical structure, not merely waveform fidelity (Tan et al., 22 Sep 2025).

The back-end uses Qwen2-7B as a pretrained general-purpose LLM. The paper’s training description states that audio tokens generated by MuCodec are paired with SongPrep’s structured lyrics output to create audio–text training pairs, and those pairs are used to perform Supervised Fine-Tuning of Qwen2-7B (Tan et al., 22 Sep 2025). Conceptually, the model is trained to estimate

$$P(\mathbf{y}\mid \mathbf{x})=\prod_{t=1}^{N} P(y_t\mid y_{<t},\mathbf{x},\mathbf{c}),$$

where $\mathbf{x}$ is the MuCodec token sequence and $\mathbf{y}$ is the tokenized text output containing section labels, timestamps, and words.

A defining property of SongPrepE2E is that structure analysis is implicitly modeled as part of the language generation task. There is no separate structure head versus lyrics head inside SongPrepE2E (Tan et al., 22 Sep 2025). The model must infer when a verse starts and ends, when choruses repeat, and when an instrumental section should receive no lyrics, while also generating lexical content. Because the entire song is in context, the model can exploit repeated choruses, rhyming schemes, verse–chorus alternation, and canonical patterns such as intro → verse → chorus → verse → chorus → bridge → chorus → outro (Tan et al., 22 Sep 2025).

3. Training regime, supervision, and data curation

The training of SongPrepE2E combines MuCodec pretraining and large-scale supervised fine-tuning of Qwen2-7B on SongPrep-generated labels (Tan et al., 22 Sep 2025). During MuCodec training, the loss combines reconstruction loss, codebook loss, representation alignment loss, MLM, and lyric recognition. SongPrepE2E itself is trained via Supervised Fine-Tuning on audio–structured-text pairs with a standard cross-entropy loss over generated text tokens:

$$\mathcal{L}_{\text{SFT}}=-\sum_{t=1}^{N}\log P\big(y_t^\ast \mid y_{<t}^\ast,\mathbf{x}\big).$$

Since structure labels and timestamps are part of the target text, there is no separate loss term for structure versus lyrics inside SongPrepE2E (Tan et al., 22 Sep 2025).

The model is trained using data curated by the SongPrep pipeline from an in-house dataset of 2 million songs, approximately 110,000 hours (Tan et al., 22 Sep 2025). The pipeline used to generate training targets is Demucs → All-In-One + DPRNN structure → Whisper + Zipformer ASR → WER-FIX → wav2vec2 alignment. Two dataset variants are defined by WER filtering. The WER < 0.3 filter contains approximately 640 billion tokens and more data with moderate noise, while the WER < 0.1 filter contains approximately 240 billion tokens, is higher quality, and is described as having more risk of overfitting (Tan et al., 22 Sep 2025). Among SongPrepE2E variants, the WER < 0.3 variant performs best, balancing size and quality.

The hand-annotated benchmark is SSLD-200, a 200-song dataset totaling 13.9 hours, with 100 English songs and 100 Chinese songs collected from YouTube (Tan et al., 22 Sep 2025). Structure segmentation is annotated at second-level resolution, using the seven refined categories intro, outro, inst, verse, chorus, bridge, and silence, and each segment has manually annotated lyrics aligned to the corresponding time range. SSLD-200 is used to compute DER for structure analysis and WER for lyrics transcription.

The paper does not list explicit augmentations for SongPrepE2E training. It does, however, describe WER-FIX for the SongPrep pipeline: Whisper ASR outputs are used to correct scraped lyrics; only songs where initial lyrics have WER < 0.7 are kept; substitution errors trust original lyrics; insertion and deletion errors trust ASR output; and Zipformer ASR is fine-tuned on approximately 7,000 hours of WER-FIX corrected data, around 100k songs (Tan et al., 22 Sep 2025). It also notes that MuCodec’s MLM objective implies masked token augmentation on latent features. No explicit singing-specific augmentations such as pitch-shifting or tempo changes are described.

4. End-to-end inference, evaluation, and error profile

At inference, SongPrepE2E does not run Demucs and operates directly on raw mixed audio, including vocals and instruments (Tan et al., 22 Sep 2025). The paper attributes its performance without explicit source separation to three factors: MuCodec’s representation alignment with HuBERT, Qwen2-7B’s semantic robustness, and full-song context. HuBERT-based alignment is described as helping the codec emphasize vocal and phonetic cues even in a mixture; the LLM can disambiguate words under noisy conditions and infer missing or partially masked lyrics from context; and long-context processing avoids chunking into short windows, allowing the model to exploit repeated choruses and song-wide syntax and semantics (Tan et al., 22 Sep 2025).

The system is also described as handling overlapping voices, background instruments, and non-lexical vocals implicitly via learned patterns (Tan et al., 22 Sep 2025). Overlapping voices are represented in MuCodec tokens as mixture content, while Qwen2-7B uses semantics and global context to decide the dominant lyrics. Background instruments are not explicitly removed, but their patterns help structure inference, especially for inst, intro, bridge, and outro. Non-lexical vocals such as “oh,” “la,” and humming may be treated as lyrics tokens or skipped, depending on SongPrep training data.

On SSLD-200, the paper reports the following pipeline-level comparison (Tan et al., 22 Sep 2025):

Model	Data Filter	DER ↓ / WER ↓ / RTF ↓
SongPrep (pipeline)	–	15.8% / 27.7% / 0.235
SongPrepE2E	WER < 0.3	18.1% / 24.3% / 0.108
SongPrepE2E	WER < 0.1	18.8% / 27.3% / 0.108

The best SongPrepE2E configuration, trained with WER < 0.3, achieves lower WER than the pipelined SongPrep, with 24.3% versus 27.7%, and much better efficiency, with RTF 0.108 versus 0.235 (Tan et al., 22 Sep 2025). Its DER of 18.1% is slightly higher than the pipeline’s 15.8%, but the paper characterizes the degradation as modest given that SongPrepE2E has no explicit structure head and uses no separate alignment model.

The paper also reports module-level baselines on SSLD-200. For structure analysis, Original ALL-IN-ONE has DER 25.0%, bilingual fine-tuning on 3,700 songs gives DER 20.8%, and adding DPRNN for global context yields DER 16.1% (Tan et al., 22 Sep 2025). For lyric recognition, Whisper without separation yields WER 47.2%, Whisper + Demucs 27.7%, Zipformer without separation 30.6%, and Zipformer + Demucs 25.8% (Tan et al., 22 Sep 2025). These numbers establish the competitive position of the end-to-end model: it surpasses the SongPrep pipeline in WER despite eliminating explicit source separation and chunked ASR.

The error analysis given in the paper centers on structure mis-segmentation, confusion between repeated choruses, and lyric misrecognition under reverb, backing vocals, strong instrumental leads, non-standard pronunciation, shouting, or whisper singing (Tan et al., 22 Sep 2025). The paper also notes that performance still depends heavily on the amount of training data and that more data reduces such errors.

5. Use in song generation

SongPrepE2E’s output is used to train song generation models, notably LeVo (Tan et al., 22 Sep 2025). For each song, the conditioning representation is the structured sequence [structure] [start:end] lyric, optionally accompanied by audio tokens or representations for teacher-forcing models. The paper states that models like LeVo learn mappings from structure+lyrics to audio, consistent alignment between segments and melody, and multi-preference alignment involving musicality, lyric coherence, and listener preference.

The experimental comparison uses identical LeVo architectures and internal copyright data, differing only in preprocessing quality: one system is trained on data processed by the original All-In-One + Whisper pipeline, and the other is trained on data processed by SongPrepE2E (Tan et al., 22 Sep 2025). In a subjective evaluation of 40 generated songs per model with 30 listeners with basic musical knowledge, three metrics on a 1–5 scale are reported. Musicality Structure improves from 2.52 to 3.00, Lyric Matching Degree improves from 2.82 to 4.52, and Subjective Bias improves from 2.07 to 2.73 when SongPrepE2E preprocessing is used (Tan et al., 22 Sep 2025).

These results are presented as evidence that SongPrepE2E-derived data leads to generated songs that more closely resemble human-produced songs in structure and lyric coherence and are more appealing to listeners (Tan et al., 22 Sep 2025). Within the SongPrep paper, this downstream result is not ancillary: it functions as validation that the structured representation extracted by SongPrepE2E is usable by a lyrics-to-song generator rather than merely well-scored under DER and WER.

A plausible implication is that SongPrepE2E serves as an interface between large-scale song corpora and controllable song generation. Its representation compresses a full mix into structural sections, time spans, and lexical content, which is precisely the kind of conditioning object used by long-form generative systems.

6. Relation to adjacent work, innovations, and limitations

SongPrepE2E belongs to a line of research that treats music understanding tasks as end-to-end sequence modeling, but its scope is broader than earlier lyrics alignment or singing ASR systems. “End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model” predicts character probabilities directly from raw audio using a modified Wave-U-Net and weak line-level annotations, avoiding explicit separation and pronunciation dictionaries (Stoller et al., 2019). “End-to-end lyrics Recognition with Voice to Singing Style Transfer” uses a Transformer ASR with hybrid CTC+attention, synthetic singing generated by V2S augmentation, transfer learning, and lyrics language modeling, and reports that source-separated vocals remain important for polyphonic conditions (Basak et al., 2021). “Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Framework” addresses singer diarization by converting choral singing into clean, non-overlapped audio before training EEND (Munakata et al., 2024). “YuE: Scaling Open Foundation Models for Long-Form Music Generation” uses structure labels such as [intro], [verse], [chorus], [bridge], and [outro] as conditioning in long-form lyrics-to-song generation up to five minutes (Yuan et al., 11 Mar 2025). This suggests that SongPrepE2E occupies an intermediate position between song understanding and long-form conditioned song generation.

The SongPrep paper identifies five key innovations: end-to-end structured lyrics recognition for full songs; MuCodec + LLM fusion; joint implicit modeling of song structure and lyrics; massive-scale weakly supervised training via SongPrep on approximately 2M songs; and direct benefit for downstream song generation (Tan et al., 22 Sep 2025). The central methodological shift is that song structure and lyrics transcription are treated as a single, long-form conditional generation problem over discrete audio tokens, rather than as a cascade of source separation, structure recognition, ASR, and alignment modules.

Several limitations are also explicit. First, DER is slightly worse than the pipeline, 18.1% versus 15.8%, and the paper links this to implicit structure generation without explicit structure loss or alignment (Tan et al., 22 Sep 2025). Second, SongPrepE2E relies on SongPrep’s automatically generated structured labels for training, so errors or biases in SongPrep propagate into SongPrepE2E. Third, SSLD-200 contains only 200 songs, English and Chinese only, and the paper does not report detailed genre or language breakdowns beyond that benchmark. Fourth, robustness to highly noisy or live recordings is not explicitly evaluated, and no explicit data augmentation for such scenarios is described. Fifth, songs longer than approximately 4 minutes are not directly supported in one pass (Tan et al., 22 Sep 2025).

These points address two common misconceptions. One is that end-to-end processing removes dependence on preprocessing or auxiliary supervision; in fact, SongPrepE2E depends on MuCodec pretraining and on SongPrep-generated labels at very large scale. The other is that end-to-end processing uniformly replaces the pipeline; the reported results show a more specific trade-off, namely better WER and RTF, but slightly worse DER (Tan et al., 22 Sep 2025). Future work directions implied by the paper include scaling training data further, introducing explicit multi-task training with separate structure and lyrics losses, extending the approach to other LLM architectures and audio encoders, enhancing robustness to live and noisy recordings, and integrating SongPrepE2E outputs with RL-based preference optimization for song generation (Tan et al., 22 Sep 2025).