SongPrep: Automated Song Preprocessing

• SongPrep is an automated preprocessing framework for song data that converts raw audio into structured, training-ready annotations through a multi-stage processing pipeline.
• It combines source separation, structure parsing, and lyric recognition to reduce manual annotation efforts and enable scalable song corpus construction.
• SongPrepE2E extends the framework into a unified model that jointly predicts structured, timestamped lyrics using audio tokenization and pretrained language models.

SongPrep is an automated preprocessing framework for song data and, in its extended form, an end-to-end structured lyrics recognition system designed to convert raw songs into training-ready annotations for song generation models. Its central output format is $[structure][start:end]lyric$, coupling section labels, timestamps, and transcribed text so that full songs can be represented as structured supervision rather than unsegmented audio. The framework was introduced to address the cost and scalability limits of manual song annotation, especially for structure parsing and lyrics transcription in large in-the-wild corpora, and it is paired with SongPrepE2E, a MuCodec-plus-LLM model that predicts structured, timestamped lyrics directly from full-song audio (Tan et al., 22 Sep 2025).

1. Conceptual scope and motivation

SongPrep is motivated by a specific bottleneck in music AIGC: raw songs are abundant, but aligned lyrics and section annotations are not. The framework therefore targets three preparation problems simultaneously: source separation, structure analysis, and lyric recognition. In the formulation reported for SongPrep, inaccurate or noisy text-audio pairs can induce hallucinations in downstream song models, while missing section labels remove musically meaningful structure such as verse–chorus alternation, weakening melody and arrangement learning (Tan et al., 22 Sep 2025).

The framework’s intended role is not generation itself, but data transformation. It converts songs into structured annotations that can be consumed by downstream systems such as LeVo. This makes SongPrep closer to a corpus-construction and supervision-generation layer than to a standalone composition model. SongPrepE2E extends this role by replacing the multi-stage preprocessing stack with a single model that analyzes structure and lyrics jointly from the whole song, using pretrained semantic knowledge and long-context modeling (Tan et al., 22 Sep 2025).

A common misconception is to read SongPrep primarily as a diarization or ASR system. In fact, its reported “DER” is used as a segment-structure accuracy metric rather than singer diarization, and its pipeline couples section parsing with lyric recognition because both are required for training-ready full-song supervision (Tan et al., 22 Sep 2025).

2. Multi-stage preprocessing pipeline

The original SongPrep pipeline is organized as a sequence of specialized modules. Source separation is performed with Demucs, which decomposes a full mix into vocals, drums, bass, and “other” stems. The vocal stem is routed to lyric recognition, while the reassembled stems are used for structure analysis so that the parser retains access to the full arrangement (Tan et al., 22 Sep 2025).

Structure analysis begins from All-In-One and is modified in two ways. First, 1D and 2D dilated Neighborhood Attention are added to capture local patterns. Second, a Dual-Path RNN block is inserted after every three All-In-One blocks to strengthen long-range dependency modeling. The refined label set comprises seven categories: intro, outro, inst, verse, chorus, bridge, and silence. Because the original All-In-One was English-only, SongPrep retrains the structure parser on 3,700 internal bilingual songs in English and Chinese (Tan et al., 22 Sep 2025).

Lyric recognition uses Whisper for initial recognition and a WER-FIX procedure to repair scraped web lyrics. Step 1 retains only lyric texts with WER $<0.7$ versus Whisper outputs. Step 2 keeps original lyrics for substitutions but trusts ASR outputs for insertions and deletions. A Zipformer-based ASR is then fine-tuned on about 100,000 songs, or about 7,000 hours, from the WER-FIX-validated set. A wav2vec2-based word alignment module calibrates structure predictions so that accompaniment-only spans are less likely to be mislabeled as verse or chorus (Tan et al., 22 Sep 2025).

Stage	Model or tool	Reported role
Source separation	Demucs	Produces vocals, drums, bass, other stems
Structure analysis	All-In-One + Neighborhood Attention + DPRNN	Predicts intro/outro/inst/verse/chorus/bridge/silence
Lyric recognition and calibration	Whisper, WER-FIX, Zipformer, wav2vec2	Transcribes lyrics and calibrates segment labels

The pipeline’s output is explicitly structured. Rather than returning plain text or coarse section boundaries, it emits per-section lyrics with timestamps in the form $[structure][start:end]lyric$, which is the representation used downstream for training song generation systems (Tan et al., 22 Sep 2025).

3. SongPrepE2E and end-to-end structured recognition

SongPrepE2E removes explicit source separation and staged parsing at inference time. Its audio tokenizer is MuCodec, whose MuEncoder contains 13 stacked Conformer blocks trained with Masked Language Modeling, reconstruction, and lyric recognition constraints. After MuEncoder training, the encoder is frozen, and RVQ plus Flow Matching are trained jointly with reconstruction, codebook, and HuBERT-alignment terms. The resulting token stream is discrete at 25 Hz with codebook size 16,384 (Tan et al., 22 Sep 2025).

On the language side, SongPrepE2E uses Qwen2-7B as the pretrained backbone. The model receives full-song MuCodec token sequences, up to about 4 minutes, and generates structured timestamped lyrics directly in the same $[structure][start:end]lyric$ format as the pipeline. Its supervised objective is autoregressive cross-entropy,

$$L_{\mathrm{SFT}} = -\sum_t \log p(y_t \mid y_{<t}, a),$$

where $a$ denotes the audio-token sequence and $y_t$ denotes the structured lyric output token at step $t$ (Tan et al., 22 Sep 2025).

The distinction between SongPrep and SongPrepE2E is therefore architectural rather than semantic. The former externalizes separation, structure parsing, ASR, and alignment as separate modules; the latter internalizes them into a single audio-to-structured-text model. The paper reports no additional forced-alignment stage inside SongPrepE2E: timestamps are learned directly because the targets themselves contain $[start:end]$ spans (Tan et al., 22 Sep 2025).

This design makes SongPrepE2E a whole-song structured recognition model rather than a mere transcription backend. It uses long-context song audio, pretrained language knowledge, and audio-token supervision derived from SongPrep outputs, so the end-to-end model is trained on the structured representation that the pipeline itself was built to produce (Tan et al., 22 Sep 2025).

4. Dataset, metrics, and empirical results

The evaluation benchmark introduced with SongPrep is SSLD-200, a 200-song dataset totaling 13.9 hours, evenly split between 100 English and 100 Chinese songs. Each song is annotated with the seven structure labels, second-level boundaries, and segment-level lyrics (Tan et al., 22 Sep 2025).

The principal metrics are Word Error Rate and a repurposed Diarization Error Rate. WER is reported in the standard form

$\mathrm{WER} = \frac{S + D + I}{N},$

where $<0.7$0, $<0.7$1, and $<0.7$2 are substitutions, deletions, and insertions, and $<0.7$3 is the number of reference words. The paper also reports real-time factor. For structure analysis, DER is used as a segment-label and boundary accuracy metric rather than speaker diarization (Tan et al., 22 Sep 2025).

At the module level, bilingual retraining and DPRNN augmentation materially improve structure parsing. The original All-In-One baseline has DER $<0.7$4, bilingual fine-tuning reduces this to $<0.7$5, and adding DPRNN reduces it further to $<0.7$6. After ASR-based alignment calibration, the full SongPrep pipeline reaches DER $<0.7$7 (Tan et al., 22 Sep 2025).

Source separation has a large effect on lyric recognition. On SSLD-200, Whisper improves from WER $<0.7$8 on full-mix audio to $<0.7$9 on Demucs-separated vocals, while Zipformer improves from $[structure][start:end]lyric$0 to $[structure][start:end]lyric$1 under the same separation condition (Tan et al., 22 Sep 2025).

The pipeline and end-to-end model trade off structure precision against lyric accuracy and speed. SongPrep reports DER $[structure][start:end]lyric$2, WER $[structure][start:end]lyric$3, and RTF $[structure][start:end]lyric$4. SongPrepE2E trained on the looser $[structure][start:end]lyric$5 filtered corpus reports DER $[structure][start:end]lyric$6, WER $[structure][start:end]lyric$7, and RTF $[structure][start:end]lyric$8; with a stricter $[structure][start:end]lyric$9 filter, DER becomes $[structure][start:end]lyric$0 and WER $[structure][start:end]lyric$1. The paper interprets the stronger lyric performance of the looser filter as an effect of larger retained data volume, whereas stricter filtering reduces data and increases overfitting risk (Tan et al., 22 Sep 2025).

Downstream evaluation uses LeVo trained on SongPrepE2E-processed data and compares it with a baseline using All-In-One plus Whisper outputs. In a 40-song human study with 30 participants, SongPrepE2E preprocessing raises Musicality Structure from $[structure][start:end]lyric$2 to $[structure][start:end]lyric$3, Lyric Matching Degree from $[structure][start:end]lyric$4 to $[structure][start:end]lyric$5, and Subjective Bias from $[structure][start:end]lyric$6 to $[structure][start:end]lyric$7. The most pronounced gain is in lyric matching, consistent with the framework’s emphasis on structure-aware text-audio supervision (Tan et al., 22 Sep 2025).

5. Position within transcription and alignment research

SongPrep sits within a longer line of work on lyrics transcription and alignment, but differs in scope by centering full-song preprocessing rather than isolated subproblems. Earlier hybrid-ASR work on commercial English vocal stems reported a best WER of $[structure][start:end]lyric$8 with TDNN-LSTM, 3-fold speed perturbation, a lyric-specific LLM, and singing-aware lexicon modifications, illustrating the difficulty of singing ASR before large pretrained audio-LLMs became available (Tsai et al., 2018).

For Mandarin polyphonic music, adaptation of Whisper medium plus HT Demucs and a BiGRU aligner achieved CER below $[structure][start:end]lyric$9 and character-level alignment MAE of $$L_{\mathrm{SFT}} = -\sum_t \log p(y_t \mid y_{<t}, a),$$0 seconds, showing that low-resource singing transcription could be made practical through pretrained speech models and source separation (Wang et al., 2023). SongPrep generalizes this philosophy to bilingual full-song corpora and extends it from transcription-plus-alignment to structure-conditioned dataset production (Tan et al., 22 Sep 2025).

Unsupervised alignment research offered an alternative trajectory. A vowel-centric method based on robust PCA, WS-NMF, and Canonical Time Warping aligned lyrics to audio without supervised ASR and reached $$L_{\mathrm{SFT}} = -\sum_t \log p(y_t \mid y_{<t}, a),$$1 word-level accuracy on English pop and $$L_{\mathrm{SFT}} = -\sum_t \log p(y_t \mid y_{<t}, a),$$2 syllable-level accuracy on Korean pop. That line of work emphasized per-song self-learned vowel structure rather than large-scale corpus curation (Chang et al., 2017).

More recent unified systems move closer to SongPrepE2E’s objectives. SongTrans predicts lyrics, word durations, note counts, note boundaries, and note pitches directly from raw mixed audio, and is explicitly described as the first model capable of aligning lyrics with notes while avoiding preprocessing at inference (Wu et al., 2024). VocalParse pushes further by using a Large Audio LLM with interleaved prompting to jointly predict lyrics, melody, and word-note correspondence, and reports state-of-the-art singing voice transcription performance on multiple datasets (Chen et al., 6 May 2026). Relative to these systems, SongPrepE2E is broader at the song-structure level but narrower at the explicit note-alignment level.

6. Broader role in the song AI stack, limitations, and likely directions

In later literature, SongPrep functions not only as the title of a specific preprocessing framework but also as an operational context for preparing songs for editing, transcription, and generation. SketchSong explicitly states that its preprocessing pipeline uses SongPrep for structure parsing, lyrics transcription with WER-FIX refinement and Zipformer ASR, wav2vec 2.0-based alignment, and related metadata extraction before training a hierarchical full-song generator (Duan et al., 2 Jun 2026). HeartMuLa likewise depends on structured lyrics, tags, and audio tokenization for controllable long-form song generation, and its framing suggests that high-quality song preparation is foundational for large-scale generative modeling (Yang et al., 15 Jan 2026).

Other task-specific systems can be read as neighboring layers rather than replacements. MeloDISinger addresses text-based singing voice editing under melody, duration, and non-edited-region preservation constraints, effectively operating after a song has already been prepared into aligned lyric and acoustic spans (Park et al., 29 Jun 2026). SongComposer and SongMASS target symbolic lyric–melody generation with alignment-aware representations and can use structured lyric and form information as upstream conditioning or evaluation context (Ding et al., 2024, Sheng et al., 2020). Llambada adds text-controlled accompaniment generation conditioned on vocals, and SRS provides single-stage complex-domain vocal restoration for severely degraded recordings, both of which suggest ancillary modules that could improve a SongPrep-like pipeline in practice (Trinh et al., 2024, Zang et al., 24 Oct 2025).

The primary limitations reported for SongPrep are boundary precision and data sensitivity. The pipeline remains better than SongPrepE2E on structure DER, while SongPrepE2E remains better on WER and RTF. The end-to-end model also becomes worse when filtering is too strict, because smaller datasets outweigh gains in label cleanliness. The paper further notes that structure analysis accuracy is still bottlenecked by training data scale, and that extremely long songs, heavy overlapping vocals, and underrepresented languages remain challenging (Tan et al., 22 Sep 2025).

A plausible implication is that future SongPrep-like systems will combine SongPrep’s whole-song structure supervision with note-aware unified transcription from SongTrans (Wu et al., 2024) or interleaved lyric–melody decoding from VocalParse (Chen et al., 6 May 2026). Another plausible implication is that restoration modules such as SRS could be inserted ahead of structure and lyric recognition when consumer-recorded vocals suffer from compound degradations, thereby improving supervision quality before dataset construction (Zang et al., 24 Oct 2025). Even in its present form, however, SongPrep establishes a distinct research object: full-song preprocessing as a first-class problem, with structure parsing, lyrics recognition, and timestamp generation treated as jointly necessary for scalable song-generation data curation (Tan et al., 22 Sep 2025).