VocalParse: Unified Singing Voice Transcription

• VocalParse is a unified singing voice transcription model that jointly decodes lyrics, note-level melody, and explicit word–note alignment into a structured symbolic score.
• It employs a two-stage chain-of-thought prompting strategy that first generates a lyrics-only scaffold, then interleaves note tokens to maintain semantic and musical continuity.
• The approach overcomes cascaded pipeline limitations, achieving improved error rates and precise melody transcription, enabling scalable annotation for singing voice synthesis.

VocalParse is a unified singing voice transcription (SVT) model built on a Large Audio LLM (LALM). It treats SVT as a single sequence generation problem that jointly recovers lyrics, note-level melody, and explicit word–note correspondence, producing a structured symbolic score directly usable for Singing Voice Synthesis (SVS). Its central design combines an interleaved sequence formulation, in which each word is followed by its associated notes, with a Chain-of-Thought (CoT) prompting strategy that first decodes a lyrics-only scaffold and then decodes the interleaved lyric–note structure (Chen et al., 6 May 2026).

1. Problem definition and scope

VocalParse targets a version of SVT in which the output must capture three coupled objects: lyrics $L$, melody $N$, and alignment $A$. In the formulation reported for the model, lyrics are represented as words or subword tokens, melody is represented as a sequence of discrete notes with pitch $p$, duration $d$, and implicit onset $t$, and alignment assigns each word $w_i$ to its realized notes, allowing melisma and yielding a structured musical score suitable for SVS training and MusicXML/MIDI export (Chen et al., 6 May 2026).

The model is motivated by the limitations of prior SVT pipelines. Those pipelines typically decompose the task into ASR, word or phoneme alignment, melody transcription, and downstream merging. The paper characterizes such systems as brittle, dependent on hand-tuned interfaces, and vulnerable to cascading errors. It also identifies poor text–note alignment as a core weakness of systems that predict lyrics and melody independently, because those systems require a separate forced-alignment stage and do not natively model melisma or local timing correlations. A further limitation is out-of-distribution degradation on singing data with large pitch ranges, prolonged vowels, ornamentation, and varied singing styles (Chen et al., 6 May 2026).

This formulation places VocalParse closer to symbolic score generation than to isolated subtasks such as vocal separation or framewise pitch tracking. A plausible implication is that its unit of prediction is not merely an acoustic contour but a musically structured object in which textual and melodic organization are explicitly coupled.

2. Interleaved sequence representation

The model recasts SVT as autoregressive generation over a structured token sequence. For a word $w_i$ with $K_i$ notes, VocalParse emits the word followed by note pairs:

$$(p_{i,1}, n_{i,1}), \ldots, (p_{i,K_i}, n_{i,K_i}),$$

with a single <[BPM](https://www.emergentmind.com/topics/behavior-planner-module-bpm)> token appearing once per sequence as a tempo suffix. Pitch uses 128 discrete <PITCH_x> tokens mapped to MIDI $N$0–$N$1, while duration uses a 12-token vocabulary of standard and dotted note values, including tokens such as <NOTE_8> and <NOTE_DOT_4> (Chen et al., 6 May 2026).

The paper formalizes the CoT target as

$N$2

where $N$3 is the pure-lyrics scaffold and $N$4 is the interleaved lyric–note sequence. It also gives the equivalent factorization over latent musical structure,

$N$5

Training uses sequence-level cross-entropy over the full target sequence:

$N$6

Monotonicity is enforced implicitly by the interleaving: every note group for a word follows that word and precedes the next word, while melisma is represented by $N$7 (Chen et al., 6 May 2026).

The representation makes onsets reconstructible rather than explicitly predicted. VocalParse does not emit onset tokens; instead, onsets are recovered deterministically from note order, note durations, and BPM. Given note value $N$8 in quarter-note units, duration in seconds is computed as

$N$9

This design encodes word–note alignment directly in the output stream rather than as a post hoc alignment artifact (Chen et al., 6 May 2026).

Token type	Representation	Role
Lyrics	Native language tokens	Semantic scaffold
Pitch	128 <PITCH_x> tokens	MIDI-coded melody
Duration	12 note-value tokens	Symbolic rhythmic value
Tempo	Single <BPM_xxx> token	Global tempo for reconstruction

A common misconception is that unified generation simply concatenates lyrics and notes. In VocalParse, the crucial point is not mere concatenation but interleaving: each word is immediately associated with its local note group, so the sequence itself encodes correspondence structure (Chen et al., 6 May 2026).

3. Model architecture and prompting strategy

VocalParse is built on Qwen3-ASR with 1.7B parameters. Audio is converted to discrete tokens at 12.5 Hz and fed to a causal decoder that predicts a single token stream. No architectural change is introduced specifically for joint SVT; the paper states that the structure is carried by the target sequence design rather than by a specialized acoustic-symbolic fusion module (Chen et al., 6 May 2026).

The paper’s main modeling claim is that direct interleaving of music tokens between words disrupts the continuous text context critical for pretrained ASR decoding. VocalParse addresses this with a CoT-style prompting strategy. Stage 1 decodes lyrics only as a pure semantic prefix,

$A$0

and Stage 2 decodes the interleaved lyric–note sequence conditioned on that prefix. The stated purpose is to preserve text continuity and semantics, reduce homophone errors, and still retain the structural advantages of interleaved generation (Chen et al., 6 May 2026).

Two inference modes are supported. In audio-only inference, the model first decodes lyrics from audio tokens and then continues decoding the interleaved lyric–note structure. In audio+lyric inference, lyrics are supplied as a prefix and the model decodes only the interleaved lyric–note continuation, thereby eliminating lyric-recognition error propagation. The paper also describes constrained decoding rules: <BPM_xxx> should appear once as a suffix, duration tokens should alternate with pitch tokens within a word’s note group, and invalid token orders can be rejected during decoding (Chen et al., 6 May 2026).

This prompting strategy is central rather than auxiliary. The ablation reported for removing CoT causes large degradations on Opencpop audio-only evaluation: WER rises to 7.18%, while MAE_pitch, MAE_note, and MAE_dur become 0.92, 0.46, and 0.38, respectively. The paper therefore treats CoT as critical for preserving semantic decoding and improving melody prediction (Chen et al., 6 May 2026).

4. Data construction, pseudo-labeling, and training

The training regime combines public singing corpora with a large pseudo-labeled corpus. SingCrawl, introduced in the work, begins from approximately 65k crawled songs and yields approximately 1.7M singing segments and approximately 2000 hours of usable training audio with pseudo-labels for lyrics, word boundaries, notes, and BPM. Public corpora contribute an additional approximately 50 hours from the Chinese subset of GTSinger plus M4Singer (Chen et al., 6 May 2026).

The SingCrawl processing pipeline consists of metadata filtering, segmentation with silence snapping, vocal extraction and dereverberation using the mel-RoFormer family, forced alignment with SOFA retrained for singing using mixed weak/full labels, note boundary and pitch extraction with ROSVOT, and BPM estimation via an EM-like estimator. BPM is normalized to the range $A$1 (Chen et al., 6 May 2026).

Training initializes from Qwen3-ASR. The reported setup uses 2× NVIDIA H100 with DDP and dynamic batching constrained to at most 18k tokens per GPU and at most 64 samples per GPU. Training runs for 120k steps in approximately 17 hours, with a cosine learning-rate schedule, 12k warmup steps, and peak learning rate $A$2 (Chen et al., 6 May 2026).

The paper frames scalability as a direct consequence of the unified formulation. By removing multi-stage cascades and their interfaces, the model can generate self-contained symbolic outputs suitable for automatic SVS annotation at large scale. This suggests that the dataset-construction pipeline is not merely auxiliary infrastructure but an enabling component of the model’s empirical performance and out-of-distribution robustness.

5. Benchmarks, ablations, and downstream impact

The evaluation covers automatic melody transcription (AMT), automatic lyric transcription (ALT), ablations, and downstream SVS. On Opencpop in audio+lyric mode, VocalParse reports MAE_pitch 0.35, MAE_note 0.43, MAE_dur 0.33, and Num_note 0.11, outperforming ROSVOT and also exceeding MusicYOLO and STARS across most metrics. In audio-only mode on the same dataset, it reports 0.56, 0.44, 0.34, and 0.11. On ACE-KiSing audio-only evaluation, it reports 0.53, 0.52, 0.49, and 0.29, again better than STARS and MusicYOLO (Chen et al., 6 May 2026).

For lyric transcription, reported WER values are 3.79 on Opencpop, 5.69 on OpenSinger, and 8.16 on PopCS. These results are competitive with Qwen3-ASR itself and substantially better than LyricWhiz and Whisper-adapted baselines on the cited datasets (Chen et al., 6 May 2026).

Dataset / mode	Reported result
Opencpop, audio+lyric	MAE_pitch 0.35, MAE_note 0.43, MAE_dur 0.33, Num_note 0.11
Opencpop, audio-only	0.56 / 0.44 / 0.34 / 0.11
ACE-KiSing, audio-only	0.53 / 0.52 / 0.49 / 0.29
Opencpop ALT	WER 3.79
OpenSinger ALT	WER 5.69
PopCS ALT	WER 8.16

The ablations identify two components as decisive. Removing CoT degrades WER to 7.18% on Opencpop audio-only evaluation, and removing SingCrawl produces WER 4.86%, MAE_pitch 0.94, MAE_note 0.47, MAE_dur 0.37, and Num_note 0.11. The paper interprets these results as evidence that CoT preserves semantic decoding and that large-scale data are essential for robust pitch and duration estimation (Chen et al., 6 May 2026).

The work also reports downstream SVS impact using DiTAR trained on VocalParse-annotated SingCrawl subsets. As training data scale from 200 h to 2000 h, rhythm and melody controllability improve without harming perceptual quality: SingMOS, CE, and PQ remain nearly unchanged, while RPA and rhythm IOU improve markedly. This suggests that VocalParse is intended not only as an evaluation model for SVT, but as an annotation engine for large-scale corpus construction (Chen et al., 6 May 2026).

6. Limitations, failure modes, and research context

The paper explicitly notes several limitations. First, VocalParse assumes a single global tempo per segment; rubato and ritardando can therefore induce duration mismatches. Suggested mitigations include local tempo tokens, time-warp splines, or post hoc dynamic time warping. Second, semantic drift can occur between the lyric scaffold $A$3 and the replicated words inside the interleaved sequence; the proposed mitigation is to copy surface forms from $A$4 when rendering the final score or to impose hard decoding constraints. Third, the quality ceiling remains limited by pseudo-label noise in the teacher pipeline, even though the model surpasses the SOFA+ROSVOT pipeline on the reported benchmarks. Fourth, the system is trained and evaluated on Mandarin, so multilingual generalization would require language-aware tokenization and lexica (Chen et al., 6 May 2026).

The paper also identifies concrete acoustic failure modes: fast melisma and ornamentation can cause fragmentation or under-segmentation; vibrato can induce spurious micro-notes; accompaniment leakage from source separation degrades alignment; and non-solo contexts such as duets or chorus violate the single-voice assumption. Recommended mitigations include minimum-duration pruning, adjacent same-pitch merging, pitch smoothing before quantization, stricter source-separation quality thresholds, and future multi-source lyric/melody disentangling (Chen et al., 6 May 2026).

Within the broader literature, VocalParse differs from systems that center separation and pitch extraction rather than unified symbolic generation. DJCM is a deep joint cascade model for singing voice separation and vocal pitch estimation, designed to align train–test conditions between those two tasks (Wei et al., 2024). Mel-RoFormer addresses vocal separation and vocal melody transcription through a spectrogram-based model with Mel-band Projection and interleaved RoPE Transformers, using a two-step separation-first then fine-tuning procedure (Wang et al., 2024). Perceiver TF treats vocals as one instrument class within multitrack music transcription and predicts onset/frame pianorolls rather than explicit word–note correspondence (Lu et al., 2023). VocalParse’s distinctive claim is therefore not simply better melody or lyric recognition in isolation, but direct generation of lyrics, melody, and alignment in a single structured output (Chen et al., 6 May 2026).

A second misconception is that a stronger ASR backbone alone should solve SVT. The reported CoT ablation argues against that view: direct interleaving without the lyrics-first scaffold disrupts semantic context enough to materially degrade both lyrics and melody. The model’s contribution is thus as much about output formulation and prompting as about backbone scale (Chen et al., 6 May 2026).