VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models

Abstract: High-quality singing annotations are fundamental to modern Singing Voice Synthesis (SVS) systems. However, obtaining these annotations at scale through manual labeling is unrealistic due to the substantial labor and musical expertise required, making automatic annotation highly necessary. Despite their utility, current automatic transcription systems face significant challenges: they often rely on complex multi-stage pipelines, struggle to recover text-note alignments, and exhibit poor generalization to out-of-distribution (OOD) singing data. To alleviate these issues, we present VocalParse, a unified singing voice transcription (SVT) model built upon a Large Audio LLM (LALM). Specifically, our novel contribution is to introduce an interleaved prompting formulation that jointly models lyrics, melody, and word-note correspondence, yielding a generated sequence that directly maps to a structured musical score. Furthermore, we propose a Chain-of-Thought (CoT) style prompting strategy, which decodes lyrics first as a semantic scaffold, significantly mitigating the context disruption problem while preserving the structural benefits of interleaved generation. Experiments demonstrate that VocalParse achieves state-of-the-art SVT performance on multiple singing datasets. The source code and checkpoint are available at https://github.com/pymaster17/VocalParse.

• The paper proposes a unified framework that redefines singing voice transcription as a structured sequence generation task to jointly transcribe lyrics and melody.
• The paper introduces an interleaved prompting format paired with a Chain-of-Thought style that improves semantic continuity and note alignment.
• The paper demonstrates scalability by employing the SingCrawl pipeline to generate over 2,000 hours of pseudo-labeled data, boosting downstream SVS performance.

VocalParse: Unified and Scalable Singing Voice Transcription with Large Audio LLMs

Introduction and Motivation

Singing Voice Transcription (SVT) is a fundamental requirement for constructing modern Singing Voice Synthesis (SVS) systems, yet acquiring large-scale high-quality singing annotations is cost-prohibitive due to the need for musical expertise and significant manual labor. Existing automatic transcription pipelines rely on multi-stage systems that combine Automatic Speech Recognition (ASR), forced alignment, and melody transcription. These pipelines suffer from cascading errors, suboptimal text-note alignment, and poor generalization to out-of-distribution (OOD) styles, restricting scalability and robustness.

VocalParse addresses these challenges by casting SVT as a unified, structured sequence generation problem. Leveraging the capacity of Large Audio LLMs (LALMs), particularly Qwen3-ASR, and novel prompting strategies, VocalParse achieves joint lyric and melody transcription within a single autoregressive framework. It introduces two pivotal contributions: (1) an interleaved prompting format that natively encodes word-note alignment, and (2) a Chain-of-Thought (CoT) style prompting method that maintains compatibility with pretrained semantic decoding while supporting structured lyric-melody modeling.

Figure 1: VocalParse simplifies the traditional multi-stage SVT pipeline by unifying lyric and melody transcription within a single model.

Model Architecture and Prompting Paradigms

VocalParse operates on discrete audio tokens obtained at 12.5 Hz and formulates SVT as an autoregressive sequence generation over structured symbolic sequences. The central sequence consists of a pure lyric prefix followed by an interleaved lyric-note sequence, where each word token in the lyrics is immediately followed by symbolic note tokens encoding its melodic realization. The note vocabulary comprises 128 pitch tokens (MIDI numbers) and 12 duration tokens corresponding to standard musical durations, augmented by a global BPM token.

The core trade-off addressed is between preserving singing’s hierarchical structure (word-note correspondences) and maintaining the LALM's capacity for semantic decoding. Direct interleaving, though structurally faithful, disrupts LALM semantic modeling. The CoT-style prompting approach ameliorates this disruption by restoring a continuous lyric prefix before interleaved decoding, improving the model's robustness and accuracy in lyric generation and downstream note assignment.

Figure 2: Overview of VocalParse, its training regime with interleaved supervision and CoT-style prompting, and dual (audio-only/audio-lyric) inference modes.

Interleaved and CoT-Style Prompting

The interleaved format enables explicit word-to-note mapping: every lyric word is concatenated with its aligned note sequence, accommodating both one-to-one and melismatic (word-to-multiple-notes) relationships. However, this mixture of textual (word) and music (note) tokens challenges LALM’s standard ASR capabilities, leading to increased homophone errors and degraded semantic flow when applied naïvely.

CoT-style prompting resolves this by splitting decoding into two consecutive stages: first, the model outputs the complete lyrics (reinstating a continuous text context aligned with LALM pretraining); subsequently, it generates the interleaved word-note sequence, grounded on the preceding lyric output. This decomposition is empirically shown to reduce word error rates (WER) and improve pitch and note metric errors.

Figure 3: Illustration of CoT-style prompting: standard ASR decoding (top), direct interleaved decoding (middle), and CoT-style decoding (bottom) showing semantic scaffold preservation.

Data Pipeline: SingCrawl

Public singing datasets are insufficient in both diversity and fine-grained annotation, limiting the transfer of LALM capabilities to SVT. SingCrawl is introduced as a web-scale data pipeline for scalable pseudo-labeled singing segment creation.

SingCrawl executes the following:

1. Pre-filtering: Song selection via metadata constraints (language, style, lyric availability), quality metrics, and manual style filtering.
2. Audio Processing: Vocal extraction, dereverberation, segment alignment via silence detection, and quality control to remove artifacts.
3. Automatic Annotation: Retrained SOFA aligns word-level timestamps with mixed weak/full label supervision; ROSVOT assigns note boundaries and pitch, with conversion into discrete note tokens and global BPM estimation.

This pipeline supports the generation of over 2,000 hours of training data, directly compatible with VocalParse’s structured sequence format.

Figure 4: End-to-end data flow of the SingCrawl pipeline, illustrating the transformation from raw web songs to pseudo-labeled SVT data.

Note Quantization Algorithm

Precise symbolic duration labeling necessitates reliable BPM estimation and quantization, as web data rarely provides tempo metadata. SingCrawl uses an EM-like algorithm, assigning each observed note duration to the closest quantized value and iteratively refining BPM for best fit. This quantization ensures symbolic consistency and model compatibility.

Figure 5: Note quantization process: estimating BPM and mapping continuous note durations to discrete symbolic note tokens.

Experimental Results and Analysis

Automatic Melody Transcription (AMT): VocalParse achieves state-of-the-art performance on Opencpop and ACE-KiSing benchmarks, outperforming prior methods such as ROSVOT, MusicYOLO, and STARS, even in strictly audio-only settings. In lyric-conditioned inference, VocalParse sets a new best for $MAE_{pitch}$ (0.35), $MAE_{note}$ (0.43), $MAE_{dur}$ (0.33), and note count error (0.11). The model both surpasses the original teacher pipeline and demonstrates effective label distillation beyond imitation of noisy pseudo labels.

Automatic Lyric Transcription (ALT): Despite its unified modeling objective, VocalParse matches or exceeds specialized ASR models, achieving a WER of 3.79% on Opencpop and remaining competitive across other datasets.

Ablation Studies: Removing CoT-style prompting or large-scale SingCrawl data each results in notably degraded performance, manifested in higher WER and pitch errors. This empirically validates the necessity of both contributions.

Practical Implications and Downstream SVS Impact

The practicality of VocalParse-generated annotations is evaluated via downstream SVS experiments, using DiTAR as the backbone. Results confirm that expanding SVS training data with VocalParse-annotated SingCrawl segments yields substantial improvements in rhythm and melody similarity metrics (Boundary Error Rate, IOU, RPA), while preserving aesthetic quality as measured by SingMOS and subjective AB preference tests.

Figure 6: Validation loss of SVS decreases monotonically as more VocalParse-annotated data is introduced, indicating improved model generalization.

Figure 7: AB preference test interface; subjective evaluation confirms enhanced perceived quality from models trained with large-scale VocalParse annotations.

Limitations and Future Directions

VocalParse’s current BPM estimation uses a global tempo, limiting performance for expressive singing with local tempo variations. Autoregressive decoding can occasionally introduce inconsistencies between lyric prefix and interleaved generation. While VocalParse distills over teacher pipelines, its theoretical performance upper bound is ultimately tethered to teacher label quality. The experiments are currently restricted to Mandarin; future extensions should include adaptation to multilingual contexts and further structural refinements for music with complex tempo and articulation.

Conclusion

VocalParse represents a comprehensive advance in unified singing voice transcription, leveraging LALMs, a scalable web-based data pipeline, and structured prompting. Its formulation enables robust, state-of-the-art lyric and melody transcription within a single causal framework, directly supports sequenced-symbolic output, and provides scalable annotation with practical downstream SVS gains. As LALMs and scalable annotation pipelines continue to evolve, VocalParse sets a benchmark for unified audio-text-music modeling with direct implications for contemporary and future music information retrieval and synthesis systems.