Listening Like a Judge: A Music-Aware Framework for Automatic Singing Performance Evaluation

Abstract: Automatic singing quality assessment (SQA) requires evaluating lyrical correctness and musical fidelity while handling expressive variations. However, existing systems largely rely on either acoustic cues or lyric transcriptions exclusively, limiting holistic performance evaluation. Furthermore, their integration is non-trivial due to challenges in robust singing transcription amid melisma, vibrato, and tempo elasticity. To this end, we propose MusicJudge, a modality-guided framework for automated SQA that performs block-aligned multimodal analysis by coupling lyric correctness with pitch-rhythm fidelity. It detects semantically meaningful lyric blocks using multi-signal matching that integrates semantic embeddings, lexical similarity, and phonetic alignment. To improve singing audio transcription, we introduce Modality-Guided LoRA for ASR fine-tuning. Experiments across datasets demonstrate strong agreement with human expert judgments and validate the generalizability of MusicJudge.

• The paper presents MusicJudge, a multimodal framework that integrates ASR-based lyric segmentation and musical analysis to assess singing performance.
• It introduces Modality-Guided LoRA (MG-LoRA) for singing-adapted ASR, achieving up to 29.87% reduction in word error rate and improved robustness across genres.
• The framework fuses lyrical, pitch, and rhythm metrics via block-level scoring, resulting in strong alignment with expert judge rankings.

MusicJudge: A Multimodal, Music-Aware Framework for Automatic Singing Performance Evaluation

Motivation and Problem Statement

Automated Singing Quality Assessment (SQA) demands a nuanced evaluation of both lyrical correctness and musical fidelity while accommodating natural expressive variations such as melisma, vibrato, and improvisational phrasing. Existing approaches exhibit significant limitations, predominately leveraging either acoustic features or textual transcription in isolation, and struggle under the varied, noisy, and expressive conditions of real performance. Conventional signal-level comparisons penalize valid musical deviations, while purely transcript-based pipelines fail to capture underlying musical structure and permissible lyrical transformation. Segmental misalignment, due to vowel elongation or free-form improvisation, further degrades the robustness of current systems.

MusicJudge (\textbf{MusicJudge}), the framework presented in this paper, addresses these challenges using a block-aligned, multimodal analysis strategy. It combines advanced ASR techniques, structurally informed segmentation, and joint modeling of lyrical, pitch, and rhythm fidelity. Notably, MusicJudge introduces Modality-Guided LoRA (MG-LoRA) for singing-adapted ASR fine-tuning and an explicit aggregation pipeline for interpreting holistic performance scores.

Figure 1: MusicJudge jointly evaluates the content and musical aspects of singing audio.

System Architecture and Methodology

MusicJudge dissects a singing performance into temporally and semantically meaningful blocks. This design enables evaluation both at local (verse, chorus) and global levels, integrating lyrical accuracy and musical fidelity.

Block Segmentation and Reference Alignment

Time-aligned lyric segments are first determined using ASR-derived proto-segments, which are robustified by merging into overlapping sliding windows—accounting for expressive uncertainties. Each candidate block is assessed and refined using multi-signal logic: semantic embedding similarity, fuzzy lexical matching, and phonetic proximity are computed relative to reference lyrics. This composite alignment produces blocks corresponding to musically and linguistically coherent song sections, and enhances resilience to structural uncertainty inherent in live singing.

Modality-Guided LoRA for Singing ASR

To mitigate ASR errors induced by singing-specific distortions, the authors fine-tune whisper-large-v3 using a novel MG-LoRA approach. The training objective augments standard cross-entropy loss with terms that penalize unstable token durations, enforce monotonic alignment, reinforce pitch-contour continuity, and synchronize predicted boundaries with vocal onsets. The SwaraLyrics dataset, which includes Indian solo songs with a broad variety of performance artifacts (audience noise, timed commentary), is used for effective domain adaptation. Empirically tuned regularization coefficients are shown to strongly influence ASR robustness across multiple language and genre conditions, yielding substantial performance improvements.

Block-Level Lyrical and Musical Scoring

For each detected block $B_k$, lyrical accuracy is quantified as a weighted combination of semantic embedding similarity, fuzzy edit distance, and phonetic match. This multi-signal construct—shown via ablations to decisively outperform single-signal strategies—yields the content score $\mathcal{C}_k$.

Musical fidelity is jointly assessed via pitch and rhythm analysis:

• Global Key Estimation: Uses chroma profile matching on accompaniment to extract the global key, avoiding erroneous penalization from intentional transpositions.
• Pitch Deviation: Employs pYIN to extract vocal contours, quantifying in-key distance, stability, and voiced fraction. These are aggregated as a normalized penalty term underpinning the pitch score $\mathcal{P}_k$.
• Rhythmic Deviation: Quantifies timing offset between vocal onsets and accompaniment beat grid, integrating timing error, bias, and temporal stability for the rhythm score $\mathcal{R}_k$.

The final musical score for each block, $\mathcal{M}_k$, fuses pitch and rhythm fidelity, with aggregation weights determined by validation set correlation to expert human scoring.

Structured Aggregation and Feedback

Block-wise scores are aggregated with duration-proportional weights, producing a unified performance metric $\mathcal{S}(x, \mathcal{G})$ that exhibits strong correlation with expert judge rankings. Additionally, MusicJudge supports the generation of detailed, block-aware natural language feedback using a specialized LLM, facilitating actionable performance critique.

Empirical Results and Ablation Analysis

Quantitative evaluation on the SwaraLyrics and SingMOS-Pro datasets demonstrates that MusicJudge achieves strong correlation with human judges: Spearman $\rho=0.683$ on SwaraLyrics, a 32% absolute improvement over strong baselines, and $\rho=0.483$ on SingMOS-Pro (2606.26451). The methodology provides robust generalization across genres and languages, with MG-LoRA yielding up to 29.87% reduction in word error rate (WER) over the next best method (see below).

(Figure 2)

Figure 2: Singing ASR transcription accuracy (WER/CER) across architectures and datasets, highlighting MG-LoRA’s substantial improvement.

Ablation studies confirm the necessity of joint content and musical modeling. Evaluations using only content or only musical cues yield lower correlation compared to the full pipeline. The multi-signal lyric alignment further outperforms all single-signal variants, particularly under noisy or structurally ambiguous conditions.

Cross-genre and cross-lingual robustness is empirically validated; MG-LoRA reduces WER by $20.1\pm7.52\%$ and $27.7\pm10.87\%$ across genres and languages, respectively.

Qualitative analysis (see supplementary materials cited in the paper) confirms that MG-LoRA improves lyric recognition in the presence of melisma, ornamentation, and live performance noise. The system-generated natural language feedback is semantically aligned with expert commentaries (cosine similarity 63.97 using MiniLM embeddings).

Implications and Future Directions

MusicJudge advances the state-of-the-art in SQA by constructing an integrated, interpretable measure of singing performance that captures both linguistic and musical dimensions at fine temporal granularity. The MG-LoRA adaptation strategy for ASR and the multi-signal block matching represent decisive steps beyond current paradigm limitations, providing meaningful gains in correlation with expert opinion and substantial gains in ASR robustness.

Practically, this approach enables scalable and unbiased evaluation in assistive singing pedagogy, synthetic music generation, and music competition automation. Theoretically, the probabilistic interpretation of block penalties suggests potential for Bayesian extensions and unsupervised music-structure modeling.

Potential future research avenues include multi-singer diarization-aware MG-LoRA adaptation, extension to expressive dimensions beyond pitch and rhythm (e.g., vibrato, timbral nuance), and integration with generative audio-feedback systems for real-time SQA.

Conclusion

MusicJudge provides an effective framework for automatic, explainable assessment of singing performance, substantially enhancing system agreement with human judges by employing block-aligned, multimodal analysis and robust ASR adaptation tailored for singing. Its architecture, grounded in both information-theoretic and music-theoretic principles, establishes a scalable foundation for subsequent SQA research, practical music education tools, and more sophisticated music understanding systems in AI.