SongEval: Unified Music Evaluation Framework

• SongEval is a comprehensive framework that integrates computational methods and expert-annotated datasets to assess music aesthetics and lyric translation.
• It employs deep learning models and statistical metrics to quantify dimensions such as coherence, memorability, naturalness, clarity, and overall musicality.
• The framework supports multi-language, multi-genre evaluation, enabling reproducible and human-aligned assessments of both song generation and translation tasks.

SongEval is an umbrella term for a set of computational frameworks and datasets enabling the systematic evaluation of music generation, with an emphasis on the automatic, reproducible measurement of human-perceived aesthetic and structural qualities—including singability, musicality, and nuanced song characteristics—across diverse musical tasks. Most notably instantiated by "SongEval: A Benchmark Dataset for Song Aesthetics Evaluation" (Yao et al., 16 May 2025) and the lyric translation evaluation framework of (Kim et al., 2023), SongEval provides both the protocols and open-source resources to align algorithmic assessment with the nuanced judgments of professional music annotators and listeners.

1. Benchmark Datasets and Evaluation Dimensions

SongEval, as operationalized in (Yao et al., 16 May 2025), comprises a large-scale resource of 2,399 full-length songs (140.32 hours), with coverage of nine mainstream genres in both Chinese (1,093) and English (1,306) and annotation across five salient dimensions:

• Overall Coherence: Musical and emotional continuity throughout the song, including section transitions and unity of tone.
• Memorability: Presence and distinctiveness of hooks, motifs, or lyrical elements increasing recall.
• Naturalness of Vocal Breathing & Phrasing: Alignment of breathing and phrasing with syntactic and rhythmic structure, detecting unnatural breaks.
• Clarity of Song Structure: Detectability and logic of segmentations (e.g., verse, chorus, bridge), whether adhering to conventions or demonstrating innovative but musically sound segmentation.
• Overall Musicality: Broad musical enjoyment, covering melodic, harmonic, instrumental, and vocal-instrumental integration.

Each song is independently annotated by four professional raters (conservatory students, industry practitioners), using 1–5 integer scales for each dimension.

In the lyric translation context, SongEval is adapted to quantitatively assess the singability and musical conformality of translated lyrics using four algorithmic metrics: line syllable count distance, phoneme repetition similarity, musical structure distance, and semantic similarity (Kim et al., 2023). This extension is supported by a multilingual dataset of 162 songs with line- and section-level alignments in English, Japanese, and Korean.

2. Annotation Protocols and Inter-Rater Reliability

For song aesthetic evaluation (Yao et al., 16 May 2025), a third-party professional annotation service administers blinded web-based rating sessions, pairing audio playback with synchronized spectrograms. Raters receive detailed written and audio-visual guidelines and are compensated \$5 per song. Each dimension is annotated independently on a 1–5 scale, with aggregation by per-song mean. No explicit inter-rater reliability statistics (e.g., Cohen’s κ, Cronbach’s α) are published, but formulae for their computation are provided:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ denotes observed and $p_e$ chance agreement.

In lyric translation (Kim et al., 2023), line and section alignments are manually curated to ensure faithful evaluation of syllabic and structural properties. Syllable and phoneme counts are language-specific: ARPAbet for English, character-to-CV for Japanese, and dictionary-assisted Unicode decomposition for Korean, with language-specific vowel clustering.

3. Quantitative Metrics and Computational Framework

SongEval evaluation embraces both prediction of human ratings and computational quantification of translation or structural fidelity:

For full-song aesthetics (Yao et al., 16 May 2025):

• Models regress the five aesthetic dimensions from audio using deep learning architectures such as MOSNet (CNN + BLSTM), LDNet, SSL-based MuQ, and ensemble UTMOS predictors.
• Metrics:
  • Mean Squared Error (MSE)
  • Pearson’s r (LCC)
  • Spearman’s ρ (SRCC)
  • Kendall’s τ (KTAU)
• Levels: Utterance (per rating), System (per song generator)

For singable lyric translation (Kim et al., 2023):

• Line Syllable Count Distance

$$Dis_{syl}(X, \hat Y) = \frac{1}{2n} \sum_{i=1}^n \left( \frac{|s_i - t_i|}{s_i} + \frac{|s_i - t_i|}{t_i} \right)$$

• Phoneme Repetition Similarity

$$Sim_{pho}(X, \hat Y) = \text{SpearmanCorr}\left(\left[\text{pho}(X_1), \ldots, \text{pho}(X_m)\right],\left[\text{pho}(\hat Y_1), \ldots, \text{pho}(\hat Y_m)\right]\right)$$

with

$$\text{pho}(S) = \frac{\# \text{unique bigrams}}{\# \text{bigrams}}$$

• Musical Structure Distance

$$Dis_{mus}(X, \hat Y) = \frac{1}{m^2} \sqrt{\sum_{i,j=1}^m [D_{ij} - \hat D_{ij}]^2}$$

• Section-level Semantic Similarity

$$Sim_{sem}(X, \hat Y) = \sum_{i=1}^m \frac{n_i}{N} \cdot sts(X_i, \hat Y_i)$$

where $sts(\cdot, \cdot)$ is Sentence-BERT cosine similarity.

Python pseudo-code is referenced for each metric, e.g.:

def line_syllable_distance(src_lines, tgt_lines): ...

Table: Example System-level Results for UTMOS on SongEval (Yao et al., 16 May 2025)

Dimension	MSE	LCC	SRCC	KTAU
Coherence	0.073	0.962	0.954	0.844
Memorability	0.096	0.955	0.958	0.849
Naturalness	0.081	0.957	0.941	0.809
Clarity	0.091	0.951	0.939	0.804
Musicality	0.072	0.966	0.969	0.859

4. Empirical Findings and Baseline Performance

In (Yao et al., 16 May 2025), baseline models trained on SongEval exhibit high alignment with professional human ratings, particularly for system-level SRCC (coherence: 0.954, memorability: 0.958, naturalness: 0.941, clarity: 0.939, musicality: 0.969) with corresponding Pearson’s r values exceeding 0.95, markedly outperforming conventional objective audio metrics (e.g., CE, CU from AudioBox yield r ≈ 0.61–0.66).

For lyric translation (Kim et al., 2023), metrics robustly separate singable and non-singable translations. For example, EN→JP singable pairs have average Disₛᵧₗ = 0.17 (vs. 0.74 for non-singable), Simₚₕₒ = 0.69 (vs. 0.56), with statistical significance established via default Spearman p-value < 0.05.

Empirical analysis links line-level metrics to concrete musical constraints (e.g., phrasing, rhythmic fit), while section-level semantics support preservation of global meaning.

5. Comparative Evaluation and Extensions

SongEval provides a unified platform for evaluating both algorithm performance (automatic predictors, e.g., UTMOS, SSL-MuQ) and the adequacy of generated musical artifacts against human aesthetic standards. The architecture generalizes across languages (Chinese, English, Japanese, Korean) and genres (nine mainstream types, animation, K-pop), facilitating cross-domain comparative analysis.

The underlying framework permits adaptation across further musical and linguistic domains—contingent upon appropriate definition of syllabification, phoneme mapping, and section segmentation—without requiring changes to core metric formulations. Notable implementation details include the need for language- and genre-adjusted preprocessing (G2P, vowel clustering, robust section alignment), and the potential for weighting metric contributions to reflect genre-specific structural priorities (e.g., upweighting phoneme repetition in rap).

6. Limitations and Future Research

Limitations of current SongEval instantiations include dimension entanglement (e.g., overlap between coherence and structure), the absence of explicit inter-rater reliability statistics in several released datasets, and potential underrepresentation of certain genres and linguistic typologies. The datasets focus on English and Chinese languages for full-song evaluation, and the lyric translation corpus—though trilingual—targets a specific subset of genres.

Future work, as outlined in (Yao et al., 16 May 2025) and (Kim et al., 2023), targets finer-grained and more style-robust automatic aesthetics evaluators, disentanglement of correlated subjective dimensions, adaptation for tone-sensitive languages, integration of note-pitch and MIDI alignments, and the extension of framework metrics to properly account for form, story, and musical-theatrical content.

7. Significance and Broader Impact

SongEval—the combination of open, expert-annotated datasets and rigorously defined computational metrics—represents a foundational resource for the evaluation of music generation and translation models. It enables objective measurement of subjective qualities using standard machine learning protocol, facilitating human-aligned development and benchmarking in the era of generative music systems. The released datasets and code sketches allow for direct reproduction and flexible extension, supporting the broader goal of reproducible, comparable research in AI-powered music generation and analysis (Yao et al., 16 May 2025, Kim et al., 2023).