TuneJury: An Open Metric for Improving Music Generation Preference Alignment

Abstract: We introduce TuneJury, an open, instance-level pairwise reward model for text-to-music that predicts a music preference score from a text prompt and an audio clip. The released checkpoint is trained on publicly available human-preference labels covering arena-style (A vs. B) votes, metric-alignment preference pairs, crowdsourced pairwise comparisons, and expert aesthetic ratings. The predicted score margin between two clips is well calibrated on our held-out test split, supporting data filtering via a simple score threshold. TuneJury generalizes to both held-out test pairs and out-of-distribution benchmarks, remaining competitive with prior baselines on the latter. For generators released after training, we introduce anchor calibration, a post-hoc, per-system Bradley-Terry calibration that recovers agreement at substantially better data efficiency than from-scratch retraining. The same frozen reward drives consistent reward-axis gains across three downstream applications: inference-time best-of-N selection, DITTO-style latent optimization, and expert-iteration post-training. TuneJury is available at https://github.com/yonghyunk1m/TuneJury.

• The paper introduces an instance-level pairwise reward model that improves music generation preference alignment using only human-labeled data.
• It employs a 4-layer MLP over concatenated audio and text embeddings, achieving robust calibration and competitive performance across multiple benchmarks.
• The model supports applications such as best-of-N selection, latent optimization, and expert-iteration post-training, with rapid out-of-distribution adaptation via anchor calibration.

TuneJury: An Open Metric for Improving Music Generation Preference Alignment

Introduction and Motivation

Evaluating text-to-music (TTM) generation models presents unique challenges due to the inherently subjective nature of musical preference. Traditional metrics such as Fréchet Audio Distance (FAD) and its variants, while useful for measuring audio distribution similarity, do not directly capture human musical preference at the instance level. Mean opinion score (MOS) regression, although common in adjacent domains like speech, is sensitive to rater-scale drift and thus unreliable for subjective elements such as musicality. Instead, pairwise preference modeling, which asks which instance is preferred in A-vs-B comparisons, offers a more precise, variance-reducing approach that aligns with how subjectivity can be meaningfully quantified in TTM settings.

Model Architecture and Training

TuneJury is designed as an instance-level pairwise reward model for TTM, outputting a scalar preference score for any (prompt, audio) pair. The architecture comprises a multilayer perceptron (MLP) head over fixed, pretrained audio and text encoder embeddings. Specifically, in the CLAP+MERT configuration, the input features are a concatenation of:

• LAION-CLAP-Music: $512$-d audio and $512$-d text embeddings
• MERT-v1-330M: $1024$-d audio embedding

These are concatenated to yield a $2048$-d input which passes through a $4$-layer MLP ($1024, 512, 256, 128$ widths), totaling approximately $2.8$M parameters.

For training supervision, TuneJury leverages a curated pool of $\sim$17.5K human-annotated preference pairs sourced from Music Arena, MusicPrefs, AIME, and SongEval. This stands in contrast to prior models such as CMI-RewardModel (CMI-RM), which utilize both pseudo-labeled and human-labeled data and operate with a broader input set (including lyrics and reference audio). Importantly, TuneJury is trained only with human-labeled pairs and on (prompt, audio) as the input, making the approach both leaner and more robust for instrumental music evaluation.

The model is trained with the RankNet-style pairwise logistic objective, minimizing the binary cross-entropy on the score difference between pairs.

Evaluation and Comparative Analysis

Internal Calibration and Robustness

On the held-out test set of $2,035$ non-tied pairs, TuneJury achieves a pairwise accuracy of $0.7086$ with an expected calibration error (ECE) of $512$0. Confidence, as inferred from score margins, monotonically tracks empirical accuracy, indicating reliable model calibration across margin bins.

Ablation analyses demonstrate that every dataset in the training mix carries distinctive value, but the model maintains robustness across input modality and encoder choices. For example, replacing the CLAP+MERT stack with MuQ-MuLan embeddings resulted in comparable or superior out-of-distribution (OOD) generalization on several external benchmarks.

External Benchmarks

TuneJury is benchmarked against five significant prior works on the CMI-RewardBench suite, including pairwise models (CMI-RM) and MOS regressors (SongEval-RM, Audiobox-Aesthetics, MuQ-Eval) as well as zero-shot audio-LLMs (PAM score).

Quantitatively, TuneJury trained on human-only annotations (no pseudo-labels) yields substantial performance:

• Pairwise accuracy on CMI-RewardBench MA: $512$1
• PAM and MusicEval Spearman rank correlations: mean SRCC improvement of $512$2 over the no-pseudo CMI-RM ablation
• Competitive results within $512$3–$512$4 percentage points of the full pseudo-augmented CMI-RM on OOD splits

The model's calibration also translates to real-world data as shown by instance-level numerical probes into OOD test splits and calibration reliability diagrams.

Per-System and Cross-Domain Evaluation

TuneJury's reward ordering tracks human win rates on held-out test splits extremely well (AIME: $512$5, MusicPrefs: $512$6). Further, external singing-voice MOS validation (SingMOS-Pro, SVCC 2025) finds that per-system rankings produced by TuneJury correlate with human vocal MOS at moderate strength.

Downstream Applications

TuneJury is evaluated in three operational modes, all using the same frozen reward model:

Mode 1: Best-of-N Selection

Generated $512$7 samples per prompt are ranked by TuneJury reward; the top-1 is selected. Across four TTM backbones—spanning transformers and diffusion models—reward is strictly monotonic in $512$8 up to $512$9.

Figure 1: Mode~1 best-of-N sweep ($1024$0) showing monotonic reward gains across architectures.

Mode 1 also improves alignment as a byproduct, evidenced by rising CLAP scores with increasing $1024$1. However, distributional metrics (FAD-CLAP, FAD-MERT, MAD) do not always agree in their assessment of improved fidelity, revealing a classic tension between selection for reward and other aspects of audio quality.

Mode 2: DITTO-Style Latent Optimization

In a reward-driven inference-time optimization setup, back-propagating TuneJury into the latent variables (e.g., the initial noise for diffusion) increases mean reward, with the effect size dependent on baseline model quality. For weaker baselines (e.g., TangoFlux), optimization improves both reward and CLAP alignment without substantially harming distributional fidelity, whereas for stronger baselines (e.g., SAO-small), improvements in reward come at the cost of distributional drift—a well-known reward exploitation phenomenon.

Mode 3: Expert-Iteration Post-Training

Here, the backbone is fine-tuned solely on its self-selected high-reward outputs according to TuneJury. A sweep over fine-tune learning rates traces out a clear reward-vs-fidelity Pareto frontier: higher reward gains are achieved at the expense of increased divergence from a reference set (MAD rises), while text-audio alignment remains stable.

This observation is a signature of reward-exploitation, evidencing that TuneJury genuinely encodes non-trivial musical preference information.

OOD Adaptation: Anchor Calibration

Once a reward model is fixed, newly-released TTM systems may exhibit global scale biases in their TuneJury scores due to covariate shift. TuneJury introduces anchor calibration: a per-system Bradley-Terry offset fitted post-hoc, requiring orders-of-magnitude less calibration data than retraining. This achieves agreement with full retraining on post-cutoff systems with approximately $1024$2 times less labeled data.

Figure 2: OOD recovery on the post-cutoff Music Arena slice, demonstrating rapid adaptation via per-system anchor calibration.

Reward Signal Probes and Release

The authors provide extensive adversarial, boundary, and population-level probes demonstrating that TuneJury robustly distinguishes musical audio from noise, tracks human rater scale, and generalizes across styles, lengths, and even real recordings (weak vocal signal notwithstanding).

Figure 3: Per-system TuneJury reward vs. win rate on bench-clean held-out splits; high rank correlation with actual human preference.

Figure 4: Three probes examining the reward signal's structure: lyrics proxy gap, system-level singing MOS correlation, and popularity gradient in FMA-Large.

The publicly released artifacts include pre-computed reward scores on open-license music collections, implementation code, and documentation, all under CC-BY-NC 4.0.

Figure 5: TuneJury reward distribution across seven large open music datasets, demonstrating musicality gradient tracking and dataset-level effects.

Theoretical and Practical Implications

TuneJury demonstrates that a small, purely human-supervised, instance-level music reward model is sufficient to:

• Achieve high calibration and accuracy on music preference alignment tasks without relying on pseudo-label augmentation.
• Drive consistent improvements across multiple downstream alignment tasks without additional labels.
• Provide a robust, open, and extensible signal for both benchmarking and practical system development.
• Enable fast OOD adaptation via lightweight anchor calibration rather than expensive retraining.

At the practical level, developers of generative music systems can reliably use TuneJury for candidate selection ("best-of-N"), offline or online reward-driven optimization, and self-supervised post-training. The method's extensibility is evident in its encoder agnosticism and open design, making it suitable for future extension to vocal-music, multi-language, or mixed-modal scoring.

Future Directions

The paper identifies possible improvements and research directions, including:

• Integrating mixed instance-level and pairwise supervision (multi-objective heads).
• Explicitly modeling and separating musicality and alignment axes.
• Expanding coverage and calibration for vocal and real-music domains.
• Scaling reward-driven training to larger models and more diverse domains.
• Combining with policy-gradient RL approaches for online reward maximization.

Conclusion

TuneJury stands out as an open, robust instance-level reward model for TTM, trained only on human preferences, and providing strong preference alignment, calibration, and extensibility without reliance on pseudo-labeling or closed data. It defines a compelling blueprint for both benchmarking and practical system alignment in generative music modeling. All accompanying models, code, and data are released under open licenses, ensuring high reproducibility and ease of integration for the community.