Improving Text-to-Music Generation with Human Preference Rewards

Abstract: We describe our entry to the efficiency track of the Academic Text-to-Music (ATTM) Grand Challenge at ICME 2026. Beyond the challenge protocol's FAD-CLAP and CLAP score, we add a learned human-preference reward from TuneJury, a twin pairwise ranker trained over open music-preference datasets. The reward serves both as a training-time conditioning signal and as a sample-selection criterion. The pipeline combines five engineering decisions on a 120M-parameter FluxAudio-S backbone, four at training time and one at inference: (i) training-time reward conditioning that doubles as an inference-time CFG axis, (ii) a sweep over five score-conditioning architectures, where training and inference use different variants, (iii) expert iteration on the top decile, (iv) a short preference-tuning pass (CRPO) for audio-text alignment, and (v) inference post-processing via joint CFG, source separation, and loudness normalization. Per-stage decomposition on 100 Song Describer prompts shows training-time reward conditioning as a functional conditioning axis, expert iteration as the dominant contributor, the preference-tuning pass adding only noise-level gain, and the inference-time score scalar already saturated by the end of the chain.

• The paper introduces a reward-conditioned training pipeline that effectively improves alignment of generated music with human aesthetic preferences.
• The pipeline leverages score-conditioning, expert iteration, and CRPO to boost FAD-CLAP and CLAP metrics using minimal architectural changes.
• Practical results show enhanced instrumentality and reduced vocal artifacts, supporting scalable and efficient text-to-music generation.

Improving Text-to-Music Generation with Human Preference Rewards

Problem Formulation and Motivation

The paper "Improving Text-to-Music Generation with Human Preference Rewards" (2606.21670) addresses text-to-music generation within the constraints of the ICME Academic Text-to-Music (ATTM) Grand Challenge, specifically the efficiency track ($\leq 500$M parameters). The central issue tackled is the alignment of model outputs with human preferences, leveraging objective metrics like FAD-CLAP and CLAP score, while introducing an explicit learned human-preference reward derived from open music-preference datasets. The approach prioritizes not only audio-text semantic alignment but also the aesthetic quality as evaluated by a pairwise human ranker, TuneJury.

Methodological Innovations

The pipeline utilizes a 120M parameter FluxAudio-S flow-matching transformer backbone (Fei et al., 2024, Li et al., 8 Aug 2025), enhanced with several nontrivial engineering and learning decisions:

1. Training-Time Reward Conditioning: Each training sample is annotated with a scalar score from the TuneJury ranker (Kim et al., 15 Jun 2026), which is Fourier-embedded and fed as a side input. This enables model steerability via score-conditioning, analogous to classifier-free guidance (CFG) (Ho et al., 2022).
2. Score-Conditioning Head Architecture Diversity: Five injection strategies for conditioning are compared; InputAdd (v2) dominates in FAD-CLAP and CLAP metrics. However, GlobalAdaLN (v1) forward offers greater stability and is used for initial stages before cross-loading weights into v2 at the final fine-tuning stage.
3. Expert Iteration: A self-improvement cycle samples outputs from the model’s SFT checkpoint, filters by a blend of TuneJury reward and CLAP-text similarity, and fine-tunes on the top decile. This yields the most substantial metric improvements.
4. CRPO Preference Optimization: A brief DPO-style (Gulcehre et al., 2023) fine-tuning pass using CLAP-aligned winner/loser pairs further enhances text-audio alignment, albeit with marginal gains.
5. Inference-Time Postprocessing: Joint CFG on text and reward, plus source separation (Demucs (Défossez et al., 2019)) and loudness normalization, optimize output quality and metric performance.

Extensive ablation reveals that the reward-conditioning axis is effective during training but saturates in inference, with subsequent fine-tunes absorbing most steerable margin.

Experimental Results and Analysis

The pipeline is validated on SDD-100 prompts against SDD-706 instrumentals using FAD-CLAP and CLAP score protocols [kilgour2019fad, wu2023clap]. Key numerical results include:

• Reward Conditioning Impact: At SFT stage, score conditioning improves FAD-CLAP by 0.025–0.040 absolute.
• Expert Iteration Effectiveness: Yields a dominant shift, with ΔFAD-CLAP = -0.0362 and Reward +0.496.
• CRPO Contribution: Marginal gain at noise level, ΔFAD-CLAP = -0.003.
• Inference Score Saturation: Post-chain, the inference-time score scalar does not materially shift reward or quality metrics, indicating its influence is largely subsumed into model weights.

Mechanism transfer analysis shows asymmetric stability: v1 (GlobalAdaLN) → v2 (InputAdd) cross-loading is benign, reverse direction collapses metrics, thereby justifying the hybrid approach.

Theoretical and Practical Implications

The results establish that reward conditioning, guided by explicit human preference signals, is a functional axis during the training phase but becomes saturated post expert iteration and CRPO tuning. The pipeline achieves high compositionality and generalization without architecturally modifying the backbone outside of the score-conditioning head. The integration of open music-preference datasets for reward annotation demonstrates scalable preference modeling and offers principled guidance for self-improvement, especially when combined with semantic scoring.

Practically, the approach attains superior instrumentality and reduced vocal artifacts suitable for downstream applications and industry benchmarks. The method’s reliance on efficient architectures ($\sim$120M parameters) and compute ($\sim$40 GPU-hours) makes it feasible for real-world deployment in creative and assistive music generation contexts.

Speculative Outlook

The findings prompt several directions for future work:

• Score-Response Curve Analysis: Formal characterization of the reward scalar’s extrapolation at inference and its limits.
• Backbone Generalization: Replication across alternative architectures to assess transferability and robustness of reward-conditioning.
• Preference Data Expansion: Incorporation of richer, hierarchical, or cross-domain preference datasets can refine aesthetic modeling and potentially unlock stronger alignment.
• Human-Agent Co-design: Continued AI-driven research workflows, leveraging advanced LLM agents (e.g., Claude Opus), may expedite methodological innovation and reproducibility in creative AI research.

Conclusion

The submission demonstrates a pipeline for text-to-music generation that leverages learned human-preference rewards as both conditioning signals and sample selection criteria. The strongest performance gains derive from reward-guided expert iteration, with further gains from CRPO being marginal. Reward conditioning is a viable and effective training-time steerable axis, but its influence at inference is largely absorbed by upstream optimization. These results have direct applicability to contemporary generative audio systems, supporting the integration of explicit human preference modeling for improved aesthetic and semantic alignment in automated music creation.