CMI Reward Models for Music Generation

• CMI-RMs are a unified framework that assesses both musicality and prompt alignment for multimodal (text, lyrics, audio) music generation.
• They use a two-tower, parameter-efficient design with frozen MuQ encoders and lightweight transformers for joint evaluation.
• Training leverages large-scale pseudo and expert annotated datasets, achieving state-of-the-art correlation with human judgments.

CMI Reward Models (CMI-RMs) are a family of music reward models developed for evaluation and reranking under Compositional Multimodal Instruction (CMI), a setting in which music generation may be conditioned on any combination of text descriptions, lyrics, and reference audio. The formulation requires simultaneous assessment of musicality and prompt alignment across optional and heterogeneous conditioning modalities, rather than treating text-to-music or audio-to-audio alignment in isolation. Within the same ecosystem, the associated work introduces the datasets CMI-Pref-Pseudo and CMI-Pref, and the benchmark CMI-RewardBench, positioning CMI-RMs as both a modeling family and a unifying evaluation interface for aligned music generation (Ma et al., 28 Feb 2026).

1. Compositional multimodal instruction as the target setting

Compositional Multimodal Instruction refers to a user-provided specification for music generation that may include any combination of text descriptions such as mood, genre, instrumentation, and structure, lyrics to set phrasing and semantic constraints, and reference audio for style transfer or continuation anchors (Ma et al., 28 Feb 2026). Under this formulation, generated music must satisfy aesthetic quality, termed musicality, while also following all provided instruction modalities precisely.

This setting differs from prior evaluation settings that treat text-to-music or audio-to-audio alignment in isolation. The defining requirement is a single reward model that can adaptively assess a generated clip against optional and heterogeneous conditioning. A common misunderstanding is to reduce the problem to text-music alignment alone; the CMI formulation explicitly includes lyric-conditioned and audio-conditioned evaluation, as well as full multimodal conditioning, and therefore broadens the alignment problem beyond unimodal prompting (Ma et al., 28 Feb 2026).

The broader significance of the CMI formulation lies in its attempt to align evaluation with the operating conditions of contemporary music generation systems. The source work states that music generation models have evolved to handle complex multimodal inputs mixing text, lyrics, and reference audio, whereas evaluation mechanisms have lagged behind. CMI-RMs are presented as a response to that discrepancy by defining a reward-modeling regime in which conditioning heterogeneity is part of the primary design constraint rather than an afterthought (Ma et al., 28 Feb 2026).

2. Architecture and scoring functions

CMI-RMs adopt a two-tower, parameter-efficient design with approximately 30 M trainable parameters, built atop frozen multimodal encoders from MuQ-MuLan (Ma et al., 28 Feb 2026). The architecture separates prompt-side encoding from generated-audio encoding and then combines them in a lightweight interaction module.

The prompt encoder receives the text description $t$, lyrics $l$, and reference audio $a_{\mathrm{ref}}$. These are encoded as

$$E_t = \mathrm{TextEnc}(t) \in \mathbb{R}^{d \times T_t}, \qquad E_l = \mathrm{TextEnc}(l) \in \mathbb{R}^{d \times T_l}, \qquad E_{a_{\mathrm{ref}}} = \mathrm{AudioEnc}(a_{\mathrm{ref}}) \in \mathbb{R}^{d \times L_{\mathrm{ref}}}.$$

The prompt representations are concatenated and fused by a 4-layer Transformer:

$$h_{\mathrm{prompt}} = \mathrm{PromptTF}([E_t; E_l; E_{a_{\mathrm{ref}}}]).$$

The evaluation encoder processes the generated audio $a_{\mathrm{eval}}$:

$$E_{a_{\mathrm{eval}}} = \mathrm{AudioEnc}(a_{\mathrm{eval}}) \in \mathbb{R}^{d \times L_{\mathrm{eval}}}.$$

Prompt and evaluation embeddings are then concatenated and passed through a 1-layer Joint Transformer:

$$h_{\mathrm{joint}} = \mathrm{JointTF}([h_{\mathrm{prompt}}; E_{a_{\mathrm{eval}}}]).$$

After temporal pooling, an MLP head projects the joint representation to two scalar outputs:

$$(s_{\mathrm{MUS}}, s_{\mathrm{ALI}}) = \mathrm{MLP}(\mathrm{pool}(h_{\mathrm{joint}})),$$

where $s_{\mathrm{MUS}} \in \mathbb{R}$ assesses musicality and $l$0 assesses prompt alignment (Ma et al., 28 Feb 2026).

This design encodes the central modeling claim of CMI-RMs: optional conditioning modalities can be handled within one reward model without fully retraining the underlying multimodal encoders. A plausible implication is that the frozen-encoder strategy trades representational adaptability for training efficiency and modularity, which is consistent with the paper’s later discussion of parameter efficiency and its stated limitation that reliance on frozen MuQ encoders may restrict domain-specific feature learning (Ma et al., 28 Feb 2026).

3. Training data and optimization pipeline

The training pipeline is organized around two datasets with distinct roles. CMI-Pref-Pseudo is a large-scale preference dataset of approximately 110 k pairs, while CMI-Pref is a human-annotated corpus of 4,027 pairs with a 500-pair test split (Ma et al., 28 Feb 2026).

CMI-Pref-Pseudo is constructed from generations produced by 12 open-source models + 11 commercial APIs, covering text-only, lyrics-only, audio-only, and full CMI conditions. Pseudo-labeling is performed by Qwen3-Omni LLM under a position-consistency protocol: for each pair $l$1, the system queries $l$2 and $l$3 and retains the pair only if the LLM prefers the same clip in both orders. The final result is 110 k consistent preference pairs (Ma et al., 28 Feb 2026).

CMI-Pref is annotated by 31 music experts using a Music-Arena-style platform. For each pair, annotators provide a binary preference on musicality, framed as “which sounds more like a finished, natural piece?”, and a binary preference on instruction following, framed as “which better satisfies the CMI?”, along with a confidence score (1–5) and free-text rationale. The dataset is balanced across text-only, lyrics, audio, and full CMI conditions (Ma et al., 28 Feb 2026).

Training jointly optimizes the musicality and alignment heads with

$l$4

Stage 1 is preference pre-training on CMI-Pref-Pseudo using a pairwise Bradley–Terry likelihood

$l$5

trained with cross-entropy. This stage uses label smoothing $l$6, batch size 48, and 2 k steps (Ma et al., 28 Feb 2026).

Stage 2 performs expert fine-tuning on CMI-Pref + MusicEval. Pairwise preferences again use the Bradley–Terry loss, while scalar MOS ratings $l$7 use the regression objective

$l$8

with $l$9 initialized to $a_{\mathrm{ref}}$0. The procedure applies early-stopping after approximately 250 steps on validation and uses AdamW with standard hyperparameters (Ma et al., 28 Feb 2026).

The dataset design supports a division of labor between scale and annotation fidelity. The pseudo-labeled corpus provides breadth across generation conditions, whereas the expert corpus provides fine-grained alignment supervision. The paper’s own ablation results later state that CMI-Pref data drives cross-benchmark gains, while MusicEval complements for regression, which suggests that the two-stage pipeline is not merely cumulative but functionally specialized (Ma et al., 28 Feb 2026).

4. CMI-RewardBench and the evaluation landscape

To unify evaluation, the work introduces CMI-RewardBench, a benchmark for music reward models over heterogeneous samples spanning musicality, text-music alignment, and compositional instruction alignment (Ma et al., 28 Feb 2026). The benchmark includes both absolute and pairwise tasks.

Task category	Datasets / subsets	Metric family
Absolute Musicality	PAM (500 clips, MOS), MusicEval (413 clips, MOS)	LCC, SRCC, Kendall’s $a_{\mathrm{ref}}$1
Preference Musicality	Music Arena (1,340 pairs)	Accuracy
Text-Music Alignment	PAM (500 pairs), CMI-Pref subsets without audio	LCC, SRCC, Kendall’s $a_{\mathrm{ref}}$2 or Accuracy
Lyrics-Music Alignment	CMI-Pref lyric-only subsets	Accuracy
Audio-Music Alignment	CMI-Pref audio-only subsets	Accuracy
Full Compositional Alignment	CMI-Pref full-CMI pairs (text+lyrics+audio)	Accuracy

For regression tasks, the benchmark uses Linear Correlation Coefficient (LCC), Spearman Rank Correlation (SRCC), and Kendall’s $a_{\mathrm{ref}}$3; the paper gives the example

$a_{\mathrm{ref}}$4

For pairwise preference tasks, the principal metric is Accuracy, defined as the fraction where the model’s top-score pick matches human choice (Ma et al., 28 Feb 2026).

The evaluation design is notable for spanning both absolute quality estimation and comparative preference prediction. It also partitions alignment by modality—text, lyrics, audio, and full compositional conditioning—rather than collapsing them into a single score. This matters because a model can perform well on musicality while underperforming on prompt adherence, or vice versa. CMI-RewardBench makes that separation explicit in both the architecture of the benchmark and the dual-head design of the reward model (Ma et al., 28 Feb 2026).

5. Empirical results and inference-time reranking

On musicality correlation, the reported results place CMI-RMs at or above prior baselines after expert fine-tuning. On PAM (SRCC) and MusicEval (SRCC), SongEval-RM achieves approximately 0.6977 / 0.6949, CMI-RM distill only achieves 0.4304 / 0.5117, and CMI-RM + CMI-Pref ft reaches 0.6988 / 0.7315, which the paper identifies as state-of-the-art (Ma et al., 28 Feb 2026). On Music Arena accuracy, the same section reports 73.88% for SongEval-RM, 77.80% for CMI-RM + CMI-Pref, and 78.20% for CMI-RM + CMI+MusicEval.

On compositional alignment, the benchmark excerpt reports PAM text-music LCC/SRCC values of (0.5201, 0.5373) for Gemini3-Pro, (0.4418, 0.4321) for CMI-RM + CMI+MusicEval, and (0.5200, 0.5243) for CMI-RM + CMI-Pref (Ma et al., 28 Feb 2026). On CMI-Pref full-CMI accuracy (text+lyrics+audio), Gemini2.5-Pro reaches 72.80%, whereas CMI-RM + CMI-Pref reaches 82.40% and CMI-RM + CMI+MusicEval reaches 79.20%.

The paper’s training-set ablation further disaggregates the role of data sources across PAM, MusicEval, Arena, and CMI-Pref. The reported settings are: Distill only with mean SRCC 0.3925 / 0.5117, Arena 64.78%, CMI-Pref 71.10%; Distill+CMI-Pref with 0.6116 / 0.7315, 71.12%, 75.90%; Distill+MusicEval with 0.4338 / 0.3460, 64.78%, 69.00%; and Distill+Both with 0.5464 / 0.8266, 72.46%, 76.05%. The accompanying interpretation is explicit: CMI-Pref data drives cross-benchmark gains; MusicEval complements for regression (Ma et al., 28 Feb 2026).

CMI-RMs are also used for inference-time scaling via top-k reranking. The protocol uses 2,183 MusicCaps eval prompts, generates $a_{\mathrm{ref}}$5 candidates from MusicGen-small and Stable-Audio-Open-small, reranks candidates by the average of $a_{\mathrm{ref}}$6, and selects the best (Ma et al., 28 Feb 2026). For MusicGen, the reported objective-metric improvements are: MuQ-MuLan $a_{\mathrm{ref}}$7 0.298→0.323→0.339, AudioBox-CE 6.046→6.405→6.647, and SongEval 2.143→2.213→2.273. For StableAudio, MuQ-MuLan 0.293→0.301→0.307, with smaller but consistent gains. In human A/B testing, best-of-$a_{\mathrm{ref}}$8 reranking yields significant preference lifts vs $a_{\mathrm{ref}}$9 and vs random, while gains saturate beyond $$E_t = \mathrm{TextEnc}(t) \in \mathbb{R}^{d \times T_t}, \qquad E_l = \mathrm{TextEnc}(l) \in \mathbb{R}^{d \times T_l}, \qquad E_{a_{\mathrm{ref}}} = \mathrm{AudioEnc}(a_{\mathrm{ref}}) \in \mathbb{R}^{d \times L_{\mathrm{ref}}}.$$0–$$E_t = \mathrm{TextEnc}(t) \in \mathbb{R}^{d \times T_t}, \qquad E_l = \mathrm{TextEnc}(l) \in \mathbb{R}^{d \times T_l}, \qquad E_{a_{\mathrm{ref}}} = \mathrm{AudioEnc}(a_{\mathrm{ref}}) \in \mathbb{R}^{d \times L_{\mathrm{ref}}}.$$1, indicating diminishing returns (Ma et al., 28 Feb 2026).

These results establish two distinct uses for the model family: offline evaluation against human judgments and online candidate selection during generation. The latter is especially important because it shows that a reward model trained for evaluative correlation can also serve as a practical reranking mechanism without modifying the generation backbone itself.

6. Strengths, limitations, extensions, and terminological distinctions

The source work identifies several strengths of CMI-RMs: unified handling of optional text, lyrics, and audio prompts in a single model; state-of-the-art correlation with human judgments on both musicality and alignment; parameter efficiency due to the approximately 30 M trainable parameters atop frozen encoders; and support for practical inference-time scaling through best-of-$$E_t = \mathrm{TextEnc}(t) \in \mathbb{R}^{d \times T_t}, \qquad E_l = \mathrm{TextEnc}(l) \in \mathbb{R}^{d \times T_l}, \qquad E_{a_{\mathrm{ref}}} = \mathrm{AudioEnc}(a_{\mathrm{ref}}) \in \mathbb{R}^{d \times L_{\mathrm{ref}}}.$$2 reranking (Ma et al., 28 Feb 2026).

It also identifies explicit limitations. First, reliance on frozen MuQ encoders may limit domain-specific feature learning. Second, evaluation is focused on short audio ($$E_t = \mathrm{TextEnc}(t) \in \mathbb{R}^{d \times T_t}, \qquad E_l = \mathrm{TextEnc}(l) \in \mathbb{R}^{d \times T_l}, \qquad E_{a_{\mathrm{ref}}} = \mathrm{AudioEnc}(a_{\mathrm{ref}}) \in \mathbb{R}^{d \times L_{\mathrm{ref}}}.$$3 s), and long-form structure may require hierarchical or sparse attention. Third, pseudo-labels from LLM judges may inherit systematic biases (Ma et al., 28 Feb 2026). These limitations delimit the claims of the framework: the reported performance pertains to the benchmarked duration regime and to the representational capacity of frozen multimodal backbones.

The paper lists several potential extensions: end-to-end fine-tuning of audio-text encoders to capture longer-range dependencies; incorporation of rhythm/melody-specific modules for symbolic alignment signals; adapting CMI-RM for interactive feedback loops during generation (online RLHF); and expanding CMI conditions to include symbolic scores, chord charts, or user sketches (Ma et al., 28 Feb 2026). This suggests that CMI-RMs are intended as a foundation for broader aligned music generation research rather than as a closed benchmark-only artifact.

A separate source introduces contrastive pointwise mutual information (CPMI) for process reward modeling in chain-of-thought reasoning, and its title creates a possible acronym collision with Compositional Multimodal Instruction (Lee et al., 12 Apr 2026). The two are unrelated research programs: the CPMI work concerns step-level supervision for reasoning trajectories, with process reward models trained from automatically generated labels, whereas CMI-RMs are music reward models for multimodal instruction following (Lee et al., 12 Apr 2026). The overlap is therefore terminological rather than methodological. Recognizing that distinction prevents a common confusion between CMI-RM in music generation and reward models based on contrastive mutual information in mathematical reasoning.