---
title: CMI Reward Models for Music Generation
url: https://www.emergentmind.com/topics/cmi-reward-models-cmi-rms
type: topic
---

# CMI Reward Models for Music Generation

CMI Reward Models (CMI-RMs) are a family of music reward models developed for evaluation and reranking under **Compositional Multimodal Instruction (CMI)**, a setting in which music generation may be conditioned on any combination of text descriptions, lyrics, and reference audio. The formulation requires simultaneous assessment of **musicality** and **prompt alignment** across optional and heterogeneous conditioning modalities, rather than treating text-to-music or audio-to-audio alignment in isolation. Within the same ecosystem, the associated work introduces the datasets **CMI-Pref-Pseudo** and **CMI-Pref**, and the benchmark **CMI-RewardBench**, positioning CMI-RMs as both a modeling family and a unifying evaluation interface for aligned music generation [2603.00610].

## 1. Compositional multimodal instruction as the target setting

Compositional Multimodal Instruction refers to a user-provided specification for music generation that may include any combination of **text descriptions** such as mood, genre, instrumentation, and structure, **lyrics** to set phrasing and semantic constraints, and **reference audio** for style transfer or continuation anchors [2603.00610]. Under this formulation, generated music must satisfy aesthetic quality, termed **musicality**, while also following all provided instruction modalities precisely.

This setting differs from prior evaluation settings that treat text-to-music or audio-to-audio alignment in isolation. The defining requirement is a **single reward model** that can adaptively assess a generated clip against optional and heterogeneous conditioning. A common misunderstanding is to reduce the problem to text-music alignment alone; the CMI formulation explicitly includes lyric-conditioned and audio-conditioned evaluation, as well as full multimodal conditioning, and therefore broadens the alignment problem beyond unimodal prompting [2603.00610].

The broader significance of the CMI formulation lies in its attempt to align evaluation with the operating conditions of contemporary music generation systems. The source work states that music generation models have evolved to handle complex multimodal inputs mixing text, lyrics, and reference audio, whereas evaluation mechanisms have lagged behind. CMI-RMs are presented as a response to that discrepancy by defining a reward-modeling regime in which conditioning heterogeneity is part of the primary design constraint rather than an afterthought [2603.00610].

## 2. Architecture and scoring functions

CMI-RMs adopt a **two-tower, parameter-efficient design** with approximately **30 M trainable parameters**, built atop **frozen multimodal encoders from MuQ-MuLan** [2603.00610]. The architecture separates prompt-side encoding from generated-audio encoding and then combines them in a lightweight interaction module.

The **prompt encoder** receives the text description $t$, lyrics $l$, and reference audio $a_{\mathrm{ref}}$. These are encoded as
$$
E_t = \mathrm{TextEnc}(t) \in \mathbb{R}^{d \times T_t}, \qquad
E_l = \mathrm{TextEnc}(l) \in \mathbb{R}^{d \times T_l}, \qquad
E_{a_{\mathrm{ref}}} = \mathrm{AudioEnc}(a_{\mathrm{ref}}) \in \mathbb{R}^{d \times L_{\mathrm{ref}}}.
$$
The prompt representations are concatenated and fused by a **4-layer Transformer**:
$$
h_{\mathrm{prompt}} = \mathrm{PromptTF}([E_t; E_l; E_{a_{\mathrm{ref}}}]).
$$

The **evaluation encoder** processes the generated audio $a_{\mathrm{eval}}$:
$$
E_{a_{\mathrm{eval}}} = \mathrm{AudioEnc}(a_{\mathrm{eval}}) \in \mathbb{R}^{d \times L_{\mathrm{eval}}}.
$$
Prompt and evaluation embeddings are then concatenated and passed through a **1-layer Joint Transformer**:
$$
h_{\mathrm{joint}} = \mathrm{JointTF}([h_{\mathrm{prompt}}; E_{a_{\mathrm{eval}}}]).
$$
After temporal pooling, an MLP head projects the joint representation to two scalar outputs:
$$
(s_{\mathrm{MUS}}, s_{\mathrm{ALI}}) = \mathrm{MLP}(\mathrm{pool}(h_{\mathrm{joint}})),
$$
where $s_{\mathrm{MUS}} \in \mathbb{R}$ assesses musicality and $s_{\mathrm{ALI}} \in \mathbb{R}$ assesses prompt alignment [2603.00610].

This design encodes the central modeling claim of CMI-RMs: optional conditioning modalities can be handled within one reward model without fully retraining the underlying multimodal encoders. A plausible implication is that the frozen-encoder strategy trades representational adaptability for training efficiency and modularity, which is consistent with the paper’s later discussion of parameter efficiency and its stated limitation that reliance on frozen MuQ encoders may restrict domain-specific feature learning [2603.00610].

## 3. Training data and optimization pipeline

The training pipeline is organized around two datasets with distinct roles. **CMI-Pref-Pseudo** is a large-scale preference dataset of approximately **110 k pairs**, while **CMI-Pref** is a human-annotated corpus of **4,027 pairs** with a **500-pair test** split [2603.00610].

CMI-Pref-Pseudo is constructed from generations produced by **12 open-source models + 11 commercial APIs**, covering **text-only, lyrics-only, audio-only, and full CMI conditions**. Pseudo-labeling is performed by **Qwen3-Omni LLM** under a **position-consistency protocol**: for each pair $(A,B)$, the system queries $(A,B)$ and $(B,A)$ and retains the pair only if the LLM prefers the same clip in both orders. The final result is **110 k consistent preference pairs** [2603.00610].

CMI-Pref is annotated by **31 music experts** using a **Music-Arena-style platform**. For each pair, annotators provide a binary preference on musicality, framed as “which sounds more like a finished, natural piece?”, and a binary preference on instruction following, framed as “which better satisfies the CMI?”, along with a **confidence score (1–5)** and **free-text rationale**. The dataset is balanced across **text-only, lyrics, audio, and full CMI** conditions [2603.00610].

Training jointly optimizes the musicality and alignment heads with
$$
L_{\mathrm{total}} = 0.5\,L_{\mathrm{MUS}} + 0.5\,L_{\mathrm{ALI}}.
$$
**Stage 1** is preference pre-training on CMI-Pref-Pseudo using a pairwise **Bradley–Terry likelihood**
$$
P(A \succ B) = \sigma(s_\theta(P,A) - s_\theta(P,B)),
$$
trained with cross-entropy. This stage uses **label smoothing $\epsilon=0.2$**, **batch size 48**, and **2 k steps** [2603.00610].

**Stage 2** performs expert fine-tuning on **CMI-Pref + MusicEval**. Pairwise preferences again use the Bradley–Terry loss, while scalar MOS ratings $y \in [1,5]$ use the regression objective
$$
L_{\mathrm{reg}} = \mathrm{MSE}(2 \tanh(a s + b) + 3, y),
$$
with $(a,b)$ initialized to $(0.2,0)$. The procedure applies **early-stopping after approximately 250 steps on validation** and uses **AdamW with standard hyperparameters** [2603.00610].

The dataset design supports a division of labor between scale and annotation fidelity. The pseudo-labeled corpus provides breadth across generation conditions, whereas the expert corpus provides fine-grained alignment supervision. The paper’s own ablation results later state that **CMI-Pref data drives cross-benchmark gains**, while **MusicEval complements for regression**, which suggests that the two-stage pipeline is not merely cumulative but functionally specialized [2603.00610].

## 4. CMI-RewardBench and the evaluation landscape

To unify evaluation, the work introduces **CMI-RewardBench**, a benchmark for music reward models over heterogeneous samples spanning musicality, text-music alignment, and compositional instruction alignment [2603.00610]. The benchmark includes both absolute and pairwise tasks.

| Task category | Datasets / subsets | Metric family |
|---|---|---|
| Absolute Musicality | PAM (500 clips, MOS), MusicEval (413 clips, MOS) | LCC, SRCC, Kendall’s $\tau$ |
| Preference Musicality | Music Arena (1,340 pairs) | Accuracy |
| Text-Music Alignment | PAM (500 pairs), CMI-Pref subsets without audio | LCC, SRCC, Kendall’s $\tau$ or Accuracy |
| Lyrics-Music Alignment | CMI-Pref lyric-only subsets | Accuracy |
| Audio-Music Alignment | CMI-Pref audio-only subsets | Accuracy |
| Full Compositional Alignment | CMI-Pref full-CMI pairs (text+lyrics+audio) | Accuracy |

For **regression tasks**, the benchmark uses **Linear Correlation Coefficient (LCC)**, **Spearman Rank Correlation (SRCC)**, and **Kendall’s $\tau$**; the paper gives the example
$$
\mathrm{SRCC}\ \rho = \mathrm{corr}_{\mathrm{rank}}(\{\hat y_i\}, \{y_i\}).
$$
For **pairwise preference tasks**, the principal metric is **Accuracy**, defined as the fraction where the model’s top-score pick matches human choice [2603.00610].

The evaluation design is notable for spanning both **absolute quality estimation** and **comparative preference prediction**. It also partitions alignment by modality—text, lyrics, audio, and full compositional conditioning—rather than collapsing them into a single score. This matters because a model can perform well on musicality while underperforming on prompt adherence, or vice versa. CMI-RewardBench makes that separation explicit in both the architecture of the benchmark and the dual-head design of the reward model [2603.00610].

## 5. Empirical results and inference-time reranking

On **musicality correlation**, the reported results place CMI-RMs at or above prior baselines after expert fine-tuning. On **PAM (SRCC)** and **MusicEval (SRCC)**, **SongEval-RM** achieves approximately **0.6977 / 0.6949**, **CMI-RM distill only** achieves **0.4304 / 0.5117**, and **CMI-RM + CMI-Pref ft** reaches **0.6988 / 0.7315**, which the paper identifies as **state-of-the-art** [2603.00610]. On **Music Arena accuracy**, the same section reports **73.88%** for SongEval-RM, **77.80%** for **CMI-RM + CMI-Pref**, and **78.20%** for **CMI-RM + CMI+MusicEval**.

On **compositional alignment**, the benchmark excerpt reports **PAM text-music LCC/SRCC** values of **(0.5201, 0.5373)** for **Gemini3-Pro**, **(0.4418, 0.4321)** for **CMI-RM + CMI+MusicEval**, and **(0.5200, 0.5243)** for **CMI-RM + CMI-Pref** [2603.00610]. On **CMI-Pref full-CMI accuracy (text+lyrics+audio)**, **Gemini2.5-Pro** reaches **72.80%**, whereas **CMI-RM + CMI-Pref** reaches **82.40%** and **CMI-RM + CMI+MusicEval** reaches **79.20%**.

The paper’s **training-set ablation** further disaggregates the role of data sources across **PAM, MusicEval, Arena, and CMI-Pref**. The reported settings are: **Distill only** with mean **SRCC 0.3925 / 0.5117**, **Arena 64.78%**, **CMI-Pref 71.10%**; **Distill+CMI-Pref** with **0.6116 / 0.7315**, **71.12%**, **75.90%**; **Distill+MusicEval** with **0.4338 / 0.3460**, **64.78%**, **69.00%**; and **Distill+Both** with **0.5464 / 0.8266**, **72.46%**, **76.05%**. The accompanying interpretation is explicit: **CMI-Pref data drives cross-benchmark gains; MusicEval complements for regression** [2603.00610].

CMI-RMs are also used for **inference-time scaling via top-k reranking**. The protocol uses **2,183 MusicCaps eval prompts**, generates $N \in \{1,3,10\}$ candidates from **MusicGen-small** and **Stable-Audio-Open-small**, reranks candidates by the average of $(s_{\mathrm{MUS}} + s_{\mathrm{ALI}})$, and selects the best [2603.00610]. For **MusicGen**, the reported objective-metric improvements are: **MuQ-MuLan $\rho$ 0.298→0.323→0.339**, **AudioBox-CE 6.046→6.405→6.647**, and **SongEval 2.143→2.213→2.273**. For **StableAudio**, **MuQ-MuLan 0.293→0.301→0.307**, with **smaller but consistent gains**. In **human A/B testing**, best-of-$N$ reranking yields **significant preference lifts vs $N=1$ and vs random**, while gains **saturate beyond $N=3$–$10$**, indicating diminishing returns [2603.00610].

These results establish two distinct uses for the model family: offline evaluation against human judgments and online candidate selection during generation. The latter is especially important because it shows that a reward model trained for evaluative correlation can also serve as a practical reranking mechanism without modifying the generation backbone itself.

## 6. Strengths, limitations, extensions, and terminological distinctions

The source work identifies several strengths of CMI-RMs: **unified handling of optional text, lyrics, and audio prompts in a single model**; **state-of-the-art correlation with human judgments on both musicality and alignment**; **parameter efficiency** due to the approximately **30 M** trainable parameters atop frozen encoders; and support for **practical inference-time scaling** through **best-of-$N$ reranking** [2603.00610].

It also identifies explicit limitations. First, **reliance on frozen MuQ encoders may limit domain-specific feature learning**. Second, evaluation is focused on **short audio ($\leq 120$ s)**, and **long-form structure may require hierarchical or sparse attention**. Third, **pseudo-labels from LLM judges may inherit systematic biases** [2603.00610]. These limitations delimit the claims of the framework: the reported performance pertains to the benchmarked duration regime and to the representational capacity of frozen multimodal backbones.

The paper lists several potential extensions: **end-to-end fine-tuning of audio-text encoders to capture longer-range dependencies**; **incorporation of rhythm/melody-specific modules for symbolic alignment signals**; **adapting CMI-RM for interactive feedback loops during generation (online RLHF)**; and **expanding CMI conditions to include symbolic scores, chord charts, or user sketches** [2603.00610]. This suggests that CMI-RMs are intended as a foundation for broader aligned music generation research rather than as a closed benchmark-only artifact.

A separate source introduces **contrastive pointwise mutual information (CPMI)** for **process reward modeling** in chain-of-thought reasoning, and its title creates a possible acronym collision with **Compositional Multimodal Instruction** [2604.10660]. The two are unrelated research programs: the CPMI work concerns step-level supervision for reasoning trajectories, with process reward models trained from automatically generated labels, whereas CMI-RMs are music reward models for multimodal instruction following [2604.10660]. The overlap is therefore terminological rather than methodological. Recognizing that distinction prevents a common confusion between **CMI-RM** in music generation and reward models based on **contrastive mutual information** in mathematical reasoning.

Source: https://www.emergentmind.com/topics/cmi-reward-models-cmi-rms