CMI-RewardBench: Multimodal Music Reward Benchmark

• CMI-RewardBench is a unified benchmark that evaluates music reward models under compositional multimodal instruction using heterogeneous inputs.
• It benchmarks reward models across three axes: musicality, text-music alignment, and compositional instruction alignment for music generation.
• The ecosystem integrates two datasets, CMI-Pref-Pseudo and CMI-Pref, along with a parameter-efficient reward model family (CMI-RM) for scalable evaluation.

Searching arXiv for the named benchmark and closely related RewardBench work to ground the article in current papers. CMI-RewardBench is a benchmark for music reward modeling under Compositional Multimodal Instruction (CMI), introduced together with the datasets CMI-Pref-Pseudo, CMI-Pref, and the CMI-RM model family in “CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction” (Ma et al., 28 Feb 2026). It is designed for settings in which music generation is conditioned on heterogeneous inputs that may include text descriptions, lyrics, and reference audio, and it addresses the evaluation gap between increasingly capable multimodal music generators and comparatively underdeveloped reward modeling infrastructure. Within this ecosystem, CMI-RewardBench serves as the unified benchmark for heterogeneous samples across musicality, text-music alignment, and compositional instruction alignment, while CMI-RM is presented as a parameter-efficient reward model family for scoring such inputs (Ma et al., 28 Feb 2026).

1. Definition and conceptual scope

CMI-RewardBench is presented as part of a “comprehensive ecosystem for music reward modeling under Compositional Multimodal Instruction (CMI)” (Ma et al., 28 Feb 2026). The benchmark is intended for evaluating music reward models rather than music generators directly. Its target setting is one in which the generated music may be conditioned on text descriptions, lyrics, and audio prompts, and where evaluation must therefore account for multimodal and compositional conditioning rather than only single-modal prompt following (Ma et al., 28 Feb 2026).

The benchmark is described as unified, and its unifying role is explicitly tied to the need to evaluate reward models on heterogeneous samples across three axes: musicality, text-music alignment, and compositional instruction alignment (Ma et al., 28 Feb 2026). This suggests that CMI-RewardBench is not organized around a single scalar notion of “better music,” but around multiple dimensions of response quality that are specific to multimodal music generation. A plausible implication is that the benchmark is intended to assess whether a reward model can recover human preference structure in settings where conditioning information is itself heterogeneous and partially compositional.

The benchmark’s emphasis on compositional multimodal instruction distinguishes it from broader music instruction-following evaluation such as CMI-Bench, which reinterprets MIR annotations as instruction-following tasks and evaluates audio-text LLMs across music information retrieval problems using task-specific MIR metrics (Ma et al., 14 Jun 2025). CMI-RewardBench instead centers reward modeling and preference evaluation for music generation conditioned on multimodal instructions (Ma et al., 28 Feb 2026).

2. Ecosystem components and benchmark role

The paper introduces four tightly coupled resources: CMI-Pref-Pseudo, CMI-Pref, CMI-RewardBench, and CMI-RM (Ma et al., 28 Feb 2026). Their roles are differentiated but mutually supporting.

CMI-Pref-Pseudo is a “large-scale preference dataset comprising 110k pseudo-labeled samples” (Ma et al., 28 Feb 2026). CMI-Pref is a “high-quality, human-annotated corpus tailored for fine-grained alignment tasks” (Ma et al., 28 Feb 2026). CMI-RewardBench is then introduced “to unify the evaluation landscape,” and CMI-RM is the corresponding reward-model family developed using these resources (Ma et al., 28 Feb 2026).

The benchmark therefore functions as the evaluation layer in a broader reward-modeling pipeline. The paper states that CMI-RM is evaluated for its “correlation with human judgments scores on musicality and alignment on CMI-Pref along with previous datasets,” and that further experiments show CMI-RM “enables effective inference-time scaling via top-k filtering” (Ma et al., 28 Feb 2026). This places CMI-RewardBench in a role analogous to RewardBench-style evaluation in language modeling, where the benchmark is used to compare reward models or judges against human preference labels (Lambert et al., 2024).

The following table summarizes the ecosystem components exactly at the level stated in the data.

Component	Description
CMI-Pref-Pseudo	110k pseudo-labeled samples
CMI-Pref	Human-annotated corpus for fine-grained alignment
CMI-RewardBench	Unified benchmark across musicality, text-music alignment, and compositional instruction alignment
CMI-RM	Parameter-efficient reward model family for heterogeneous inputs

This structure suggests a standard separation between training resources, high-quality human supervision, benchmark evaluation, and deployable reward models, but only the roles listed above are explicitly specified in the source text (Ma et al., 28 Feb 2026).

3. Evaluation targets: musicality and alignment

CMI-RewardBench evaluates music reward models on “heterogeneous samples across musicality, text-music alignment, and compositional instruction alignment” (Ma et al., 28 Feb 2026). These three axes are the benchmark’s central evaluative categories.

Musicality refers to human judgment of the generated music as music. The paper later states that CMI-RM is evaluated for correlation “with human judgments scores on musicality and alignment on CMI-Pref” (Ma et al., 28 Feb 2026). Although the source text does not provide a formal decomposition of musicality, its treatment as a separate judgment axis indicates that reward quality is not reducible to prompt adherence alone.

Text-music alignment concerns whether music matches textual conditioning. This is a narrower alignment axis than full compositional instruction following and corresponds to text-conditioned music generation scenarios (Ma et al., 28 Feb 2026).

Compositional instruction alignment is the benchmark’s most distinctive category. Because CMI covers mixed conditioning signals including text, lyrics, and reference audio, this alignment dimension concerns adherence to the full multimodal instruction composition rather than to any single conditioning source (Ma et al., 28 Feb 2026). This suggests that CMI-RewardBench targets cases where evaluation must account for interactions among instruction components, not merely their independent satisfaction.

A plausible implication is that the benchmark is designed for failure modes in which a generated sample is musically plausible but violates one part of the compositional instruction, or follows the text while ignoring lyrics or reference-audio constraints. The paper does not enumerate such failure modes directly in the provided text, so this remains an inference.

4. Data foundations: pseudo-labeled and human-annotated preferences

The benchmark is situated on top of two preference datasets with distinct annotation regimes (Ma et al., 28 Feb 2026).

The first, CMI-Pref-Pseudo, contains 110k pseudo-labeled samples (Ma et al., 28 Feb 2026). The naming and description indicate that it is intended for large-scale supervision where fully manual annotation would be expensive. The source text does not specify the pseudo-labeling mechanism, label schema, or pairwise/listwise format.

The second, CMI-Pref, is described as a “high-quality, human-annotated corpus tailored for fine-grained alignment tasks” (Ma et al., 28 Feb 2026). The paper further states that CMI-RM is evaluated for correlation with “human judgments scores on musicality and alignment on CMI-Pref” (Ma et al., 28 Feb 2026). This makes CMI-Pref the explicit human-grounded reference set in the ecosystem. It also indicates that at least part of the evaluation landscape is score-based rather than only binary preference-based, since the wording refers to “human judgments scores.”

This dual-dataset arrangement parallels broader reward-modeling practice in which larger synthetic or pseudo-labeled corpora are used for scale, while smaller human-annotated datasets anchor evaluation or fine-grained alignment. Related reward-benchmark work uses analogous separations between large preference collections and benchmark-grade human annotation, though in different domains and modalities (Jin et al., 27 Oct 2025, Ding et al., 29 Aug 2025, Hu et al., 18 Dec 2025).

5. CMI-RM and benchmark-driven reward modeling

The paper introduces CMI reward models (CMI-RMs) as “a parameter-efficient reward model family capable of processing heterogeneous inputs” (Ma et al., 28 Feb 2026). Their intended input regime matches the benchmark’s CMI setting, namely conditioning that may combine text descriptions, lyrics, and reference audio (Ma et al., 28 Feb 2026).

The explicit benchmark-linked claims about CMI-RM are threefold. First, the models are evaluated “on CMI-Pref along with previous datasets” for correlation with human judgments on musicality and alignment (Ma et al., 28 Feb 2026). Second, they “correlate strongly with human judgments” (Ma et al., 28 Feb 2026). Third, they “enable effective inference-time scaling via top-k filtering” (Ma et al., 28 Feb 2026).

These claims place CMI-RM within a standard reward-model deployment pattern in which a scalar reward is used to rank or filter multiple generated candidates. The paper does not print an explicit reward function, Bradley–Terry loss, or inference-time selection formula in the provided excerpt. A plausible implication is that the reward model is used as a reranker over candidate music generations, but the benchmark-specific scoring procedure is not given in the supplied text.

The link to RewardBench-style methodology is nevertheless clear. RewardBench was introduced for language-model reward evaluation through pairwise prompt-chosen-rejected judgments (Lambert et al., 2024). Later benchmarks extended this paradigm to multimodal, medical, omni-modal, long-context, and long-form settings (Ding et al., 29 Aug 2025, Jin et al., 27 Oct 2025, Hu et al., 18 Dec 2025, Tang et al., 8 Oct 2025, Huang et al., 13 Mar 2026). CMI-RewardBench specializes the same general benchmark logic to music reward models under compositional multimodal conditioning (Ma et al., 28 Feb 2026).

6. Relation to adjacent benchmark traditions

CMI-RewardBench sits at the intersection of three benchmark traditions visible in the cited literature: RewardBench-style reward-model evaluation, music instruction-following evaluation, and multimodal reward benchmarking.

From the RewardBench lineage, the most direct inheritance is the idea that reward models should be benchmarked as evaluators rather than assessed only indirectly through downstream RLHF outcomes (Lambert et al., 2024). RewardBench evaluated prompt-chosen-rejected trios spanning chat, reasoning, and safety (Lambert et al., 2024). RewardBench 2 later increased difficulty, used unseen human prompts, and moved to best-of-4 evaluation across multiple domains (Malik et al., 2 Jun 2025). CMI-RewardBench adopts the same broad evaluative stance, but in a music-generation setting where outputs are conditioned on multimodal instructions (Ma et al., 28 Feb 2026).

From the music-benchmark side, CMI-Bench is the closest adjacent work in name and domain, but its focus is different. CMI-Bench is “a comprehensive music instruction following benchmark” for audio-text LLMs across MIR tasks such as genre classification, emotion regression, key detection, lyrics transcription, melody extraction, and beat tracking (Ma et al., 14 Jun 2025). It does not define reward-model or preference-learning tasks directly, but it provides prompt templates, structured output formats, task-native metrics, and fine-grained music-evaluation categories (Ma et al., 14 Jun 2025). A plausible implication is that CMI-Bench offers task taxonomy and evaluation tooling that could inform or complement music reward-model benchmarking, but CMI-RewardBench is the paper that explicitly frames the problem as one of reward modeling (Ma et al., 28 Feb 2026).

From the multimodal reward-benchmark side, several works are especially relevant. Omni-RewardBench extends reward modeling to text, image, video, audio, and 3D under free-form criteria (Jin et al., 27 Oct 2025). Multimodal RewardBench 2 evaluates omni reward models for interleaved text and image outputs across text-to-image, editing, interleaved generation, and multimodal reasoning (Hu et al., 18 Dec 2025). Med-RewardBench specializes reward-model and judge evaluation to multimodal medical scenarios (Ding et al., 29 Aug 2025). Relative to these, CMI-RewardBench appears to be domain-specialized rather than modality-general: its distinctive contribution is evaluation for music generation conditioned by compositional multimodal instruction (Ma et al., 28 Feb 2026).

7. Public release and stated contribution

The paper states that “the necessary training data, benchmarks, and reward models are publicly available” (Ma et al., 28 Feb 2026). This includes the training data, the benchmark, and the CMI-RM models themselves. No repository URL, license, or versioning detail is provided in the supplied text.

The paper’s contribution is therefore not only a benchmark name, but a full benchmark-centered ecosystem: a pseudo-labeled dataset, a human-annotated dataset, a unified reward benchmark, and a reward-model family, all intended to close what the authors describe as a “critical gap” between multimodal music generation capability and evaluation capability (Ma et al., 28 Feb 2026).

A concise way to situate CMI-RewardBench is as a domain-specific RewardBench extension for music generation under compositional multimodal conditioning. Its novelty lies not in introducing reward modeling itself, but in defining the evaluation target for settings where the instruction may mix text descriptions, lyrics, and reference audio, and where reward quality must jointly reflect musicality, text-music alignment, and compositional instruction alignment (Ma et al., 28 Feb 2026). This suggests a shift from single-axis music evaluation toward reward modeling for heterogeneous instruction-grounded music generation.