CMI-Pref-Pseudo: Multimodal Reward Dataset

• CMI-Pref-Pseudo is a pseudo-labeled dataset that provides pairwise comparisons on musicality and instruction alignment to pretrain reward models for multimodal music generation.
• It comprises approximately 110k pairwise comparisons across diverse modalities, including text, lyrics, and audio, covering over 47,546 generated audio clips.
• The dataset enables a two-stage training pipeline with pseudo pretraining using Bradley–Terry loss and expert fine-tuning on human data, enhancing performance on benchmark tasks.

to=arxiv_search.search 重庆时时彩杀കം 天天中彩票上? It's not given exact function schema. {"query":"CMI-Pref-Pseudo CMI-RewardBench arXiv", "max_results": 5} to=arxiv_search.search code to=arxiv_search.search code 大发时时彩开奖ി 彩神争霸官方{"query":"CMI-Pref-Pseudo CMI-RewardBench", "max_results": 5} to=arxiv_search code 官网群{"query":"CMI-Pref-Pseudo CMI-RewardBench", "max_results": 5} to=arxiv_search.search 北京赛车前 code {"search_query":"CMI-Pref-Pseudo CMI-RewardBench","start":0,"max_results":5} CMI-Pref-Pseudo is a large-scale pseudo-labeled preference dataset built to train reward models for music generated under Compositional Multimodal Instruction (CMI), where instructions may include any combination of text descriptions, lyrics, and reference audio. Introduced together with CMI-Pref, CMI-RewardBench, and CMI reward models (CMI-RMs), it provides the scale and breadth of multimodal conditions needed to learn robust, general reward models that can judge both musicality and alignment to complex CMI prompts, at a cost far lower than human labeling (Ma et al., 28 Feb 2026).

1. Role within the CMI reward-modeling ecosystem

CMI-Pref-Pseudo complements two other pillars in the ecosystem. CMI-Pref is a high-quality, human-annotated corpus used to fine-tune and validate models trained on CMI-Pref-Pseudo. CMI-RewardBench is a unified benchmark aggregating PAM, MusicEval, Music Arena, and the CMI-Pref test split for evaluating musicality, text–music alignment, and compositional instruction alignment. CMI-RMs are first pre-trained on CMI-Pref-Pseudo in the “Distill” stage and then fine-tuned on human data from CMI-Pref and/or MusicEval (Ma et al., 28 Feb 2026).

The dataset is explicitly designed to handle heterogeneous inputs within a single reward model. Its pseudo labels provide pairwise preferences along two dimensions: musicality, defined in the paper as aesthetic or production quality, naturalness, and structure, and instruction alignment, defined as adherence to compositional instructions including text, lyrics, and/or audio style or reference. This dual-head structure is central to the later training and evaluation pipeline.

CMI-Pref-Pseudo is not used for testing in CMI-RewardBench. Its function is to provide the large-scale pretraining signal that enables a single, unified CMI reward model to perform well on benchmark tasks spanning absolute scoring, regression, and pairwise comparison. In that division of labor, CMI-Pref-Pseudo supplies breadth and scale, while CMI-Pref supplies depth and reliability.

2. Scale, modality coverage, and data composition

The dataset contains approximately 110k pairwise preference comparisons, retained after stringent consistency checks from an initial 129–130k pool. On the candidate side, it spans 47,546 generated audio clips totaling 797.34 hours; reference audio totals 123.95 hours. There are 10,213 unique prompts and 23 total model sources, comprising 12 open-source models and 11 commercial APIs (Ma et al., 28 Feb 2026).

The modality distribution across generated candidates is reported as follows:

Modality	Share
Text-only	44.8%
Lyrics-only	19.8%
Audio-only (reference audio)	17.0%
Audio+lyrics	18.3%

In addition, 35.6% of generations include an audio prompt. If a model cannot consume audio prompts, a caption of the reference audio is provided by Qwen3-Omni as additional text condition. This design lets the dataset cover text-only, lyrics-conditioned, and audio-reference-conditioned generation within one training resource.

The core comparison schema includes instruction context, candidate audios, pseudo labels, and metadata. The instruction context comprises prompt_text, optional lyrics, and optional reference_audio or its text caption. Each comparison includes audio_A and audio_B. The pseudo labels comprise musicality_preference and alignment_preference, each in {model_a, model_b, both, neither}, together with per-audio scalar scores on both dimensions and, in some prompt variants, a short rationale. Metadata include prompt_id, model_id/model_name, modality flags, and durations.

3. Candidate generation and pseudo-labeling pipeline

Audio was generated from a diverse set of 12 open-source models and 11 APIs to cover broad quality and style ranges and modality combinations. The open-source models include MusicGen, Stable Audio Open, YUE, SongGen, AudioLDM/AudioLDM 2, DiffRhythm, Levo, Magenta Lyria-RealTime, Jamify, MusicLDM, and ACE-step. The commercial sources include multiple Suno versions, Stable Audio 2.0, Minimax-Music-2.0, Mureka, and Loudly. To control for conditioning diversity, the generation process targets equal splits of instrumental vs. vocal and equal splits with vs. without audio prompts when applicable (Ma et al., 28 Feb 2026).

Qwen3-Omni is used as the LLM-as-a-judge for pseudo labeling. The judging prompts enforce strict JSON schemas and elicit both a discrete preference and per-audio scalar scores for musicality and instruction following. The pipeline supports text-only, lyrics-conditioned, and audio-reference-conditioned cases. For audio-reference alignment, a strict prompt variant focuses on style match to the reference audio.

Position-consistency filtering is the central quality-control mechanism. Each pair (A, B) is evaluated twice: forward (A,B) and reversed (B,A). A label is retained only if the preference is invariant to position. The paper reports substantial positional bias before filtering; for musicality, “win A” was 51.96% in original order versus 59.27% in reversed order. After filtering, the authors report 114,694 valid musicality labels and 117,828 valid alignment labels, and taking their intersection yields approximately 110k pairs.

Agreement between the two pseudo-labeled dimensions is high. Pseudo labels exhibit approximately 91% agreement between musicality and instruction-alignment preferences, whereas human data show approximately 81% agreement. The paper presents this as evidence of consistency, while also identifying residual coupling between alignment and aesthetic quality as a caveat.

4. Training objectives enabled by the dataset

CMI-Pref-Pseudo is used in a two-stage training pipeline. In Stage 1, preference pre-training on CMI-Pref-Pseudo, pairwise preferences are modeled with a Bradley–Terry formulation. For a prompt $P$ and two candidates $A, B$,

$$P(A > B) = \sigma(s_\theta(P, A) - s_\theta(P, B)).$$

The model is optimized with cross-entropy on pairwise outcomes, excluding ties. The paper also states the standard pairwise ranking loss as

$L = -\log \sigma(f(x^+) - f(x^-)).$

To reduce over-confident boundaries caused by pseudo-label noise and distribution shift, label smoothing with $\epsilon = 0.2$ is applied:

$\tilde{y} = (1 - \epsilon) y + \epsilon/2.$

In Stage 2, expert fine-tuning on human data, the same Bradley–Terry loss is used for pairwise data. For scalar MOS ratings $y \in [1,5]$, the paper regresses a transformed score

$$L_{\mathrm{reg}} = \mathrm{MSE}(2 \tanh(a s + b) + 3, y),$$

with $a = 0.2$ and $b = 0$ during fine-tuning. The total multi-task loss combines musicality and alignment heads:

$A, B$0

The dataset therefore functions as the large-scale preference pretraining substrate for a parameter-efficient reward model family capable of processing heterogeneous inputs. Empty modalities are represented as zero tensors in the CMI-RM architecture; the paper identifies this as a modeling choice rather than a dataset field (Ma et al., 28 Feb 2026).

5. Empirical transfer, benchmark effects, and inference-time scaling

When trained solely on CMI-Pref-Pseudo, the Distill-only model achieves a PAM mean SRCC of 0.3925, a MusicEval SRCC of 0.5117, a Music Arena pairwise accuracy of 64.78%, and a CMI-Pref test mean accuracy of 71.10%. With additional fine-tuning on CMI-Pref, the corresponding values become 0.6116, 0.7315, 71.12%, and 75.90%. With joint fine-tuning on CMI-Pref and MusicEval, they become 0.5464, 0.8266, 72.46%, and 76.05% (Ma et al., 28 Feb 2026).

The paper also reports a label-smoothing ablation directly tied to pseudo pretraining. Before fine-tuning, performance on the real human test set is 71.0% for musicality and 71.8% for alignment. After fine-tuning with smoothing, the scores become 77.8% and 74.0%, compared with 75.0% and 69.3% for direct fine-tuning from scratch and 75.2% and 71.0% for pseudo pretraining without smoothing. In the paper’s interpretation, pseudo pretraining plus smoothing transfers better to human labels.

A pseudo-data-size ablation shows gains up to approximately 64k–110k examples, with diminishing returns beyond 64k under the same-epoch protocol. Under that protocol, Pref-Test accuracy rises from 0.582 at 4k to 0.711 at 64k and 0.720 at 110k. After fine-tuning, downstream accuracies are similar for 64k and 110k, with musicality approximately 0.778 versus 0.761 and alignment approximately 0.740 versus 0.735. The paper notes a practical rule-of-thumb that pseudo data should be more than 10 times the human data scale.

The trained reward models also enable inference-time scaling via best-of-$A, B$1 selection. For top-$A, B$2 generation experiments, the paper uses

$A, B$3

For Music Arena analyses, it uses $A, B$4 to match the overall preference signal. Human A/B testing confirms that best-of-$A, B$5 improves quality, with gains growing from $A, B$6 to $A, B$7, although the paper notes diminishing returns depending on backbone.

6. Quality control, limitations, and release conditions

The dataset’s most explicit bias-mitigation step is the Position-Consistency filter, which removes labels that flip when pair order is reversed. This directly addresses LLM positional bias and stabilizes the label distribution. The paper nevertheless identifies residual noise and distribution shift from pseudo to human data, which manifests as over-confident decision boundaries and motivates the use of label smoothing during pseudo pretraining (Ma et al., 28 Feb 2026).

Residual coupling between axes is another limitation. Pseudo labels show approximately 91% agreement between musicality and instruction alignment choices, whereas human labels show approximately 81%. The paper presents this as evidence that pseudo labels may sometimes conflate alignment and aesthetic quality more than humans do. It also notes that the release does not describe de-duplication or decontamination beyond TOS-aware release and positional-bias filtering, so residual label noise and model or style imbalances should be expected despite efforts to balance modalities.

CMI-Pref-Pseudo is released on Hugging Face at https://huggingface.co/datasets/HaiwenXia/cmi-pref-pseudo. The release is CC-BY-NC-SA. The authors adopt a TOS-aware policy under which only redistributable components are public, and any restricted parts may be available via application-based access. A datasheet documents sources and compatibility.

Within the CMI ecosystem, the dataset’s significance lies in its combination of multimodal breadth, pairwise supervision, and pseudo-labeled scale. CMI-Pref-Pseudo and CMI-Pref together are described as the first to target compositional multimodal instruction at scale. A plausible implication is that the dataset’s main contribution is not only volume, but the alignment of volume with a reward-modeling problem in which text, lyrics, and reference audio must be evaluated jointly rather than as isolated modalities.