MusFlow: Multimodal Music Generation

• MusFlow is a lightweight, end-to-end multimodal music generator that aligns images, story texts, and music captions into the audio’s CLAP embedding space.
• It employs modality-specific 4-layer MLP adapters and a Transformer-UNet flow model to reconstruct compressed Mel-spectrograms and synthesize 16 kHz waveform audio.
• The model is trained on the MMusSet corpus using conditional flow matching, achieving competitive performance across caption-to-music, image-to-music, and multimodal settings.

Searching arXiv for MusFlow and closely related papers to ground the article. MusFlow is a lightweight, end-to-end multimodal music generator that uses Conditional Flow Matching to synthesize music from images, story texts, music captions, or mixed modality subsets. Its central design is to align diverse conditional inputs into the audio’s CLAP embedding space through modality-specific MLP adapters, then reconstruct compressed Mel-spectrograms in a pretrained VAE latent space and vocode them to waveform. The model is trained on MMusSet, a multimodal corpus of quadruples comprising image, story text, music caption, and music clip, and is evaluated across image-to-music, story-to-music, caption-to-music, and multimodal generation settings (Song et al., 18 Apr 2025).

1. Problem setting and conceptual scope

MusFlow is motivated by a practical limitation of text-to-music systems: many users do not supply precise, music-specific prompts, but instead possess contextual inputs such as images or narratives and want suitable background music. The model therefore treats music generation as a multimodal-to-audio problem rather than a caption-only task. During training, the available conditional inputs are image $I$, story text $S$, caption $C$, and paired audio $M$; at inference, any subset of $\{I,S,C\}$ can be used, with random masking during training to simulate incomplete conditions (Song et al., 18 Apr 2025).

The paper defines four experimental settings: Caption-to-Music, Story-to-Music, Image-to-Music, and Multimodal fusion. This formulation is narrower than general multimedia composition pipelines with explicit temporal control, but broader than caption-conditioned music generation. It emphasizes simple multimodal conditioning and a single generation pathway rather than separate model families per modality (Song et al., 18 Apr 2025).

A defining property of MusFlow is that it does not use a heavy learned fusion stack. Instead, it aligns all modalities into a unified semantic target space and fuses them by averaging. The paper explicitly states that no cross-attention or learned fusion module beyond the MLPs and averaging is described, and it does not specify normalization of embeddings (Song et al., 18 Apr 2025).

2. Architecture and generative pipeline

The architecture is organized into three stages: pretrained feature extraction, modality alignment and fusion, and audio generation in latent space. For feature extraction, MusFlow uses CLIP-ViT-B/32 for images and story text, and CLAP for captions and audio. The paper gives the encodings as

$e_{I}=E_{CLIP}(I), \quad e_{S} = E_{CLIP}(S)$

and

$e_{C}=E_{CLAP}(C), \quad e_{M} = E_{CLAP}(M),$

with all embeddings 512-dimensional (Song et al., 18 Apr 2025).

Alignment is performed by a separate 4-layer MLP for each conditioning modality. Each MLP maps its modality embedding into the CLAP audio embedding space. The aligned embeddings are fused by simple averaging,

$e_f = \frac{1}{n}\sum_{i} MLP_i(e_i),$

where $n$ is the number of available modalities. The alignment objective is defined against the paired audio embedding $e_M$, so the fused conditional representation is explicitly trained to approach the audio-side CLAP space rather than an abstract joint latent (Song et al., 18 Apr 2025).

Generation proceeds in the latent space of a pretrained VAE from MusicLDM. Audio is first represented as a Mel-spectrogram $S$0 via STFT, then encoded as

$S$1

A Transformer-UNet backbone parameterizes the conditional vector field $S$2, transporting a Gaussian prior $S$3 to a generated latent $S$4 through ODE integration. The VAE decoder reconstructs $S$5, and a pretrained HiFi-GAN vocoder synthesizes waveform audio. The paper states that output is at 16 kHz because of the pretrained vocoder (Song et al., 18 Apr 2025).

Inference follows the same decomposition: extract CLIP/CLAP embeddings for whichever modalities are available, map them through the corresponding MLP adapters, average them into $S$6, sample $S$7, integrate the learned ODE field, decode the resulting latent to Mel-spectrogram, and vocode to waveform. Output length control is not explicitly detailed; the reported samples are 10 seconds long (Song et al., 18 Apr 2025).

3. Conditional flow matching formulation

MusFlow adopts a continuous normalizing flow view in which the latent path is defined by an ODE. The paper writes the CNF path as

$S$8

It then gives the unconditional and conditional flow-matching objectives as

$S$9

and

$C$0

The generation model specializes this framework to an FM-OT setting in the pretrained VAE latent space (Song et al., 18 Apr 2025).

The conditioning signal in MusFlow is the aligned multimodal embedding $C$1. The paper defines the generation loss in that space as

$C$2

This formulation couples a straight-line optimal-transport path with a deterministic ODE solver at inference. In the paper’s comparative framing, this design is intended to provide stable training and faster sampling than SDE-based diffusion while keeping inference lightweight by avoiding large LLMs and complex multimodal controllers (Song et al., 18 Apr 2025).

The training procedure has three phases. First, the MLP adapters are optimized with the alignment loss $C$3. Second, the adapters are frozen and the flow model is trained with the generation loss $C$4. Third, the adapters are unfrozen and a weighted joint objective $C$5 is optimized to reduce compounding errors. Random condition masking is applied in every batch, with the constraint that at least one modality is present, so the model explicitly covers all seven non-empty modality subsets (Song et al., 18 Apr 2025).

4. MMusSet and the intelligent annotation workflow

MusFlow is trained on MMusSet, a new multimodal dataset in which each sample is a quadruple of image, story text, music caption, and music clip. The corpus is produced through an intelligent multi-agent workflow that combines BLIP, CLIP, a fine-tuned Qwen2-VL-7B, and CLAP. In sequence, a Script Writing Agent based on BLIP generates story text from each image; a Script Review Agent filters image-story pairs by CLIP score with a threshold of 0.3; a Music Composition Agent based on Qwen2-VL-7B, fine-tuned with QLoRA, produces music captions conditioned on image and story; and a Music Filtering Agent selects the best-matching audio clip by CLAP score, requiring similarity above 0.3 (Song et al., 18 Apr 2025).

The pretraining corpus for the captioning stage contains 10k image-story-caption triples from Music2Image, extended-MusicCaps, MelBench, and MUImage after BLIP+CLIP processing. After fine-tuning, the workflow is used to construct the ArtMusic subset from ArtGraph images and FMA-Medium audio. The full MMusSet totals 33.31k samples and 92.49 hours (Song et al., 18 Apr 2025).

Subset	Size and duration
Music2Image	1.50k, 4.17 h
MusicCaps	4.54k, 12.60 h
MelBench	0.98k, 2.69 h
MUImage	3.83k, 10.64 h
ArtMusic	22.46k, 62.39 h
MMusSet total	33.31k, 92.49 h

For experiments, the training split uses extended-MusicCaps, MelBench, MUImage, and 90% of ArtMusic, while evaluation uses Music2Image and 10% of ArtMusic. The paper does not detail genres, instruments, languages, or licensing across the combined sources (Song et al., 18 Apr 2025).

5. Training protocol, ablations, and empirical performance

MusFlow uses CLIP-ViT-B/32, a music-specific CLAP checkpoint, the VAE from MusicLDM for Mel compression and decompression, and HiFi-GAN for waveform synthesis. The modality adapters are 4-layer MLPs, and the flow backbone is a Transformer-UNet largely following MusicLDM’s parameterization. Exact latent dimensions $C$6 and Mel hop or window parameters are not specified (Song et al., 18 Apr 2025).

Optimization uses AdamW with learning rate $C$7 and batch size 24 on a single NVIDIA RTX 4090. Training is staged as 50 epochs for alignment, 250 epochs for generation, and 100 epochs for joint optimization. Qwen2-VL fine-tuning uses QLoRA rank 8 and runs for 20 epochs; an ablation also includes rank 4. These implementation choices underscore the paper’s characterization of MusFlow as lightweight at inference, since the final model uses simple adapters and a single flow model rather than an inference-time LLM (Song et al., 18 Apr 2025).

Quantitatively, the paper reports the following highlights from Table 1. In Caption-to-Music, MusFlow achieves FAD 3.41, KL 2.56, and CLAP 0.32, with best FAD/KL and CLAP comparable to MusicLDM at 0.34. In Story-to-Music, it achieves FAD 3.56, KL 2.78, and CLAP 0.31, described as best across all. In Image-to-Music, it reaches FAD 3.66, KL 2.79, and IB 0.82, outperforming CoDi at IB 0.60 and $C$8UGen at IB 0.71. In the Multimodal setting, it achieves FAD 3.36, CLAP 0.33, and IB 0.83, outperforming CoDi and $C$9UGen (Song et al., 18 Apr 2025).

Subjective evaluation is conducted with 20 participants over 50 samples per model, using Overall Quality (OVL) and Relevance to Input Condition (REL). MusFlow obtains OVL 78.25 and REL 74.12, the highest among the reported baselines. The ablation on alignment shows that conditioning directly on the target audio CLAP embedding yields the strongest guidance, while raw CLIP or CLAP features are weaker; the MLP alignment plus averaging fusion substantially improves semantic alignment toward the audio-CLAP upper bound. A second ablation shows that fine-tuning Qwen2-VL for image+story to music-caption generation improves CLAP_score from 0.15 to 0.24 and BART_score from 0.50 to 0.63 (Song et al., 18 Apr 2025).

6. Comparative position, limitations, and naming

Within the paper’s own comparison framework, MusFlow is positioned against hierarchical Transformers, latent diffusion systems, any-to-any multimodal generators, and LLM-mediated multimodal-to-music models. Its claim is not that multimodal conditioning requires a complex reasoning stack, but that alignment into the CLAP audio space with MLP adapters and average fusion is sufficient to obtain competitive conditioning quality. The paper further argues that FM-OT provides straight-line optimal-transport paths with constant-speed transport and deterministic ODE integration, enabling stable training and fast sampling, though exact runtime and complexity figures are not reported (Song et al., 18 Apr 2025).

The paper lists several limitations. Audio sampling remains at 16 kHz because of the pretrained HiFi-GAN. Video modality is not included. QLoRA rank is limited to $M$0 by GPU memory. Dependence on CLAP embeddings may inherit CLAP’s biases, and genre coverage, long-form musical structure, and fine-grained controllability are not deeply addressed. The paper also notes that copyright and permissions across the combined datasets are not detailed and require care for redistribution and usage (Song et al., 18 Apr 2025).

The name “MusFlow” also requires disambiguation. A distinct paper, “MusicFlow: Cascaded Flow Matching for Text Guided Music Generation,” uses a two-stage text-to-semantic and semantics-to-acoustics cascade over HuBERT and Encodec latents, targeting text-guided generation, zero-shot infilling, and continuation rather than multimodal CLAP-space alignment (Prajwal et al., 2024). The label is unrelated as well to “MSFlow,” an unsupervised anomaly-detection framework (Zhou et al., 2023), and to “$M$1Flow,” a one-class deepfake detector (Pontorno et al., 29 Jun 2026). In current arXiv usage, “MusFlow” most precisely denotes the multimodal music-generation model introduced in “MusFlow: Multimodal Music Generation via Conditional Flow Matching” (Song et al., 18 Apr 2025).