Image-to-Music Generation: Methods & Challenges

• Image-to-Music (I2M) generation is the process of converting static images into music, either symbolically or as audio, with outputs reflecting style, mood, and semantics.
• Key methodologies include direct visual mapping, latent translation, text bridging, and retrieval-augmented techniques that balance controllability and expressiveness.
• Evaluation challenges center on scarce paired data and ambiguous alignment metrics, spurring research into structured intermediates and human-in-the-loop feedback.

Image-to-Music (I2M) generation is the task of transforming a static image into music in either symbolic form or audio form. Within the broader vision-to-music literature, it differs from video-to-music because the input lacks temporal semantics and explicit rhythm, so the generated output usually reflects overall style, mood, or semantics rather than temporally grounded events, and the music duration is not constrained by the image (Wang et al., 27 Mar 2025). Early I2M systems were primarily symbolic, whereas more recent work increasingly produces audio and often relies on pretrained multimodal encoders, text intermediates, retrieval modules, diffusion backbones, or symbolic music engines (Wang et al., 27 Mar 2025).

1. Problem scope and task variants

I2M is not a single uniform problem. The literature spans symbolic melody generation, symbolic polyphonic composition, and direct audio generation. Symbolic systems produce structured outputs such as MIDI-like note sequences, chords, or other music tokens, while audio systems produce waveform music or latent audio representations that are later decoded to waveform (Wang et al., 27 Mar 2025). This distinction is consequential because symbolic systems usually offer stronger controllability and easier incorporation of music-theoretic constraints, whereas audio systems offer richer expressiveness but typically shorter generation horizons and weaker explicit control (Wang et al., 27 Mar 2025).

A second distinction concerns how directly the image conditions music. Some systems attempt a comparatively direct image-conditioned mapping. The “artificial synesthete” uses an image VAE, a translation MLP, and MusicVAE to generate monophonic symbolic melodies from images (Wienand et al., 2021). The emotion-guided CNN-Transformer model conditions a symbolic MIDI decoder on CNN image features and adds a Valence-Arousal loss so that generated MIDI should be emotionally consistent with the input image (Kundu et al., 2024). ConchShell conditions a GAN on pseudo-temporal image features and generates Jukebox-compatible latent audio codes that are decoded into 8-second piano-only waveforms (Fan et al., 2022).

Other systems are explicitly indirect. MusicAIR is not a direct image-to-score or image-to-audio model: for I2M it first uses LLMs to convert the image into lyrics, then applies a purely algorithmic lyric-to-music symbolic core that determines rhythm, pitch, and score structure from textual cues such as syllables, keywords, sentiment, and phrase structure (Liao et al., 21 Nov 2025). The painting-oriented emotion pipeline likewise proceeds through emotion labeling, BLIP captioning, FalconRW-1B enrichment into music-oriented text, and MusicGen-small as the final text-to-music backend (Hisariya et al., 2024). The survey identifies this family more generally as a text-bridged strategy: visual features are converted into text or instructions and then passed to a text-to-music model (Wang et al., 27 Mar 2025).

A third distinction separates strict image-only generation from multimodal conditioning. MeLFusion operates as image+text-to-music rather than pure image-only generation, and its ablations show that image-only conditioning is substantially weaker than text+image conditioning in that framework (Chowdhury et al., 2024). Im2Wav is adjacent rather than central to I2M: it is an image-guided open-domain audio generation system for environmental sound rather than music, but it remains relevant because it develops a CLIP-conditioned hierarchical audio-token architecture and classifier-free guidance for visual conditioning (Sheffer et al., 2022).

2. Main architectural paradigms

The field has developed through several recurring architectural patterns. Early symbolic systems emphasized explicit visual-to-musical mappings from color, texture, or semantic content into symbolic music. The survey places Synesthesia, MuSyFI, Vis2Mus, and BGT in this early symbolic line (Wang et al., 27 Mar 2025). These systems are historically important because they treated I2M as a controllable or interpretable composition problem rather than only as high-dimensional waveform synthesis.

A later line uses latent translation. The artificial synesthete does not learn a joint image-music embedding from naturally paired data. Instead, it trains separate visual and musical VAEs and then learns MLPs that map between their latent spaces, with supervision derived from an engineered note-color correspondence in which hue is associated with chromatic pitch class and brightness with octave (Wienand et al., 2021). This produces simple 2-bar or 16-bar monophonic melodies with fixed tempo and note velocity. The method is narrow, but it established a reusable idea: separately pretrained unimodal generators can be connected through a learned cross-modal translator.

A different symbolic tradition emphasizes explicit musical planning. MusicAIR is exemplary because the compositional engine is not a trained neural generator but a rule- and algorithm-driven symbolic core. Once lyrics are available—either supplied directly or generated from an image—the core performs score setup, rhythmic score construction, pitch construction, and XML score conversion, ultimately outputting MusicXML, MIDI, sheet music, and rendered playback (Liao et al., 21 Nov 2025). The image in this system influences music only indirectly through language.

Audio-domain systems increasingly depend on pretrained multimodal or text-to-audio backbones. VMB avoids direct common-embedding fusion and instead uses two explicit bridges: a text bridge produced by a Multimodal Music Description Model and a music bridge produced by Dual-track Music Retrieval, followed by Explicitly Conditioned Music Generation on top of Stable Audio Open (Wang et al., 2024). MeLFusion augments a text-to-music latent diffusion model with a “visual synapse” that injects self-attention key/value features from a pretrained Stable Diffusion model into the cross-attention key/value stream of the music diffusion decoder (Chowdhury et al., 2024). MusFlow maps CLIP image embeddings into the CLAP audio embedding space through modality-specific MLP adapters and then performs conditional flow matching in a VAE latent space of Mel-spectrograms (Song et al., 18 Apr 2025). Art2Mus extends AudioLDM 2 with an ImageBind-based image encoder and a trainable projection layer so that digitized artworks can serve as conditioning inputs for music generation (Rinaldi et al., 2024).

A parallel trend is foundation-model orchestration. The Zero-Effort framework uses Keye-8B as a VLM, LongCLIP for retrieval, MidiCaps as external memory, ABC notation as the symbolic bridge, MusPy-based self-refinement, and VLM attention maps for explanation; it requires no external training or fine-tuning for I2M (Zhao et al., 26 Sep 2025). MuMu-LLaMA, Mozart’s Touch, M2M-Gen, MTM, and related systems identified by the survey also fit this broader pattern of multimodal large-model conditioning, often with text as the bridge into MusicGen, MusicLM, Stable Audio Open, or autoregressive music generators (Wang et al., 27 Mar 2025).

3. Conditioning mechanisms and cross-modal alignment

The central technical problem in I2M is not merely generation but alignment: how an image should be converted into musically meaningful control variables. Several alignment strategies recur.

One strategy is semantic textualization. In MusicAIR, images are first converted into lyrics using LLMs, and the generated lyrics may include phrase structure and rhythmic textual organization; preference prompts can control output length (Liao et al., 21 Nov 2025). In VMB, the Multimodal Music Description Model based on InternVL2 with LoRA fine-tuning converts visual input into detailed natural-language music descriptions, while Dual-track Music Retrieval contributes paired musical references (Wang et al., 2024). In the Zero-Effort framework, the VLM first produces a textual description of the image, retrieval then supplies music-caption exemplars from MidiCaps, and the model generates music in ABC notation as text (Zhao et al., 26 Sep 2025). These systems treat language as an explicit, editable intermediate representation.

A second strategy is latent or embedding-space alignment. MusFlow encodes images with CLIP-ViT-B/32 and learns an adapter $MLP_I$ so that the mapped image embedding approaches the CLAP embedding of the paired music: $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$ For the image-only case this reduces to $e_f = MLP_I(e_I)$ and $\mathcal{L_A}^{I2M} = \|e_M - MLP_I(e_I)\|^2$ (Song et al., 18 Apr 2025). This design does not textualize the image; it instead treats the CLAP audio space as a representation of musical semantics and maps visual features into that space.

A third strategy is emotion-space alignment. The emotion-guided image-to-MIDI model trains with categorical cross-entropy for token prediction plus a VA loss computed from a pretrained music-emotion predictor: $$L_{\mathrm{total}} = \lambda_1 L_{\mathrm{VA}} + \lambda_2 L_{\mathrm{CC}},$$ with $\lambda_1 = 0.00001$ and $\lambda_2 = 1$ in the reported configuration (Kundu et al., 2024). The EIMG dataset likewise uses valence-arousal values for image-MIDI pairing (Wang et al., 27 Mar 2025). Emotion-guided painting-to-music generation takes an even narrower view by pairing paintings and music within five discrete affective categories and building prompts around emotion labels, captions, and music-oriented text enrichment (Hisariya et al., 2024). These systems make affect the primary bridge, though several papers also note that emotion alone is not a complete account of visual-musical correspondence.

A fourth strategy is retrieval augmentation. VMB uses broad retrieval for overall alignment and targeted retrieval for user control over attributes such as genre, tempo, or mood, with cosine-similarity search in CLIP or CLAP embedding spaces and conditioning injection through Music ControlFormer and a Stylization Module (Wang et al., 2024). The Zero-Effort approach also uses nearest-neighbor retrieval over MidiCaps descriptions embedded by LongCLIP and then inserts retrieved examples into the VLM prompt (Zhao et al., 26 Sep 2025). Retrieval reduces the burden on direct image-to-music mapping by supplying explicit musical references.

A fifth strategy is transformation consistency rather than free-form generation. Vis2Mus studies whether interpretable visual transformations such as brightness, contrast, and style transfer can induce corresponding transformations in symbolic music. Its central hypothesis is an equivariance-like relation $f(g(v)) \approx g(f(v))$, and it manipulates music texture latents rather than training a conventional direct mapper (Zhang et al., 2022). This is not unrestricted I2M, but it demonstrates a controllable cross-modal interface.

4. Data resources and evaluation practice

Benchmarking remains fragmented because paired image-music data are scarce and often weakly aligned. The survey lists several image-specific datasets: Music-Image with 22.6K image-audio pairs and 377 hours of music, Shuttersong with 586 user-selected image-audio pairs, IMAC with 3.8K image-audio pairs and emotion annotations, MUImage with 14.5K image-audio pairs and instruction annotations, EIMG with 3K image-MIDI pairs and valence-arousal values, and MeLBench with 11.2K image-audio pairs, 31.2 hours, and genre/caption annotations (Wang et al., 27 Mar 2025). MusFlow adds MMusSet, a 33.31k-sample, 92.49-hour dataset in which each sample is an image, story text, music caption, and music clip quadruple (Song et al., 18 Apr 2025). ConchShell introduces BOP with more than 3,000 images and more than 1,500 piano pieces across six image categories (Fan et al., 2022). Art2Mus constructs synthetic artwork-music and artwork-description datasets by pairing 10,000 digitized artworks with music tracks through ImageBind similarity and generating descriptions with LLaVA (Rinaldi et al., 2024).

These resources vary substantially in how they define alignment. Some are drawn from music videos or user-selected media; some are paired by emotion; some are paired through embedding similarity; some rely on generated captions or instructions rather than naturally authored image-music relationships (Wang et al., 27 Mar 2025). This suggests that “ground truth” in I2M is often weak supervision rather than direct compositional necessity.

Evaluation also splits into symbolic music quality, audio quality, and cross-modal alignment. Symbolic papers report metrics such as Scale Consistency, Pitch Entropy, Pitch Class Histogram Entropy, Empty Beat Rate, Average Inter-Onset Interval, Grooving Pattern Similarity, Structure Indicator, Polyphony Rate, and Groove Consistency (Wang et al., 27 Mar 2025). The emotion-guided symbolic model reports substantial gains in Polyphony Rate and Pitch Entropy when VA loss is added: after 15 epochs, the model with VA loss reports Music_Quality_Loss $0.7061$, Polyphony Rate $0.7818$, Pitch Entropy $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$0, and Groove Consistency $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$1, compared with $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$2, $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$3, $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$4, and $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$5 without VA loss (Kundu et al., 2024). MusicAIR evaluates symbolic scores with Music21 and uses key confidence, average interval, step ratio, and direction change rate; the paper reports average key confidence around 0.85 for generated songs versus 0.79 for human-composed originals in its paired-comparison experiment (Liao et al., 21 Nov 2025).

Audio systems typically report FAD, FD, KL divergence, and alignment scores derived from ImageBind or related models (Wang et al., 27 Mar 2025). VMB evaluates I2M on MUImage with 1,500 test pairs and reports $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$6, $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$7, and ImageBind score: VMB achieves $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$8, $$\mathcal{L_A} = \left\| e_M - \frac{1}{n}\sum_i MLP_i(e_i) \right\|^2, \qquad \mathcal{L_J} = \mathcal{L_G} + \lambda \mathcal{L_A}.$$9, and $e_f = MLP_I(e_I)$0, compared with CoDi’s $e_f = MLP_I(e_I)$1, $e_f = MLP_I(e_I)$2, and $e_f = MLP_I(e_I)$3, and M$e_f = MLP_I(e_I)$4UGen’s $e_f = MLP_I(e_I)$5, $e_f = MLP_I(e_I)$6, and $e_f = MLP_I(e_I)$7 (Wang et al., 2024). MusFlow reports FAD $e_f = MLP_I(e_I)$8, KL $e_f = MLP_I(e_I)$9, and ImageBind score $\mathcal{L_A}^{I2M} = \|e_M - MLP_I(e_I)\|^2$0 on image-to-music, outperforming M$\mathcal{L_A}^{I2M} = \|e_M - MLP_I(e_I)\|^2$1UGen and CoDi on all three metrics (Song et al., 18 Apr 2025). MeLFusion introduces IMSM, a text-mediated image-music similarity metric defined by

$\mathcal{L_A}^{I2M} = \|e_M - MLP_I(e_I)\|^2$2

and reports strong correlation between IMSM and human judgments on Extended MusicCaps and MeLBench (Chowdhury et al., 2024).

The literature also exposes benchmark fragility. VMB notes that MUImage did not provide a valid test split, so the first 1,500 pairs after alphabetical sorting were used, and M$\mathcal{L_A}^{I2M} = \|e_M - MLP_I(e_I)\|^2$3UGen was consequently evaluated on its training set for MUImage (Wang et al., 2024). MusicAIR includes image-origin generations in a pooled 517-song evaluation rather than reporting an isolated I2M benchmark (Liao et al., 21 Nov 2025). Art2Mus reports that its image-conditioned model does not outperform text-conditioned AudioLDM 2 baselines overall, which underscores that evaluation can penalize weakly aligned or synthetically paired datasets as much as weak models (Rinaldi et al., 2024).

5. Controllability, interpretability, and recurrent controversies

Interpretability is a recurrent design goal because the mapping from image to music is inherently ambiguous. The Zero-Effort framework treats this directly: it uses ABC notation as a human-readable symbolic bridge, retrieves relevant examples from MidiCaps, performs self-refinement through parser feedback and MusPy statistics, and exposes both textual motivations and VLM attention maps as explanations for the resulting music (Zhao et al., 26 Sep 2025). MusicAIR is likewise interpretable in a different sense: because the compositional core is algorithmic, the system explains note placement through keywords, syllables, phrase structure, sentiment, stressed beats, and music-theory rules rather than opaque latent fusion (Liao et al., 21 Nov 2025). Vis2Mus makes control explicit by turning visual edits such as brightness, contrast, and style transfer into music transformations and then validating them through user studies (Zhang et al., 2022).

Controllability is often implemented through explicit user handles. VMB provides broad retrieval for overall alignment and targeted retrieval for coarse-grained control over genre, tempo, or mood, and its broader controllable generation setting also allows custom reference music (Wang et al., 2024). MusicAIR offers prompt-controlled lyric length for image-to-lyrics generation, user key selection or random key choice, instrument selection for playback, and a choice between lyrical song and non-lyrical music (Liao et al., 21 Nov 2025). Zero-Effort allows control indirectly through retrieved exemplars and prompt design, though it does not foreground the same kind of attribute sliders or partitions (Zhao et al., 26 Sep 2025).

Several controversies recur. One concerns directness: many systems described as I2M are not direct image-to-music models. MusicAIR is explicit that its I2M path is image $\mathcal{L_A}^{I2M} = \|e_M - MLP_I(e_I)\|^2$4 lyrics $\mathcal{L_A}^{I2M} = \|e_M - MLP_I(e_I)\|^2$5 symbolic music, not pixels-to-notes or image-to-audio diffusion (Liao et al., 21 Nov 2025). The survey likewise shows that text-bridged strategies have become central rather than exceptional (Wang et al., 27 Mar 2025). A second concerns emotion reductionism: the Zero-Effort paper argues that emotion-based approaches are controversial because emotion is only “a singular aspect of art” (Zhao et al., 26 Sep 2025). Yet emotion-guided painting-to-music and the VA-guided MIDI system show that affective supervision remains practically useful, particularly when direct image-music pairs are scarce (Hisariya et al., 2024). A third concerns copyright and provenance. MusicAIR argues that copyright risk is mitigated because its music core does not train on song corpora and only uses LLMs to generate lyrics from images (Liao et al., 21 Nov 2025). Other systems rely on retrieval corpora, synthetic pairing, or large pretrained backbones, which shifts rather than eliminates questions about provenance and reuse.

6. Current limitations and likely research directions

The survey characterizes vision-to-music generation as still being in a preliminary stage because of music’s complex internal structure and the difficulty of modeling cross-modal relationships, and these difficulties are especially acute for I2M because static images do not provide explicit rhythm or temporal development (Wang et al., 27 Mar 2025). This structural underdetermination appears throughout the literature. Image-conditioned audio systems often remain short: MeLFusion reports 10-second output, Mozart’s Touch 10 seconds, MeLBench examples 10 seconds, and many symbolic or artwork-specific systems likewise operate on short clips or narrow repertoires (Wang et al., 27 Mar 2025). MusicAIR is limited to main melody generation without chord progressions or cadences (Liao et al., 21 Nov 2025). ConchShell is restricted to 8-second piano-only audio and still produces noisy outputs (Fan et al., 2022). Art2Mus is restricted by synthetic pairing quality and does not outperform text-conditioned baselines overall (Rinaldi et al., 2024).

Another limitation is that multimodal conditioning often helps more than image-only conditioning. In MeLFusion, text+image clearly outperforms both text-only and image-only variants, while random-image ablations severely degrade performance, indicating that visual semantics matter but are not sufficient by themselves in that architecture (Chowdhury et al., 2024). This suggests that pure I2M may remain harder than multimodal soundtrack generation unless image understanding, music planning, and control representations become richer.

Dataset quality is the other dominant bottleneck. Existing image-music datasets are relatively scarce, and many pair images and music by emotion, embedding similarity, or extracted video frames rather than through direct authorship (Wang et al., 27 Mar 2025). VMB’s improvised MUImage split, Art2Mus’s ImageBind pairing, MMusSet’s multi-agent annotation workflow, and MusicAIR’s pooled multimodal evaluation all illustrate that the field still lacks stable, richly annotated, high-confidence I2M benchmarks (Wang et al., 2024).

The literature points toward several concrete directions. One is stronger structured intermediate representations: lyrics in MusicAIR, music descriptions and retrieved references in VMB, ABC notation and motivations in Zero-Effort, and factorized scene/motion/style/color/composition reasoning in the music-to-image literature all suggest that underconstrained cross-modal generation benefits from explicit intermediate structure (Liao et al., 21 Nov 2025). A second is better emotion grounding, especially continuous valence-arousal supervision for both visual and musical modalities (Kundu et al., 2024). A third is hybridization of symbolic planning and audio rendering, which the survey identifies as a promising way to combine controllability with expressiveness (Wang et al., 27 Mar 2025). A fourth is more rigorous human-in-the-loop evaluation and control, including professional-composer feedback, preference alignment, and interactive editing, because many valid musical realizations can correspond to the same image (Wang et al., 27 Mar 2025).

The most precise current synthesis is that I2M has evolved from early symbolic mappings based on color, texture, and semantics into a heterogeneous family of systems that combine visual encoders, text bridges, retrieval, affective supervision, symbolic composition engines, diffusion models, autoregressive audio generators, and flow-matching backbones. The field’s strongest results now come not from a single universal mechanism, but from carefully engineered cross-modal bridges that compensate for the fact that a static image does not uniquely determine a piece of music.