Music Scene Imagination (MSI)

• Music Scene Imagination (MSI) is a cross-modal task that generates scene-oriented captions from music by emphasizing contextual settings and implied narrative elements.
• It spans diverse applications including music-to-video, image-to-music, and embodied interactions, often using intermediate representations like generated lyrics or semantic captions.
• Recent research demonstrates enhanced caption metrics and synchronization quality, though challenges remain in controllability, evaluation standardization, and realistic scene generation.

Music Scene Imagination (MSI) was introduced as a music understanding and captioning task in which a model receives music as input and generates a scene-oriented caption describing the kind of video context, atmosphere, setting, or event that the music would suit, rather than merely describing tempo, genre, instrumentation, or mood (Izzati et al., 8 Jul 2025). Across adjacent work, the term is also used in a broader survey sense for systems that translate music into dancing bodies, narrative illustrations, or still images, and for systems that generate music from images, videos, rooms, or embodied mixed-reality interaction. This suggests MSI is best understood both as a specific captioning problem and as a larger family of cross-modal scene–music imagination problems.

1. Definition, scope, and conceptual boundaries

In its strictest formulation, MSI asks a model to answer a scene-oriented question such as “what kinds of video would this piece of music be suitable for?” The point is to move from descriptions of music itself to descriptions of implied scene affordances. In the canonical example, a conventional music caption might describe “suspense and tension” and “a steady rhythm,” whereas an MSI caption describes “a scene of sports competition” and “a crucial moment in a basketball game” (Izzati et al., 8 Jul 2025). The distinction is therefore not between audio and text, but between two kinds of semantics: music-intrinsic semantics and scene-level semantics.

The broader literature shows that MSI is not confined to music-to-text. Some systems imagine a dancing human from music, some infer narrative illustrations, some generate images aligned with music-derived captions and valence–arousal signals, and some generate music from visual scenes, video, or environmental context. At the same time, several papers explicitly delimit their scope. “DanceIt,” for example, is a music-to-video system, but its imagined scene is specifically a dancing human body rendered as a target-person video, not arbitrary objects, environments, or narratives (Guo et al., 2020). “MR4MR” constructs place-specific ambient melody from mixed-reality collisions in a room rather than from abstract semantic scene understanding (Kobayashi et al., 2022). “ConchShell” generates short piano clips from still images in only six categories (Fan et al., 2022). These examples make clear that MSI ranges from narrow, person-centric or environment-centric settings to broader semantic scene imagination.

A common misconception is that MSI is equivalent to emotion recognition plus emotion-conditioned generation. Multiple papers argue against that reduction. The zero-effort image-to-music paper explicitly states that emotion is only “a singular aspect of art,” while “LUMIA” reports that overemphasis on literal objects degraded audio and therefore biases prompts toward contextual and atmospheric descriptors, with genre, tempo, and key kept consistent across segments (Zhao et al., 26 Sep 2025, Huang et al., 19 Dec 2025). MSI is therefore better characterized as cross-modal scene construction than as simple affect transfer.

2. Major task families and representative systems

One major MSI family proceeds from music to scenes or scene-like visuals. MusiScene formalizes MSI as music-to-scene captioning by fine-tuning Music Understanding LLaMA on VACAD, a dataset with 3,371 video-audio-caption pairs, and reports direct captioning gains over MU-LLaMA from BLEU $0.190 \rightarrow 0.421$, METEOR $0.207 \rightarrow 0.403$, ROUGE-L $0.219 \rightarrow 0.404$, and BERTScore $0.863 \rightarrow 0.901$ (Izzati et al., 8 Jul 2025). MusicJam converts music into generated lyrics with an AST-conditioned GPT-2 VAE, then uses Stable Diffusion to render narrative illustrations and synchronizes them with the original music as an MP4 video (Chen et al., 2023). DanceIt maps music to dancing video through shared audio–pose matching, retrieval of pose fragments, spatial-temporal alignment, and pose-guided video synthesis, yielding a person-centric but technically concrete MSI pipeline (Guo et al., 2020). MESA-MIG likewise takes music to image, but does so through music-derived captions, a valence–arousal regression head, and specialized Scene, Verb, Style, Color, Composition, Validator, and Prompt agents before text-to-image generation with WanX 2.1-Turbo (Shi et al., 29 Dec 2025).

A second family proceeds from scene representations to music. MeLFusion, also described as VisualBeats in the detailed exposition, formulates the task as $\mathcal{M}(\mathbf{w}\mid \mathbf{I}, \mathbf{Y})$, that is, image-plus-text to music waveform, and uses a latent diffusion music model with a visual synapse that injects Stable Diffusion attention features into decoder cross-attention layers (Chowdhury et al., 2024). VMAS takes video clips and generates EnCodec audio tokens with a generative video-music Transformer trained by a joint autoregressive and contrastive objective, plus a video-beat alignment scheme tied to optical-flow-derived motion peaks (Lin et al., 2024). ConchShell maps a $224\times224$ still image to an 8-second, 16 kHz, piano-only waveform by conditioning a Jukebox-style latent audio generator on pseudo-temporal image features produced by TCNN and I3D (Fan et al., 2022). MusicAIR supports lyrics-to-song, text-to-music, and image-to-music, but its image branch first converts images into lyrics with LLMs and then generates a symbolic melody via an algorithm-driven core (Liao et al., 21 Nov 2025).

A third family is embodied and place-responsive. MR4MR uses HoloLens spatial understanding so that virtual objects collide with reconstructed real-world surfaces; collision timbre depends on object type, pitch on collision height, volume on collision velocity, and dominant field-of-view color and brightness map to tonic and major/minor scale before MelodyRNN continuation and MusicVAE-based “melody reincarnation” (Kobayashi et al., 2022). LUMIA turns handheld camera framing into loopable musical sections: GPT-4 Vision produces a structured JSON caption with scene description, salient objects, mood, section role, genre, and BPM, which are merged with user-selected instruments and sent to Stable Audio 2.0 (Huang et al., 19 Dec 2025). MetaBGM translates continuously changing backend state into procedural narratives and then concise music descriptions for downstream audio generation, targeting multi-scene game or film contexts (Liu et al., 2024).

3. Representations, mediation layers, and architectural patterns

A defining property of MSI systems is their reliance on intermediate representations. Direct scene-to-waveform or music-to-image generation exists, but many successful systems insert a semantic or structural bottleneck. MusicJam uses a learned lyric line as the semantic bridge from audio to illustration [(Yusuf et al., 2023)?]

Correction: MusicJam uses generated lyrics as prompts between music and images (Chen et al., 2023). MusiScene uses scene captions as the target representation learned from cross-modal video–music supervision (Izzati et al., 8 Jul 2025). MESA-MIG uses music-derived captions and then decomposes them into scene, motion, style, color, and composition attributes (Shi et al., 29 Dec 2025). MusicAIR uses lyrics, text, or image-derived lyrics to derive phrase structure, keywords, syllables, sentiment, time signature, key signature, number of measures, rhythmic score, and finally MusicXML or MIDI-like outputs (Liao et al., 21 Nov 2025). The zero-effort image-to-music framework uses ABC notation as a bridge between language and music and augments a VLM with multimodal RAG and self-refinement, explicitly so that music can be generated using natural language without task-specific training (Zhao et al., 26 Sep 2025).

The underlying music or motion representations vary sharply. DanceIt uses MFCC audio features, 2D pose skeleton sequences extracted by OpenPose, 16D shared latent embeddings, retrieved pose fragments from a database $\mathbb{D}$, and then aligned pose sequences before pose-guided rendering (Guo et al., 2020). VMAS uses EnCodec discrete audio tokens with 4 codebooks of size 2048 and a dense temporal video encoder that processes 96 frames per 10-second clip (Lin et al., 2024). MeLFusion uses spectrograms encoded by an Audio-VAE into latent tensors $\mathbf{z}_1^M$, then performs latent diffusion with the visual synapse

$$\mathbf{K}^M_l=\alpha_l \mathbf{K}^I_l+(1-\alpha_l)\mathbf{K}^M_l,\qquad \mathbf{V}^M_l=\alpha_l \mathbf{V}^I_l+(1-\alpha_l)\mathbf{V}^M_l$$

so that image semantics modulate music generation inside the denoiser rather than only at the prompt level (Chowdhury et al., 2024). Pictures of MIDI takes yet another route by treating symbolic music as a piano-roll image and using HDiT diffusion inpainting with arbitrary mask geometries and RePaint-style repainting in the masked region (Hawley, 2024).

These choices reveal several recurrent MSI architectures. One is retrieval-and-synthesis, exemplified by DanceIt, where the system “imagines” by recombining previously seen motion fragments and then repairing them (Guo et al., 2020). Another is prompt-mediated generation, exemplified by MusicJam, MusicAIR, MESA-MIG, and the zero-effort VLM system, where language serves as the scene/music bottleneck (Chen et al., 2023, Liao et al., 21 Nov 2025, Shi et al., 29 Dec 2025, Zhao et al., 26 Sep 2025). A third is cross-modal latent fusion, exemplified by MeLFusion and VMAS, where conditioning enters internal attention or pooled latent spaces rather than only through surface prompts (Chowdhury et al., 2024, Lin et al., 2024). This suggests MSI is often less a single end-to-end model family than a design pattern built around interpretable mediation layers.

4. Embodiment, interaction, and live scene construction

MSI research also includes systems where scenes are not static inputs but are enacted through bodily movement, mixed reality, or continuous control. MR4MR turns a user’s immediate room into a mixed-reality musical scene: virtual objects are bumped into real walls, tables, and chairs reconstructed by HoloLens, collision events are spatialized with the Oculus Spatializer, and, when enough collisions accumulate, the sequence seeds MelodyRNN continuation and later MusicVAE-based regeneration (Kobayashi et al., 2022). The dominant color of the HoloLens view is clustered by k-means and then mapped via a handcrafted HSV-to-scale table, so environmental ambiance affects tonic and major/minor choice. The authors explicitly state that if there are many objects in the space, the number of sounds increases and following melodies are generated more frequently, making room geometry a direct conditioner of musical behavior.

LUMIA reframes MSI as “compose through looking.” The handheld device streams video; when the capture button is pressed, a 1280×720 JPEG frame of about 120 KB is sent to GPT-4 Vision, which returns a six-field structured JSON caption: overall description, salient objects, mood adjectives, section role in $\{\text{intro, verse, chorus, bridge, outro}\}$, music genre, and suggested BPM (Huang et al., 19 Dec 2025). Those descriptors are programmatically merged with user-selected instruments and section modifiers into a single-sentence prompt for Stable Audio 2.0, producing 15-second stereo WAV sections at 44.1 kHz. The loop engine then aligns sections by bar-quantized scheduling and uses equal-power or power-law crossfades. End-to-end latency from capture to audio entering the loop is reported as 5.0–6.5 seconds, and capture to mixed audio update as 10–13 seconds.

MetaBGM addresses adaptive soundtrack generation for continuous multi-scene experiences by translating backend state into procedural narrative text and then into a concise music description, typically every 10 seconds (Liu et al., 2024). In combat mode, its rule-based feature selection retains fields such as Scene, Health, Satiety, Status, Movement, Position, Hostile Entity, and Being Attacked; otherwise it retains Scene, Time, Weather, Temperature, Status, Movement, and Position. The LLM layer uses LLaMA-2-7B-Chat with LoRA to narrativize these structured states and then convert the narrative into a short music description intended for a downstream audio model. The system is therefore scene-aware and user-state-aware, but the imagination step is mediated by narrative text rather than direct multimodal generation.

Malakai approaches live MSI from adaptive composition. Its architecture is divided into The Curves, The Graph, and The Blocks, and its runtime control space uses Energy, Valence, and Complexity, corresponding to Greenberg’s Arousal, Valence, and Depth (Harris et al., 2021). A Composer wires algorithmic and machine-learning modules in a graph; a Listener then shapes high-level emotional trajectories through curves. In the example pipeline, MidiMe and MusicVAE produce melody and countermelody, while a procedural generator produces chord progressions. This is not scene imagination in the narrow captioning sense, but it is directly relevant to live MSI because it treats a musical scene as a controllable generative process parameterized by evolving affective state.

The broader live-performance context has now been systematized by the live music agents design space, which analyzes 184 systems and organizes them into 31 dimensions and 165 codes across Usage Context, Interaction, Technology, and Ecosystem (Kim et al., 4 Feb 2026). Its dimensions such as I/O Modality, Agent Role, System Initiative, Planning, and Temporal Structure provide a general vocabulary for positioning MSI systems within live co-creative practice.

5. Evaluation protocols and empirical evidence

MSI evaluation is heterogeneous because the field mixes captioning, symbolic generation, waveform generation, video synthesis, mixed reality, and interactive systems. MusiScene evaluates scene-imagining captions directly with BLEU, METEOR, ROUGE-L, and BERTScore, and then uses those captions as MusicGen prompts for downstream video background music generation; MSI captions achieve human-rated suitability scores of 74.2 against 61.4 for video captions and 73.5 for music captions, and 78.4 against 76.6 for fusion captions (Izzati et al., 8 Jul 2025). Here the key claim is that scene-oriented captions are more useful than plain music descriptions for downstream soundtrack generation.

MusicJam combines automatic lyric metrics with a 40-participant video study. Its lyric model improves diversity and novelty over baselines and attains CLIPScore 23.17 against 22.62 and 23.03 for baselines, while end-to-end videos are rated higher than WZRD for music-video relevance, aesthetics, meaningfulness, and comprehensibility in most reported comparisons (Chen et al., 2023). MeLFusion introduces IMSM, an image-music semantic consistency metric built by composing CLIP and CLAP similarities, and reports on Extended MusicCaps / MeLBench: FAD 1.12 / 1.05, KL 0.89 / 0.72, IMSM 0.76 / 0.83, OVL 86.78 / 88.45, and REL 85.92 / 87.39, along with relative FAD gains of 67.05% and 67.98% over MusicGen (Chowdhury et al., 2024). VMAS evaluates waveform music realism with FAD and KL plus video–music synchronization with MV Align, obtaining 2.38 FAD, 1.34 KL, and 0.35 MV Align on DISCO-MV, and reports that human raters preferred it over baselines on average over 70% for overall quality and about 67% for alignment (Lin et al., 2024).

DanceIt uses metrics tailored to person-centric MSI: beat alignment score $0.207 \rightarrow 0.403$0, Moving Distribution Distance, and Spacing Distribution Distance. Against Lee et al.’s “Dancing to Music,” its beat alignment improves from 60.05% to 76.25% with 2-second fragments, 67.93% with 3-second fragments, and 78.80% with 4-second fragments, and the matching model reaches 0.83 test correlation accuracy (Guo et al., 2020). ConchShell uses PESQ, STOI, and MOS-style measures for image-to-music relevance, melody, and cleanliness; on the test set it reports PESQ 2.9, STOI 0.66, I2M-MOS $0.207 \rightarrow 0.403$1, Muse-MOS $0.207 \rightarrow 0.403$2, and Clean-MOS $0.207 \rightarrow 0.403$3, still well below ground truth (Fan et al., 2022). The zero-effort VLM-based I2M system uses 31 human participants and reports music-image consistency average 3.82 against 2.79 for Synesthesia and 3.35 for Mozart’s Touch, with machine-judged consistency average 5.7 against 3.5 and 5.0, respectively (Zhao et al., 26 Sep 2025).

Interactive systems are evaluated more modestly but still inform MSI. MR4MR reports 27 gallery participants with mean interestingness $0.207 \rightarrow 0.403$4, pleasantness $0.207 \rightarrow 0.403$5, uniqueness $0.207 \rightarrow 0.403$6, and “musical experience” $0.207 \rightarrow 0.403$7 on 5-point Likert scales (Kobayashi et al., 2022). LUMIA reports a formative study with three professional audio engineers; authorship share averaged 4.0/10 and expectation match 6.3/10, while participants highlighted faster “vibe-finding” and requested genre/BPM locks, micro-editing, mapping legends, and lower latency (Huang et al., 19 Dec 2025). Collectively, these evaluations show that MSI still lacks a single standard protocol: overlap metrics, audio-distribution metrics, scene-consistency metrics, beat-alignment metrics, and qualitative studies coexist without a unified benchmark.

6. Limitations, misconceptions, and emerging directions

A recurring limitation is domain narrowness. DanceIt assumes fixed or stable camera, solo dancer, bright lighting, a short guidance video, and a private 154-video dance corpus, and its scene imagination is explicitly “not general music-to-scene imagination” (Guo et al., 2020). ConchShell is limited to six image categories, piano-only output, 8-second clips, and a manually paired Beach-Ocean-Piano dataset (Fan et al., 2022). MetaBGM assumes structured backend state rather than open-ended scene understanding, and its actual audio backend is weakly specified (Liu et al., 2024). MusicAIR focuses on main melody generation, not full harmony or orchestration, and its image branch depends on image-to-lyrics conversion rather than direct visual-to-musical mapping (Liao et al., 21 Nov 2025). MESA-MIG outputs still images rather than temporally evolving scenes and relies heavily on caption quality and a general-purpose text-to-image model (Shi et al., 29 Dec 2025).

Another misconception is that MSI systems are always fully generative and end-to-end. Several influential systems are not. DanceIt is predominantly retrieval plus smoothing and rendering (Guo et al., 2020). MusicJam relies on a semantic bottleneck of generated lyrics before illustration (Chen et al., 2023). MetaBGM is a scene-state-to-narrative-to-music-description interface rather than a novel audio model (Liu et al., 2024). The zero-effort image-to-music framework is a VLM-plus-RAG orchestration method rather than a trained scene-to-music network (Zhao et al., 26 Sep 2025). This suggests MSI is as much about system architecture and cross-modal mediation as about raw generative model capacity.

A third limitation concerns controllability, deployment, and infrastructure. MIDInfinite is explicitly not an MSI model, but it materially lowers the barrier to interactive local symbolic generation by running the Anticipatory Music Transformer in-browser via MLC-LLM, reaching 155 tok/s, about 51 notes/s, on M3 Max with MLC and 72.9% streamability, rising to 86.3% with 2 seconds of buffering (Zhou et al., 2024). Pictures of MIDI is likewise not scene-semantic MSI, but it demonstrates that arbitrary mask geometries over piano-roll images can function as visual compositional constraints for symbolic generation (Hawley, 2024). MRCV provides an open-source suite for explorative music generation, sound design, instrument creation, and procedural score generation, emphasizing accessibility and custom datasets rather than MSI semantics (Clarke, 2023). These systems indicate that MSI research increasingly depends on enabling toolchains, not only on benchmark models.

The open problems named across the literature are consistent. MusicJam calls for richer semantic representations such as story scripts, scene graphs, affect trajectories, or multimodal latent plans (Chen et al., 2023). MR4MR proposes object recognition so that material and object identity, not only geometry and color, affect musical output (Kobayashi et al., 2022). LUMIA proposes an intermediate prompt-editing layer and additional embodied controls such as an FX button mapped to shakes, strikes, or pressure input (Huang et al., 19 Dec 2025). MeLFusion identifies future work around semantic lyrics and richer video-conditioned settings (Chowdhury et al., 2024). Taken together, these proposals suggest that the next stage of MSI will likely require stronger scene semantics, better temporal coherence, explicit controllability, human-legible intermediate representations, and evaluation protocols that measure not just output quality but whether the generated music or imagery matches what the scene makes people imagine.