MusiScene: Music Scene Imagination
- MusiScene is a music captioning model that generates scene-centric descriptions by mapping audio cues to plausible visual scenarios.
- It fine-tunes MU-LLaMA with a music understanding adapter using cross-modal supervision on the VACAD dataset, nearly doubling key evaluation metrics.
- The generated MSI captions serve as effective prompts for video background music generation, aligning audio and narrative context for improved fit.
Searching arXiv for MusiScene and closely related background papers to ground the article with current citations. {"query":"MusiScene MU-LLaMA music scene imagination video background music generation", "max_results": 10} Searching arXiv for adaptive scene-aware background music systems and music-LLMs relevant to MusiScene. {"query":"adaptive scene-aware background music generation music-LLM MetaBGM TinyMU MuseAgent MU-LLaMA", "max_results": 10} MusiScene is a music captioning model designed for Music Scene Imagination (MSI): given a piece of music, it generates a natural-language description of a plausible visual scene, atmosphere, or situation for which that music would be an appropriate soundtrack. Built by fine-tuning MU-LLaMA on cross-modal video–audio supervision, MusiScene shifts music captioning from description of musical attributes such as genre, mood, tempo, and instrumentation toward scene-centric semantics such as environment, event type, and narrative moment. The resulting MSI captions are then used to improve Video Background Music Generation (VBMG) by serving as text prompts for text-to-music systems (Izzati et al., 8 Jul 2025).
1. Task definition and conceptual scope
The central problem formalized by MusiScene is Music Scene Imagination. In MSI, the question is not merely what the music sounds like, but what kind of scene the music would fit. The target output is therefore a description of a plausible visual scenario or video context in which the music would be suitable, rather than a purely musicological summary (Izzati et al., 8 Jul 2025).
This distinguishes MusiScene from standard music captioning systems. Conventional models, including MU-LLaMA, emphasize musical attributes such as genre, mood, rhythm, key, and instrumentation. MusiScene instead targets scenario-level semantics: environment, atmosphere, event structure, and implied narrative. In the motivating examples, slow, melancholic music may evoke heartbreak, whereas upbeat music may suggest celebration; MusiScene operationalizes that human tendency as a supervised modeling task (Izzati et al., 8 Jul 2025).
The task is intrinsically cross-modal. The model must learn associations between audio and unseen visual contexts, such as tense orchestral music aligning with a sports climax or an action scene. A plausible implication is that MSI probes a deeper layer of music understanding than attribute tagging alone, because it requires mapping audio cues to latent visual and narrative affordances rather than to isolated labels.
2. Model architecture and training objective
MusiScene reuses the base architecture of MU-LLaMA. The pipeline consists of a MERT encoder, a music understanding adapter, and LLaMA 7B-chat as the generative LLM (Izzati et al., 8 Jul 2025).
Within this stack, MERT converts the input music into high-level audio embeddings. The adapter then aggregates temporal features via Conv1D and projects them through dense layers into the embedding space expected by LLaMA. Conceptually, the computation is:
MusiScene keeps the overall multimodal fusion pattern of MU-LLaMA but changes the supervision target to MSI captions. During training, MERT and LLaMA 7B-chat are frozen, and only the music understanding adapter is fine-tuned. The learning objective is standard conditional autoregressive cross-entropy over caption tokens:
$\mathcal{L}(\theta_{\text{adapter}}) = - \sum_{t=1}^{T} \log p_{\theta_{\text{adapter}}, \theta_{\text{LLM}}, \theta_{\text{MERT}} \bigl(y_t \mid y_{<t}, \mathbf{c}, \mathbf{x}_\text{audio}\bigr).$
Here, denotes the music input, the textual MSI prompt, and the target MSI caption. In effect, the adapter learns a new projection from music embeddings into a language space specialized for scene imagination rather than ordinary captioning (Izzati et al., 8 Jul 2025).
The fine-tuning setup is intentionally lightweight. MusiScene uses 20 epochs, runs on 4× NVIDIA A100 GPUs, and requires approximately 5 hours of training time on the reported dataset split (Izzati et al., 8 Jul 2025).
3. VACAD and construction of MSI supervision
Because no pre-existing MSI dataset was available, the MusiScene work introduces VACAD (Video–Audio CAptions Dataset). VACAD is built from 3,371 video clips selected from the AudioSet test set using the label “Music” (Izzati et al., 8 Jul 2025).
Each selected clip is associated with four caption types. First, video captions are generated with SwinBERT. Second, music captions are generated with MU-LLaMA. Third, fusion captions are produced by Mixtral of Experts from the video caption and music caption jointly. Fourth, and most importantly, MSI captions are also produced by Mixtral, but with a prompt that asks what type of scene the music is suitable for (Izzati et al., 8 Jul 2025).
This data construction strategy is central to the model’s semantics. The MSI target text is derived from both video context and music description, yet the trained model receives only audio at inference time. The supervisory signal therefore embeds cross-modal knowledge into an audio-conditioned captioning problem. A plausible implication is that MusiScene approximates a distilled form of video–audio reasoning: it learns to infer scene suitability from music alone because its labels were generated from both modalities.
The paper’s basketball example illustrates the distinction among caption types. A video caption such as “A basketball game is being played in front of a crowd” encodes visible content; a MU-LLaMA caption such as “The music has a mood of suspense and tension, with a steady rhythm and a strong beat” encodes musical properties; the MSI target instead states that the music is suitable for “a scene of sports competition” and specifically “a crucial moment in a basketball game” (Izzati et al., 8 Jul 2025). That shift from property description to scene suitability is the defining annotation move in VACAD.
4. Caption characteristics and MSI behavior
MusiScene outputs are designed to include environment description, emotional atmosphere, possible storyline or event, implicit visual cues, and, in some cases, hints of temporal dynamics such as build-up or climax (Izzati et al., 8 Jul 2025). The captions are therefore scene-centric rather than music-centric.
In practical terms, MusiScene retains musical evidence only indirectly. It does not generally enumerate tempo, instrumentation, or genre unless those attributes help specify the imagined scene. Instead, musical cues are translated into contextual descriptions such as a sports climax, a calm nature scene, or an intense chase. This makes its text especially suitable as downstream control input for text-to-music models, because the prompt describes what the music should support.
The reported MSI captioning metrics show a substantial difference between the base MU-LLaMA behavior and the MusiScene-fine-tuned model (Izzati et al., 8 Jul 2025).
| Metric | MU-LLaMA | MusiScene |
|---|---|---|
| BLEU (B-U) | 0.190 | 0.421 |
| METEOR (M-R) | 0.207 | 0.403 |
| ROUGE-L (R-L) | 0.219 | 0.404 |
| BERTScore (B-S) | 0.863 | 0.901 |
These values indicate that MusiScene more closely matches the Mixtral-generated MSI references than MU-LLaMA does. The improvement is especially notable in BLEU, METEOR, and ROUGE-L, which roughly double relative to the base model. A plausible implication is that ordinary music captions and MSI captions are not stylistic variants of the same output space; they constitute materially different generation targets.
5. Role in video background music generation
The downstream motivation for MusiScene is Video Background Music Generation. In the reported setup, generated captions from several sources are fed into MusicGen to synthesize background music, enabling comparison among prompt types: MSI captions, video captions, music captions, and fusion captions (Izzati et al., 8 Jul 2025).
Objective evaluation uses Fréchet Audio Distance (FAD) and KL Divergence between generated and reference audio distributions. The reported values are:
| Text input | FAD | KL Div. |
|---|---|---|
| MSI (Ours) | 5.78 | 2.10 |
| Video Caps | 4.46 | 2.13 |
| Music Caps | 6.43 | 2.12 |
| Fusion | 6.62 | 2.05 |
By FAD, video captions yield the lowest distance, whereas by KL divergence fusion captions are lowest. MusiScene’s MSI captions are competitive on both metrics rather than dominant on either one (Izzati et al., 8 Jul 2025). The paper explicitly notes that lower FAD does not automatically imply better perceptual suitability, which is why subjective evaluation is also performed.
Human judgments are more favorable to MSI prompts. In one evaluation with 15 videos, 3 generated tracks each, and 68 respondents, the average suitability scores were 74.2 for MSI, 73.5 for music captions, and 61.4 for video captions. In a second comparison with 8 videos, 2 tracks each, and 50 respondents, MSI again led with 78.4 versus 76.6 for fusion captions (Izzati et al., 8 Jul 2025).
These findings suggest that MSI captions provide a prompt format more aligned with perceived video–music fit than either purely visual descriptions or purely musical descriptions. A plausible implication is that MSI acts as a task-specific intermediate representation: it compresses multimodal context into language already oriented toward soundtrack suitability, which may be easier for a text-to-music model to interpret than verbose fusion text or literal video description.
6. Related systems, limitations, and research trajectory
MusiScene sits at the intersection of music captioning, multimodal language modeling, and scene-aware music generation. Relative to MU-LLaMA, it redefines the captioning target from musical description to scene imagination (Izzati et al., 8 Jul 2025). Relative to LP-MusicCaps, it remains audio-grounded but moves beyond metadata-style music description. Relative to Di et al. (2021) and Herrmann-1, it functions as a music-language module that can bridge audio understanding and video-oriented generation (Izzati et al., 8 Jul 2025).
It is also distinct from systems that start from scene state and generate music prompts directly. For example, MetaBGM models interactive, continuous multi-scene background music generation by converting backend state into a procedural narrative and then into music description text for real-time soundtrack creation (Liu et al., 2024). MusiScene inverts that direction: it starts from music, imagines a suitable scene, and then reuses that scene-centric text to improve VBMG. The two systems therefore address complementary parts of a scene–music loop: one is scene-to-music prompting, the other is music-to-scene captioning.
The limitations reported for MusiScene are significant. The dataset contains only 3,371 clips, which constrains genre and scenario diversity. The MSI ground truth is LLM-generated by Mixtral rather than human-authored, so supervisory quality is bounded by the prompting and priors of that model. At inference time, MusiScene receives audio only, which means it lacks explicit visual grounding and may default to generic scene types for ambiguous music. The work also explores only limited architectural variation, since fine-tuning is confined to the adapter (Izzati et al., 8 Jul 2025).
Future directions follow directly from these constraints. The paper points toward larger and more diverse datasets, human-authored MSI captions, richer temporal and narrative structure in captions, direct incorporation of visual features when available, and interactive or real-time applications such as dynamic soundtrack generation and creator-facing film or game scoring tools (Izzati et al., 8 Jul 2025). In that sense, MusiScene can be read as an initial formulation of MSI rather than a final architecture: its main contribution is to define scene imagination as a distinct, trainable interface between music understanding and background music generation.