---
title: 'V2M-Zero: Zero-Pair Video-to-Music Generation'
url: https://www.emergentmind.com/topics/v2m-zero
type: topic
---

# V2M-Zero: Zero-Pair Video-to-Music Generation

Searching arXiv for the specified paper to ground the article.
V2M-Zero is a zero-pair video-to-music generation approach that outputs time-aligned music for video by conditioning a pretrained text-to-music model on an event curve derived from temporal change, rather than on paired cross-modal supervision. Its defining claim is that temporal synchronization depends not on what changes across modalities, but on when and how much change occurs; accordingly, it computes comparable temporal representations independently within music and video using intra-modal similarity, fine-tunes a text-to-music model on music-event curves, and substitutes video-event curves at inference time to produce synchronized soundtracks without paired video–music training data [2603.11042].

## 1. Conceptual basis

Existing text-to-music systems such as MusicLM, MusicGen, and AudioLDM produce high-quality music from text prompts but have no mechanism to follow a video’s fine-grained temporal structure. In that setting, creators must manually edit video to match generated music. V2M-Zero addresses the specific problem of temporal alignment rather than generic music generation [2603.11042].

The central observation is that visual and musical events need not match semantically in order to align temporally. Scene cuts, motion bursts, beat onsets, and instrumentation changes differ in modality and meaning, yet they may share a common temporal structure. V2M-Zero operationalizes this structure through an event curve: a one-dimensional signal that marks moments of strong temporal change. Because the curve is computed independently inside each modality, the method does not require paired video–music examples or explicit cross-modal representation learning.

A common misconception in this problem setting is that synchronized video-to-music generation must be trained with paired video–music data. V2M-Zero is explicitly constructed to test the opposite hypothesis. Its reported results indicate that within-modality temporal features can suffice for synchronization when semantic control is handled separately through text prompts. This suggests a decomposition between temporal control and semantic control rather than a single fused cross-modal objective.

## 2. Event curves from intra-modal similarity

V2M-Zero converts a sequence of high-level features into a 1D event curve that indicates moments of strong temporal change. The same procedure is used for music during training and for video during inference [2603.11042].

For music, a pretrained music encoder such as MusicFM extracts
$$
\mathbf f_m \in \mathbb R^{d_m\times l_m},\quad \mathbf f_m^k\in\mathbb R^{d_m}
$$
for \(k=1,\dots,l_m\). For video, a pretrained visual encoder such as DINOv2 is applied on each frame and spatially pooled to obtain
$$
\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.
$$

Given a generic feature sequence \(\mathbf f\in\mathbb R^{d_f\times l_f}\), adjacent cosine similarities are computed as
\[
s^k \;=\;
\frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}}
     {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|},
\quad
k=1,\dots,l_f-1.
\]
These are converted into a dissimilarity or novelty sequence,
\[
a^k \;=\; 1 - s^k,\quad
\mathbf A = [\,a^1,\dots,a^{l_f-1}\,].
\]

To make the resulting curves comparable across modalities despite different scales and sampling rates, the novelty sequence is standardized, resampled, and smoothed:
\[
\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)},
\quad
k=1,\dots,l_f-1,
\]
followed by resampling to length \(l\), the number of audio latent frames, and Hann-window smoothing:
\[
\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.
\]
At training time this yields the music-event curve \(\mathbf e_m\); at test time it yields the video-event curve \(\mathbf e_v\). The default smoothing kernel is a Hann window of size \(31\), selected by ablation.

The methodological significance of this construction is that it avoids direct alignment of visual and musical semantics. Only the temporal profile of change is transferred across modalities. This is the technical mechanism underlying the “zero-pair” designation.

## 3. Base model and event-conditioned fine-tuning

V2M-Zero builds on a pretrained latent rectified-flow text-to-music model with a DiT backbone [2603.11042]. An audio autoencoder converts stereo 44.1 kHz waveforms into continuous latents
\[
\mathbf x_0\in\mathbb R^{d\times l},
\]
with \(d=64\) and \(l=394\) for 32 s.

The rectified-flow formulation defines a noising path
\[
\mathbf x_t = t\,\boldsymbol\epsilon + (1-t)\,\mathbf x_0
\]
with \(\boldsymbol\epsilon\sim\mathcal N(0,I)\), and learns a velocity field \(f_\theta\) mapping from \(\mathcal N(0,I)\) to the data distribution. Without event-curve conditioning, the training objective is
\[
\mathcal L_{\mathrm{rf}}
= \mathbb E_{\mathbf x_0,\boldsymbol\epsilon,t,\mathbf c}
\bigl\|\,(\boldsymbol\epsilon-\mathbf x_0)
- f_\theta(\mathbf x_t,\,\mathbf c,\,t)\bigr\|_2^2,
\]
where \(\mathbf c\) is the text embedding provided via cross-attention.

Event-curve conditioning is introduced by concatenating the 1D curve as an additional input channel:
\[
\widetilde{\mathbf x}_t
= \bigl[\mathbf x_t\,;\,\mathbf e\bigr]
\;\in\;\mathbb R^{(d+1)\times l}.
\]
Only the input projection of the DiT is extended from \(d\) to \(d+1\), adding approximately \(2\) K parameters. During fine-tuning, \(\mathbf e\) is replaced by the music-event curve \(\mathbf e_m\), giving
\[
\mathcal L_\mathrm{V2M}
= \mathbb E_{\mathbf x_0,\boldsymbol\epsilon,t,\mathbf c,\mathbf e_m}
\bigl\|\,(\boldsymbol\epsilon-\mathbf x_0)
- f_\theta\bigl(\widetilde{\mathbf x}_t,\mathbf c,t\bigr)\bigr\|_2^2.
\]

Fine-tuning is performed on approximately \(25\) K h of instrumental music with AdamW, learning rate \(1\times 10^{-4}\), and classifier-free guidance with \(10\%\) text/event drop. Reported fine-tuning cost is \(192\)–\(768\) GPU h on \(4\)–\(8\) A100s, corresponding to \(2\)–\(4\) days. Architecturally, the minimal parameter increase is notable: temporal control is introduced without redesigning the backbone or adding an explicit cross-modal module.

## 4. Zero-pair inference and prompt construction

At test time, no weights are changed; the music-event curve is simply replaced with the video-event curve computed by the same standardize \(\rightarrow\) resample \(\rightarrow\) Hann smooth pipeline [2603.11042]. This direct substitution is the core inference-time mechanism of V2M-Zero.

Semantic conditioning is obtained from the input video through a multi-stage prompt-generation procedure. ASR with Whisper extracts a transcript \(T\). Sampled frames are captioned by a vision-language model such as Gemma-4B to produce \(\{C_i\}\). An LLM, again exemplified by Gemma-4B, summarizes these captions into a visual summary \(S\). A final LLM prompt applied to \((T,S)\) yields the music prompt \(\mathbf c\). The appendix pseudocode is:
```python
# Music prompt from video
T = ASR(video)
C = { VL_caption(f) for f in sampled_frames(video) }
S = LLM_summarize(C)
c = LLM_generate_music_prompt(T, S)
```

Sampling proceeds by solving the rectified-flow ODE
\[
d\mathbf x_t = -\,f_\theta(\,\mathbf x_t\,,\,\mathbf c\,,\,t)\,dt
\]
from \(t=1\) to \(0\) in \(96\) steps with classifier-free guidance. The decoded \(\mathbf x_0\) waveform is then added to the video.

This inference design separates two sources of control. The text prompt \(\mathbf c\) governs semantic attributes of the soundtrack, while \(\mathbf e_v\) governs temporal alignment. A plausible implication is that the model can inherit semantic flexibility from a general text-to-music system while imposing synchronization through a lightweight temporal side channel.

## 5. Evaluation protocol and reported results

The reported evaluation uses three datasets: OES-Pub, MovieGenBench-Music, and AIST++ [2603.11042]. OES-Pub is the ISMIR ’25 public split of OSSL and contains \(115\) public-domain movie clips of approximately \(30\) s with royalty-free music and human-annotated prompts. MovieGenBench-Music contains \(527\) video–music pairs of approximately \(10\) s spanning diverse content and includes music prompts. AIST++ contains \(20\) dance clips of approximately \(7\) s across \(10\) genres with ground-truth BPM.

The metric suite covers audio fidelity, semantic alignment, temporal synchronization, and dance-specific beat tracking. Audio fidelity is measured by Fréchet Audio Distance in VGGish space, with OES-Pub additionally using FAD* against an external music reference set. Semantic alignment is measured by CLAP score, defined as cosine similarity between text and generated music. Temporal synchronization includes Scene-Cut Hit, the fraction of scene cuts with at least one beat onset within \(\pm 100\) ms, and pairwise human preference on quality and synchronization. Dance-specific evaluation on AIST++ uses Beat Coverage Score, Beat Hit Score, \(F1\), and Temporal Deviation.

On OES-Pub and MovieGenBench, V2M-Zero is reported to outperform supervised baselines including VidMuse, AudioX, GVMGen, and SONIQUE. On OES-Pub it achieves FAD* \(4.95\), CLAP \(0.23\), and SCH \(0.61\), corresponding to \(21\)–\(52\%\) lower FAD*, \(13\)–\(15\%\) higher CLAP, and \(21\)–\(52\%\) higher SCH relative to the cited paired-data baselines. On MovieGenBench it reports FAD \(2.68\), CLAP \(0.18\), and SCH \(0.58\). In human A/B tests with \(1\,403\) votes, it wins approximately \(69\%\) on music quality and approximately \(63\%\) on synchronization.

On AIST++, V2M-Zero reports BCS \(0.5818\), BHS \(0.6274\), \(F1\) \(0.5856\), and TD \(12.24\) s, compared with a prior best \(F1\) of \(0.4572\) and TD of \(17.80\) s. The abstract summarizes these outcomes as \(5\)–\(21\%\) higher audio quality, \(13\)–\(15\%\) better semantic alignment, \(21\)–\(52\%\) improved temporal synchronization, and \(28\%\) higher beat alignment on dance videos.

## 6. Ablations, implementation considerations, and research significance

The ablation studies focus on modality-gap mitigation, encoder selection, domain adaptation, prompting, and the necessity of event-curve conditioning [2603.11042]. For the smoothing kernel, larger Hann windows from \(9\) to \(63\) reduce domain shift and improve FAD, but excessive smoothing degrades SCH by flattening peaks; the best trade-off is reported at size \(31\).

For encoder selection, the default pairing of MusicFM and DINOv2 yields the strongest overall alignment. The reported comparison is summarized below.

| Music encoder + Video encoder | FAD* ↓ | CLAP ↑ / SCH ↑ |
|---|---:|---:|
| AVSiam | 4.52 | 0.19 / 0.35 |
| VAE + V-JEPA | 5.13 | 0.18 / 0.41 |
| VAE + DINOv2 | 4.77 | 0.16 / 0.31 |
| MusicFM + V-JEPA | 5.02 | 0.18 / 0.48 |
| MusicFM + DINOv2 | 4.95 | 0.23 / 0.61 |

For dance-specific inputs, replacing DINOv2 with CoTracker improves AIST++ performance from BCS \(0.5522\) to \(0.5818\) and reduces TD from \(17.23\) s to \(12.24\) s. This indicates that a domain-specific visual encoder can improve temporal representations when motion dynamics are especially salient.

LLM choice for prompt generation appears comparatively unimportant: Gemma-4B, Qwen3-4B, and Llama-3B differ by less than \(5\%\) on all metrics. By contrast, removing the event curve is consequential. A text-only baseline using the same text-to-music model and a video-derived text prompt but no event curve reduces SCH from \(0.61\) to \(0.35\), supporting the claim that text alone cannot enforce tight synchronization.

The implementation guidance emphasizes portability. The required components are any pretrained text-to-music rectified-flow model such as Stable-Audio-ControlNet, an audio autoencoder, off-the-shelf encoders such as MusicFM, DINOv2, or CoTracker, and an LLM for prompting. The event-curve module itself is reusable and framework-agnostic: extract features \(\mathbf f\), compute \(\{s^k\}\) and \(\{a^k\}\), standardize, resample to \(\ell\), and Hann-smooth. To adapt to new domains, the visual encoder can be swapped for an optical-flow or pose-tracking model.

Taken together, these findings support the paper’s main thesis that temporal alignment through within-modality features, rather than paired cross-modal supervision, is effective for video-to-music generation. This suggests that synchronized audiovisual generation need not rely on semantically shared latent spaces if the objective is primarily temporal coordination.

Source: https://www.emergentmind.com/topics/v2m-zero