V2M-Zero: Zero-Pair Video-to-Music Generation

• The paper introduces a zero-pair approach that achieves temporal synchronization by conditioning a text-to-music model on independently computed event curves.
• It computes intra-modal event curves using cosine dissimilarity, standardization, resampling, and Hann-window smoothing to capture moments of strong temporal change.
• Experimental results show improved audio fidelity, semantic alignment, and temporal accuracy compared to paired-data models across diverse benchmarks.

Searching arXiv for the specified paper to ground the article. V2M-Zero is a zero-pair video-to-music generation approach that outputs time-aligned music for video by conditioning a pretrained text-to-music model on an event curve derived from temporal change, rather than on paired cross-modal supervision. Its defining claim is that temporal synchronization depends not on what changes across modalities, but on when and how much change occurs; accordingly, it computes comparable temporal representations independently within music and video using intra-modal similarity, fine-tunes a text-to-music model on music-event curves, and substitutes video-event curves at inference time to produce synchronized soundtracks without paired video–music training data (Lin et al., 11 Mar 2026).

1. Conceptual basis

Existing text-to-music systems such as MusicLM, MusicGen, and AudioLDM produce high-quality music from text prompts but have no mechanism to follow a video’s fine-grained temporal structure. In that setting, creators must manually edit video to match generated music. V2M-Zero addresses the specific problem of temporal alignment rather than generic music generation (Lin et al., 11 Mar 2026).

The central observation is that visual and musical events need not match semantically in order to align temporally. Scene cuts, motion bursts, beat onsets, and instrumentation changes differ in modality and meaning, yet they may share a common temporal structure. V2M-Zero operationalizes this structure through an event curve: a one-dimensional signal that marks moments of strong temporal change. Because the curve is computed independently inside each modality, the method does not require paired video–music examples or explicit cross-modal representation learning.

A common misconception in this problem setting is that synchronized video-to-music generation must be trained with paired video–music data. V2M-Zero is explicitly constructed to test the opposite hypothesis. Its reported results indicate that within-modality temporal features can suffice for synchronization when semantic control is handled separately through text prompts. This suggests a decomposition between temporal control and semantic control rather than a single fused cross-modal objective.

2. Event curves from intra-modal similarity

V2M-Zero converts a sequence of high-level features into a 1D event curve that indicates moments of strong temporal change. The same procedure is used for music during training and for video during inference (Lin et al., 11 Mar 2026).

For music, a pretrained music encoder such as MusicFM extracts

$$\mathbf f_m \in \mathbb R^{d_m\times l_m},\quad \mathbf f_m^k\in\mathbb R^{d_m}$$

for $k=1,\dots,l_m$. For video, a pretrained visual encoder such as DINOv2 is applied on each frame and spatially pooled to obtain

$$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$

Given a generic feature sequence $\mathbf f\in\mathbb R^{d_f\times l_f}$, adjacent cosine similarities are computed as

$$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$

These are converted into a dissimilarity or novelty sequence,

$$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$

To make the resulting curves comparable across modalities despite different scales and sampling rates, the novelty sequence is standardized, resampled, and smoothed: $$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$ followed by resampling to length $l$, the number of audio latent frames, and Hann-window smoothing: $$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$ At training time this yields the music-event curve $\mathbf e_m$; at test time it yields the video-event curve $k=1,\dots,l_m$0. The default smoothing kernel is a Hann window of size $k=1,\dots,l_m$1, selected by ablation.

The methodological significance of this construction is that it avoids direct alignment of visual and musical semantics. Only the temporal profile of change is transferred across modalities. This is the technical mechanism underlying the “zero-pair” designation.

3. Base model and event-conditioned fine-tuning

V2M-Zero builds on a pretrained latent rectified-flow text-to-music model with a DiT backbone (Lin et al., 11 Mar 2026). An audio autoencoder converts stereo 44.1 kHz waveforms into continuous latents

$k=1,\dots,l_m$2

with $k=1,\dots,l_m$3 and $k=1,\dots,l_m$4 for 32 s.

The rectified-flow formulation defines a noising path

$k=1,\dots,l_m$5

with $k=1,\dots,l_m$6, and learns a velocity field $k=1,\dots,l_m$7 mapping from $k=1,\dots,l_m$8 to the data distribution. Without event-curve conditioning, the training objective is

$k=1,\dots,l_m$9

where $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$0 is the text embedding provided via cross-attention.

Event-curve conditioning is introduced by concatenating the 1D curve as an additional input channel: $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$1 Only the input projection of the DiT is extended from $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$2 to $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$3, adding approximately $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$4 K parameters. During fine-tuning, $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$5 is replaced by the music-event curve $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$6, giving

$$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$7

Fine-tuning is performed on approximately $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$8 K h of instrumental music with AdamW, learning rate $$\mathbf f_v \in \mathbb R^{d_v\times l_v},\quad \mathbf f_v^k\in\mathbb R^{d_v}.$$9, and classifier-free guidance with $\mathbf f\in\mathbb R^{d_f\times l_f}$0 text/event drop. Reported fine-tuning cost is $\mathbf f\in\mathbb R^{d_f\times l_f}$1–$\mathbf f\in\mathbb R^{d_f\times l_f}$2 GPU h on $\mathbf f\in\mathbb R^{d_f\times l_f}$3–$\mathbf f\in\mathbb R^{d_f\times l_f}$4 A100s, corresponding to $\mathbf f\in\mathbb R^{d_f\times l_f}$5–$\mathbf f\in\mathbb R^{d_f\times l_f}$6 days. Architecturally, the minimal parameter increase is notable: temporal control is introduced without redesigning the backbone or adding an explicit cross-modal module.

4. Zero-pair inference and prompt construction

At test time, no weights are changed; the music-event curve is simply replaced with the video-event curve computed by the same standardize $\mathbf f\in\mathbb R^{d_f\times l_f}$7 resample $\mathbf f\in\mathbb R^{d_f\times l_f}$8 Hann smooth pipeline (Lin et al., 11 Mar 2026). This direct substitution is the core inference-time mechanism of V2M-Zero.

Semantic conditioning is obtained from the input video through a multi-stage prompt-generation procedure. ASR with Whisper extracts a transcript $\mathbf f\in\mathbb R^{d_f\times l_f}$9. Sampled frames are captioned by a vision-LLM such as Gemma-4B to produce $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$0. An LLM, again exemplified by Gemma-4B, summarizes these captions into a visual summary $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$1. A final LLM prompt applied to $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$2 yields the music prompt $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$3. The appendix pseudocode is: $k=1,\dots,l_m$04

Sampling proceeds by solving the rectified-flow ODE

$$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$4

from $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$5 to $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$6 in $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$7 steps with classifier-free guidance. The decoded $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$8 waveform is then added to the video.

This inference design separates two sources of control. The text prompt $$s^k \;=\; \frac{\mathbf f^k \!\cdot\!\mathbf f^{k+1}} {\|\mathbf f^k\|\;\|\mathbf f^{k+1}\|}, \quad k=1,\dots,l_f-1.$$9 governs semantic attributes of the soundtrack, while $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$0 governs temporal alignment. A plausible implication is that the model can inherit semantic flexibility from a general text-to-music system while imposing synchronization through a lightweight temporal side channel.

5. Evaluation protocol and reported results

The reported evaluation uses three datasets: OES-Pub, MovieGenBench-Music, and AIST++ (Lin et al., 11 Mar 2026). OES-Pub is the ISMIR ’25 public split of OSSL and contains $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$1 public-domain movie clips of approximately $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$2 s with royalty-free music and human-annotated prompts. MovieGenBench-Music contains $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$3 video–music pairs of approximately $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$4 s spanning diverse content and includes music prompts. AIST++ contains $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$5 dance clips of approximately $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$6 s across $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$7 genres with ground-truth BPM.

The metric suite covers audio fidelity, semantic alignment, temporal synchronization, and dance-specific beat tracking. Audio fidelity is measured by Fréchet Audio Distance in VGGish space, with OES-Pub additionally using FAD* against an external music reference set. Semantic alignment is measured by CLAP score, defined as cosine similarity between text and generated music. Temporal synchronization includes Scene-Cut Hit, the fraction of scene cuts with at least one beat onset within $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$8 ms, and pairwise human preference on quality and synchronization. Dance-specific evaluation on AIST++ uses Beat Coverage Score, Beat Hit Score, $$a^k \;=\; 1 - s^k,\quad \mathbf A = [\,a^1,\dots,a^{l_f-1}\,].$$9, and Temporal Deviation.

On OES-Pub and MovieGenBench, V2M-Zero is reported to outperform supervised baselines including VidMuse, AudioX, GVMGen, and SONIQUE. On OES-Pub it achieves FAD* $$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$0, CLAP $$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$1, and SCH $$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$2, corresponding to $$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$3–$$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$4 lower FAD*, $$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$5–$$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$6 higher CLAP, and $$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$7–$$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$8 higher SCH relative to the cited paired-data baselines. On MovieGenBench it reports FAD $$\bar a^k \;=\;\frac{a^k-\mu(\mathbf A)}{\sigma(\mathbf A)}, \quad k=1,\dots,l_f-1,$$9, CLAP $l$0, and SCH $l$1. In human A/B tests with $l$2 votes, it wins approximately $l$3 on music quality and approximately $l$4 on synchronization.

On AIST++, V2M-Zero reports BCS $l$5, BHS $l$6, $l$7 $l$8, and TD $l$9 s, compared with a prior best $$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$0 of $$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$1 and TD of $$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$2 s. The abstract summarizes these outcomes as $$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$3–$$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$4 higher audio quality, $$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$5–$$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$6 better semantic alignment, $$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$7–$$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$8 improved temporal synchronization, and $$\mathbf e = \mathrm{Smooth}\bigl(\mathrm{Resample}(\bar{\mathbf A},\,l)\bigr)\;\in\;\mathbb R^{l}.$$9 higher beat alignment on dance videos.

6. Ablations, implementation considerations, and research significance

The ablation studies focus on modality-gap mitigation, encoder selection, domain adaptation, prompting, and the necessity of event-curve conditioning (Lin et al., 11 Mar 2026). For the smoothing kernel, larger Hann windows from $\mathbf e_m$0 to $\mathbf e_m$1 reduce domain shift and improve FAD, but excessive smoothing degrades SCH by flattening peaks; the best trade-off is reported at size $\mathbf e_m$2.

For encoder selection, the default pairing of MusicFM and DINOv2 yields the strongest overall alignment. The reported comparison is summarized below.

Music encoder + Video encoder	FAD* ↓	CLAP ↑ / SCH ↑
AVSiam	4.52	0.19 / 0.35
VAE + V-JEPA	5.13	0.18 / 0.41
VAE + DINOv2	4.77	0.16 / 0.31
MusicFM + V-JEPA	5.02	0.18 / 0.48
MusicFM + DINOv2	4.95	0.23 / 0.61

For dance-specific inputs, replacing DINOv2 with CoTracker improves AIST++ performance from BCS $\mathbf e_m$3 to $\mathbf e_m$4 and reduces TD from $\mathbf e_m$5 s to $\mathbf e_m$6 s. This indicates that a domain-specific visual encoder can improve temporal representations when motion dynamics are especially salient.

LLM choice for prompt generation appears comparatively unimportant: Gemma-4B, Qwen3-4B, and Llama-3B differ by less than $\mathbf e_m$7 on all metrics. By contrast, removing the event curve is consequential. A text-only baseline using the same text-to-music model and a video-derived text prompt but no event curve reduces SCH from $\mathbf e_m$8 to $\mathbf e_m$9, supporting the claim that text alone cannot enforce tight synchronization.

The implementation guidance emphasizes portability. The required components are any pretrained text-to-music rectified-flow model such as Stable-Audio-ControlNet, an audio autoencoder, off-the-shelf encoders such as MusicFM, DINOv2, or CoTracker, and an LLM for prompting. The event-curve module itself is reusable and framework-agnostic: extract features $k=1,\dots,l_m$00, compute $k=1,\dots,l_m$01 and $k=1,\dots,l_m$02, standardize, resample to $k=1,\dots,l_m$03, and Hann-smooth. To adapt to new domains, the visual encoder can be swapped for an optical-flow or pose-tracking model.

Taken together, these findings support the paper’s main thesis that temporal alignment through within-modality features, rather than paired cross-modal supervision, is effective for video-to-music generation. This suggests that synchronized audiovisual generation need not rely on semantically shared latent spaces if the objective is primarily temporal coordination.