Syncphony: Audio-Driven Video Generation

• Syncphony is an audio-to-video generation framework that employs a diffusion transformer backbone to produce 380×640, 24 fps videos with fine-grained audio-motion alignment.
• It integrates Motion-aware Loss and Audio Sync Guidance to emphasize dynamic regions and accurately amplify audio-driven motion during video synthesis.
• The system introduces CycleSync, a novel metric that evaluates the reconstructive quality of temporal audio cues in generated videos.

Syncphony is an audio-to-video generation framework that uses a diffusion transformer backbone to produce high-quality, temporally coherent videos whose motion is tightly synchronized to input audio. It generates $380\times 640$ resolution, 24 fps videos synchronized with diverse audio inputs, and targets fine-grained synchronization rather than merely plausible correlation between sound and motion. The system builds on a pre-trained video backbone and adds two synchronization-specific components—Motion-aware Loss and Audio Sync Guidance—together with a new evaluation metric, CycleSync, to improve synchronization accuracy while maintaining visual quality (Song et al., 26 Sep 2025).

1. Problem setting and design objective

Audio-to-video generation in Syncphony is framed as the synthesis of a video sequence conditioned on an audio signal, optionally with text and an initial image, such that visual motion is semantically meaningful and temporally aligned with the audio. The motivating observation is that text-to-video and image-to-video systems can describe what happens, but are weak at controlling when motion occurs. Audio, by contrast, provides temporal cues aligned with motion and therefore offers a direct route to controlling the onset, offset, rhythm, and intensity of motion events (Song et al., 26 Sep 2025).

The framework is positioned against several limitations in prior audio-conditioned video generation. Amplitude modulation approaches use audio amplitude to modulate cross-attention weights but are too coarse for fine-grained synchronization. Audio-to-text projection approaches map audio embeddings into text space and then rely on text-to-video backbones, which weakens temporal expressiveness. Audio-conditioned extensions of image diffusion backbones have been trained at low frame rate and resolution, which limits temporal modeling capacity and introduces artifacts such as flicker and saturation. Talking-head lip-sync systems achieve precise synchronization for speech and facial motion, but they do not address general non-speech sounds and scene-level motion (Song et al., 26 Sep 2025).

A common misconception is that audio conditioning alone is sufficient to induce precise synchronization. Syncphony explicitly rejects that assumption. Its central claim is that synchronization requires architectural access to temporally aligned audio features, a supervision signal that emphasizes motion regions, and an inference-time mechanism that amplifies the audio-driven component of the model’s denoising trajectory (Song et al., 26 Sep 2025).

2. Backbone architecture and audio conditioning

Syncphony is built on Pyramid Flow Video, an autoregressive diffusion transformer for video. The backbone uses a flow-matching objective over spatiotemporal latents, 3D Rotary Positional Encoding across time, height, and width, and resolution scheduling that uses lower resolution at early timesteps and higher resolution at later timesteps. Pyramid Flow consists of 24 transformer blocks, and Syncphony fine-tunes only the later 16 blocks, indexed 8–23, where motion refinement is concentrated (Song et al., 26 Sep 2025).

The conditioning stack combines an initial frame, a text prompt, and an audio waveform. The initial frame is encoded by a VAE, text is encoded by CLIP or T5-style encoders, and audio is encoded by DenseAV. Audio features are not injected globally and uniformly; instead, each video latent $z_l$ is associated with a local audio segment

$$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$

with $\Delta = 1$ in the reported setting. This local alignment ensures that each video latent attends to audio features from a narrow temporal neighborhood centered on the corresponding frame position (Song et al., 26 Sep 2025).

Audio is incorporated through cross-attention inserted before the joint self-attention in the later transformer blocks. For a video latent $z_l$ and audio features $a_i \in A_l$,

$$q_l = W_Q z_l,\qquad k_i = W_K a_i,\qquad v_i = W_V a_i.$$

Syncphony then applies Audio RoPE to queries and keys so that video latents receive 3D coordinates $(l,h_l,w_l)$, while aligned audio features receive interpolated temporal positions $t_i \in [l-(\Delta+0.5),\, l+(\Delta+0.5)]$. This creates a shared temporal reference frame between audio and video streams (Song et al., 26 Sep 2025).

The architectural logic is summarized below.

Component	Role	Stated effect
Pyramid Flow Video	Pre-trained video backbone	Supports high resolution and frame rate
Motion-aware Loss	Upweights errors in high-motion regions	Improves learning at high-motion regions
Audio Sync Guidance	Uses a visually aligned off-sync model	Better exploits audio cues at inference while maintaining visual quality
CycleSync	Video-to-audio-based metric	Measures motion cues sufficient to reconstruct the original audio

This design preserves the visual prior of the pre-trained backbone while restricting audio adaptation to motion-sensitive layers. A plausible implication is that Syncphony treats appearance preservation and synchronization as partially separable subproblems: the earlier blocks retain scene fidelity, whereas the later blocks absorb temporally aligned audio control (Song et al., 26 Sep 2025).

3. Synchronization-specific learning and inference mechanisms

Syncphony introduces Motion-aware Loss to address a weakness of uniform latent-space reconstruction losses. Standard mean-squared error treats static and dynamic regions equally, which allows synchronization failures in high-motion segments to be under-penalized. Syncphony instead defines motion magnitude from adjacent ground-truth latent differences,

$m_l = z^{GT}_l - z^{GT}_{l-1},$

and optimizes

$z_l$0

with $z_l$1. The second term amplifies errors where ground-truth motion is large, thereby emphasizing motion onset, offset, and magnitude rather than allowing static regions to dominate the gradient signal (Song et al., 26 Sep 2025).

The second mechanism is Audio Sync Guidance, an inference-time procedure built from two branches that share the same visual backbone. The full branch uses audio cross-attention, while the “off-sync” branch skips audio cross-attention layers. The off-sync model remains visually aligned because it uses the same image and text conditioning, but it is desynchronized with respect to audio. Syncphony then defines a guided denoising prediction

$z_l$2

so that the difference between the full and off-sync predictions acts as an estimate of the audio-driven synchronization component (Song et al., 26 Sep 2025).

Audio Sync Guidance is combined additively with Classifier-Free Guidance:

$z_l$3

with $z_l$4 and $z_l$5 in the reported experiments. The paper also reports that excessively large $z_l$6 can over-amplify motion and degrade realism, indicating a direct trade-off between synchronization strength and visual stability (Song et al., 26 Sep 2025).

Together, Motion-aware Loss and Audio Sync Guidance define Syncphony’s central synchronization strategy. The former shapes training gradients around dynamic regions; the latter amplifies audio-conditioned deviations from a visually comparable but desynchronized baseline at inference.

4. CycleSync and empirical evaluation

Syncphony introduces CycleSync as a video-to-audio-based synchronization metric. Rather than assuming one-to-one correspondence between motion peaks and audio peaks, CycleSync asks whether the generated video contains enough motion cues to reconstruct the temporal structure of the original audio. A pretrained video-to-audio model $z_l$7, instantiated with V-AURA, reconstructs audio $z_l$8 from generated video $z_l$9. Peak detectors are applied to the original and reconstructed audio,

$$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$0

and CycleSync is computed with tolerance $$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$1 ms as

$$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$2

This metric is intended for 24 fps, real-world audio-motion scenarios in which preparatory motion and residual motion make peak matching insufficient (Song et al., 26 Sep 2025).

Experiments are reported on AVSync15 and TheGreatestHits. AVSync15 is a curated subset of VGGSound with 1,500 videos across 15 action-related classes, split into 1,350 training and 150 test videos. TheGreatestHits contains videos of a person striking various objects with drumsticks, with 733 training and 244 test videos. Training uses random clip sampling, 4 RTX 3090 GPUs, batch size 32, about 33k steps, and about 34 hours of training time. Sampling uses 30 denoising steps (Song et al., 26 Sep 2025).

On AVSync15, Syncphony improves both synchronization and visual quality relative to AVSyncD. Reported numbers include FID 8.5 versus 9.2, FVD 293.1 versus 491.5, Image-Audio similarity 37.02 versus 35.23, and CycleSync $$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$3 versus $$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$4. On TheGreatestHits, Syncphony reports FID 6.7 versus 6.8, FVD 166.2 versus 327.8, Image-Audio similarity 13.83 versus 12.35, and CycleSync $$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$5 versus $$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$6. In that benchmark, CycleSync slightly exceeds the reported ground-truth value $$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$7, which the paper attributes to the fact that ground-truth videos include off-event motion and background noise (Song et al., 26 Sep 2025).

The user study on AVSync15 reports preference for Syncphony over AVSyncD on synchronization in 74% of pairwise judgments, on image quality in 90%, and on frame consistency in 94%. CycleSync also shows the strongest reported positive correlation with human scores, with Pearson $$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$8, whereas earlier synchronization metrics are described as weakly correlated or even negatively correlated with human judgments in this setting (Song et al., 26 Sep 2025).

5. Relation to prior audio-to-video paradigms

Syncphony differs from amplitude-modulation systems such as AADiff, from audio-to-text projection systems such as TPoS and TempoTokens, and from image-diffusion-based audio-conditioned systems such as AVSyncD. The distinction is not merely one of scale; it is architectural. Syncphony directly injects temporally aligned audio features into a high-capacity video diffusion transformer via cross-attention, rather than reducing audio to amplitude envelopes or projecting it into a text-conditioned bottleneck (Song et al., 26 Sep 2025).

This distinction is important because synchronization in Syncphony is modeled as a property of spatiotemporal latent dynamics rather than as a side effect of semantic conditioning. Audio is aligned to video time through local segment selection and Audio RoPE; motion supervision is explicitly biased toward high-motion latent regions; and inference-time guidance separates audio-dependent motion from visually plausible but audio-agnostic motion. This suggests a more direct operational definition of synchronization than in earlier systems: synchronization is whatever component of the denoising trajectory disappears when audio layers are disabled (Song et al., 26 Sep 2025).

The paper also challenges the idea that “busy” motion is a sufficient indicator of synchronization quality. Existing metrics such as RelSync, AlignSync, and AV-Align are reported to mis-rank audio-independent models in some cases. CycleSync is proposed precisely to avoid rewarding generic motion that is not reconstructive of the input audio’s temporal structure (Song et al., 26 Sep 2025).

6. Limitations, applications, and broader implications

Syncphony is presented as suitable for music videos, dance- or beat-driven animation, impact-driven visuals, and general audio-reactive video. Its output regime is $$A_l = \{ a_i \mid i \in [\alpha(l-\Delta), \alpha(l+\Delta)] \},$$9, 24 fps, up to 5 seconds long, and the paper states that it can complement lip-sync systems by adding body or scene motion synchronized with non-speech sounds (Song et al., 26 Sep 2025).

The system is computationally nontrivial. Reported inference times for a 5-second video are about 2m53s with Audio Sync Guidance, about 2m01s without Audio Sync Guidance, and about 1m43s without audio layers; at least 16 GB of GPU memory is required for 5-second videos. These figures indicate that Syncphony is designed for quality-oriented generation rather than low-latency interactive deployment (Song et al., 26 Sep 2025).

Several limitations are stated explicitly. Motion-aware Loss uses motion intensity as a proxy for important synchronization targets and therefore may overemphasize background or semantically irrelevant motion. CycleSync depends on a pretrained video-to-audio model and inherits its biases. Very subtle audio events, noisy environments, long-horizon narrative consistency, and object-specific motion grounding remain difficult. The paper also notes ethical concerns common to generative video systems, including fraud, impersonation, politically manipulative content, and hate speech, and points to prompt filtering and blocking of impersonation or political content as possible safeguards (Song et al., 26 Sep 2025).

Taken together, Syncphony defines a specific technical program for audio-conditioned video generation: a strong video diffusion transformer backbone, direct temporally aligned audio cross-attention, motion-sensitive supervision, inference-time audio-guided denoising, and evaluation through reconstructive synchronization. Within that program, its main contribution is not simply that audio conditions video generation, but that synchronization is elevated to a first-class modeling target.