SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning

Abstract: Recent advancements in video-audio joint generation have achieved remarkable success in semantic correspondence. However, achieving precise temporal synchronization, which requires fine-grained alignment between audio events and their visual triggers, remains a challenging problem. The post-training method for joint generation is largely dominated by Supervised Fine-Tuning, but the commonly used Mean Squared Error loss provides insufficient penalties for subtle temporal misalignments. Direct Preference Optimization offers an alternative by introducing explicit misaligned counterparts to better improve temporal sensitivity. In this paper we propose a post-training framework SyncDPO, leveraging DPO to improve the temporal sensitivity of V-A joint generation. Conventional DPO pipelines typically depend on costly sampling-and-ranking procedures to construct preference pairs, resulting in substantial computational cost. To improve efficiency, we introduce a suite of on-the-fly rule-based negative construction strategies that distort temporal structures without incurring additional annotation or sampling. We demonstrate that the temporal alignment capability can be effectively reinforced by providing explicit negative supervision through temporally distorted V-A pairs. Accordingly, we implement a curriculum learning strategy that progressively increases the difficulty of negative samples, transitioning from coarse misalignment to subtle inconsistencies. Extensive objective and subjective experiments across four diverse benchmarks, ranging from ambient sound videos to human speech videos, demonstrate that SyncDPO significantly outperforms other methods in improving model's temporal alignment capability. It also demonstrates superior generalization on out-of-distribution benchmark by capturing intrinsic motion-sound dynamics. Demo and code is available in https://syncdpo.github.io/syncdpo/.

• The paper introduces SyncDPO, which uses direct preference optimization to tackle the challenge of fine-grained temporal misalignment in video-audio joint generation.
• It employs on-the-fly rule-based negative sample construction—via scaling, replacing, shifting, masking, and synthesizing—to generate informative supervision for synchronization.
• A curriculum learning strategy is integrated to progressively expose the model to subtler perturbations, yielding superior performance on both in-domain and out-of-domain benchmarks.

SyncDPO: Preference-Based Enhancement of Temporal Synchronization in Video-Audio Joint Generation

Background and Motivation

High-quality multimodal generation, particularly the joint synthesis of video and audio (V-A), has attained substantial progress when measured by semantic consistency and cross-modal coherence. However, precise temporal synchronization—aligning the timing of generated audio events with their corresponding visual triggers—remains a critical challenge, especially for human-speech and event-triggered ambient sound videos. State-of-the-art industrial (e.g., Sora, Veo, Seedance) and academic frameworks predominantly utilize Supervised Fine-Tuning (SFT) with Mean Squared Error (MSE) loss. This protocol tends to provide insufficient signal for fine-grained temporal alignment due to its weak penalty for subtle misalignments, often failing to prevent perceptible drifts between audio and visual events.

The authors address these limitations by proposing SyncDPO, a framework leveraging Direct Preference Optimization (DPO) with cost-efficient, on-the-fly rule-based negative construction and a curriculum learning schedule to induce robust temporal discrimination during post-training. The method is validated on multiple V-A benchmarks with attention to in-domain and out-of-domain generalization.

Methodology

Rationale for Preference Optimization

SFT with MSE loss provides only weak supervision for temporal alignment, excelling at global feature reconstruction but failing at penalizing minor, yet critical, temporal deviations. In contrast, preference-based optimization—specifically DPO—directly contrasts synchronized (positive) and intentionally desynchronized (negative) V-A pairs, generating informative supervision for temporal correspondence. As evidenced by gradient norm analysis, DPO variants induce stronger discrepancy signals than SFT, accelerating convergence and enhancing the model’s temporal sensitivity.

Figure 1: DPO-based SyncDPO achieves higher gradient signal strength and efficiency compared to SFT and conventional DPO, due largely to tailored negative sample generation.

On-the-Fly Negative Construction

Traditional DPO pipelines incur high computational and annotation overhead—often requiring repeated sampling, ranking, or manual preference collection. SyncDPO proposes a scalable rule-based approach for generating temporally misaligned negatives on demand:

• Scaling: Temporally stretches or compresses video/audio to create speed inconsistencies.
• Replacing: Substitutes either the video or audio stream with another sample, ensuring semantic and temporal discord.
• Shifting: Applies random offsets simulating synchronization errors.
• Masking: Silences or freezes part of one modality.
• Synthesizing: Replaces with reference model-generated content.

Empirical results identify Scaling and Replacing as the most effective perturbations for supervision, providing strong and diverse negative examples while avoiding destructive alterations that harm semantic integrity.

Figure 2: Schematic illustration of rule-based perturbations for constructing temporally misaligned negative pairs during training.

Curriculum Learning Strategy

The effectiveness of DPO is further enhanced via curriculum learning. During training, SyncDPO gradually transitions from easily discriminable negatives (Replacing) to more subtle, harder cases (Scaling). The sampling probability between these perturbations is dynamically scheduled:

$$\begin{aligned} p_{\text{replace}}(t) = 0.5 - kt, \ p_{\text{scale}}(t) = 1 - p_{\text{replace}}(t), \end{aligned}$$

where $k$ controls the transition rate. This progressive exposure allows the model to first learn coarse temporal boundaries before mastering fine-grained synchronization. Ablations demonstrate that this curriculum yields superior lip-speech and event-sound alignment compared to single or uniformly mixed perturbation strategies.

Experimental Results

SyncDPO is benchmarked on four datasets—LRS2 (lip-speech), AVSync and GreatestHits (event-sound), and VABench (OOD generalization)—with both in-domain and out-of-domain (Koala) training. Evaluation employs domain-standard synchronization metrics (LSE-D, LSE-C, DeSync), as well as audio, video, and semantic quality metrics.

• Synchronization: SyncDPO consistently achieves the best LSE-D, LSE-C, and DeSync scores under all conditions. For example, on LRS2, SyncDPO outperforms SFT and vanilla DPO on both in-domain and OOD data, indicating sharper lip-speech and event-sound synchronization.
• Generalization: Unlike SFT, which sometimes degrades semantic alignment on OOD tests due to overfit, SyncDPO sustains both temporal and semantic fidelity.
• Subjective Evaluation: Blind human preference tests show SyncDPO is consistently favored for temporal alignment, while maintaining parity in audio and video quality.

  Figure 3: Human evaluators consistently prefer SyncDPO over SFT/vanilla DPO for temporal alignment across diverse benchmarks in both ambient sound and speech scenarios.

SyncDPO’s gains are robust to variations in perturbation scheduling rate and DPO regularization strength, with optimal performance at $k=0.01$ and $\beta=0.2$.

Implications and Future Directions

SyncDPO establishes a practical, scalable post-training paradigm for augmentation of fine-grained V-A temporal synchronization without resorting to labor-intensive sampling or annotation. The explicit construction of temporally misaligned negative samples serves as an efficient supervisory signal, complementing or surpassing existing reconstruction-based and reward-model-based methods. Its generalization performance on OOD data highlights potential as a universal synchronization enhancement module across diverse domains.

This framework opens several avenues for investigation:

• Scalability: Extension to longer, more complex videos and multi-source audio mixing scenarios.
• Granularity: Fine-tuning perturbation schedules or learning adaptive negative construction.
• Integration: Use in foundation models for open-domain V-A synthesis, dubbing, or cross-modal editing.

Continued improvement in synchronization is likely to impact a wide range of content creation tasks—including dubbing, sound design, and AI-assisted video production—while simultaneously heightening the necessity for ethical safeguards against misuse.

Conclusion

SyncDPO presents a principled, efficient method for post-training enhancement of temporal synchronization in joint video-audio generative models (2605.12179). By combining on-the-fly rule-based negative construction and curriculum-guided preference optimization within a DPO framework, it achieves superior temporal alignment across benchmarks, robust generalization, and efficient training characteristics. The approach delineates a transferable best practice for future multimodal generation models seeking fine-grained cross-modal alignment.