AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization

Abstract: Audio-visual feature extraction is a fundamental component of multimodal understanding and generation tasks. However, existing evaluation protocols for feature extraction models exhibit dimensional bias, typically focusing on either semantic matching or temporal offset detection. Moreover, their data construction remains coupled, preventing independent assessment of temporal and semantic consistency. We propose AV-SyncBench, the first benchmark to fully separate temporal and semantic evaluation for audio-visual synchronization. Built from in-the-wild videos, it spans Voice, Music, and Sound across 10 scenarios and 5 challenge tasks. Data are automatically filtered and manually verified to ensure on-screen sound sources. The benchmark contains 3,269 videos and 38,390 samples, and we evaluate five representative models to quantify feature quality for alignment and downstream tasks. The code and dataset are available at: https://fgt7t6g.github.io/AV-SyncBench.

• The paper introduces a novel benchmark, AV-SyncBench, that decouples temporal and semantic evaluations to diagnose model performance.
• It leverages controlled perturbations and semantic edits on curated in-the-wild videos to isolate specific synchronization challenges.
• Empirical results reveal that while models excel in either fine-grained temporal or semantic tasks, none achieve high accuracy across both.

Decoupled Temporal and Semantic Audio-Visual Synchronization Benchmarking with AV-SyncBench

Motivation and Problem Statement

Audio-visual feature extraction underpins a wide spectrum of multimodal understanding and generation tasks, requiring models to simultaneously parse high-level semantic correspondence and fine-grained temporal alignment. Existing evaluation protocols for such feature extraction are fundamentally limited by their conflation or neglect of these two axes. Most widely-adopted benchmarks exhibit dimensional bias, prioritizing global semantic retrieval tasks or coarse offset detection. These limitations are exacerbated by coupled data construction, preventing the independent, quantitative assessment of feature extractors' semantic and temporal modeling capacities.

This paper introduces AV-SyncBench (2607.00726), the first benchmark to fully decouple temporal and semantic evaluation for audio-visual synchronization. It enables rigorous, controlled analysis of the capabilities and deficiencies of state-of-the-art models over independently designed temporal and semantic challenge sets, providing actionable diagnostic granularity.

AV-SyncBench Framework

The AV-SyncBench pipeline (Figure 1) is built from carefully curated in-the-wild videos spanning the domains of Voice, Music, and generic Sound across 10 annotated scenarios. Data collection emphasizes stringent manual verification to guarantee on-screen, temporally aligned sources, removing off-screen or ambiguous samples.

For each annotated video-audio pair, the benchmark systematically generates two independent challenge sets:

• Temporal: Controlled perturbations (global offset, local jitter, global speed change) are introduced to the audio stream, strictly preserving its semantic content while varying temporal alignment with the video.
• Semantic: Advanced generative editing (OpenVoice V2 and DDSP-based timbre transfer) is applied to modify audio semantic attributes such as timbre or source identity, while precisely maintaining the original audio-video timing.

The framework operates under a unified segmentation protocol (0.64 s non-overlapping chunks), extracting audio and visual embeddings per chunk and quantifying synchronization via diagonal cosine similarities. Binary accuracy is computed by pairwise comparison between original and perturbed/edited samples.

Figure 1: Schematic of AV-SyncBench's decoupled evaluation process, independently generating temporal and semantic challenge sets from in-the-wild, annotated video-audio pairs and evaluating models via synchronization metric.

Dataset Construction and Challenge Design

The dataset comprises 3,269 filtered high-quality video clips (38,390 challenge samples), each subjected to:

• Temporal perturbations (three subtypes):
  • Global Offset: Stepwise shifts (50–500 ms) of the entire audio signal
  • Local Jitter: Randomized local shifts (30–700 ms) of short windows
  • Global Speed Change: Uniform playback rate adjustments (0.8–1.25×)
• Semantic editing:
  • Voice: OpenVoice V2 swaps timbres (by age/gender) without rhythm alteration
  • Music: DDSP models perform instrument replacement while preserving note/spectral onsets
• All semantic manipulations maintain the physical timing, isolating model sensitivity to purely semantic information.

This isolation of factors is critical for disentangling which embedding properties contribute to various downstream multimodal tasks.

Evaluation Protocol and Models

All audio-visual models are evaluated out-of-the-box using officially released weights and protocols: Synchformer, SparseSync, CAV-MAE Sync, ImageBind, and CAV-MAE. Binary accuracy is the criterion, measuring whether synchronization scores for the unperturbed (or semantically matched) pair outscore the corresponding altered pair.

Empirical Results and Analysis

Temporal Sensitivity

Temporal perturbation detection remains challenging for all models, particularly for fine-grained global offset. Accuracy improves monotonically with perturbation magnitude, but overall performance on small shifts (<<100 ms) is near chance, indicating that continuous embeddings lack the temporal granularity required for subtle shift detection. Notably, Synchformer outperforms others on offset detection, while SparseSync complements it on local jitter and global speed change, illustrating a training-induced bias towards temporal objectives. In contrast, CAV-MAE, reflective of its contrastive learning design, largely ignores micro-temporal structure.

Model performance varies substantially by scenario: single-source, visually explicit actions (e.g., solo instrumental, narration) yield higher scores compared to ensemble or multi-source blends, highlighting the role of visually grounded audio signals in strong temporal alignment.

Semantic Discrimination

The semantic challenge, exemplified by timbre and instrument replacement, exposes the inability of several temporally-focused models to capture deep semantic differences. ImageBind achieves the highest average accuracy in timbre changes, particularly for voice and singing (up to 93% in single and multi-speaker). CAV-MAE Sync, in contrast, nears random guessing under semantic perturbations—direct evidence of its inattention to semantic details. CAV-MAE (non-synchronized variant) achieves robust accuracy in instrument recognition tasks, confirming its contrastive semantic alignment training objective.

This dichotomy between “temporal” and “semantic” models is stark: models sensitive to timing distortions are notably less robust in discriminating acoustic attribute transfer under fixed timing, and vice-versa. There is no current model architecture that achieves optimality on both axes, underscoring a major shortcoming in the current paradigm for universal audio-visual representation learning.

Limitations

While AV-SyncBench marks a clear advance in benchmarking methodology, limitations remain. The current semantic editing operations are restricted by the available generative models and are limited to speech/music domains, with less coverage for object or ambient sound source replacement. Subtle acoustic artifacts may be introduced by editing pipelines, which, although minor, complicate the absolute disambiguation of semantic manipulations. The dataset also limits context to short videos (<13 s), not fully exploring long-range temporal dependencies or highly complex multi-source situations.

Implications and Future Directions

AV-SyncBench provides strong diagnostic evidence that current self-supervised and contrastive training regimes produce representations fundamentally biased by their objective functions. Models excel at either micro-temporal synchronization or semantic embedding, but not both jointly, which restricts the ceiling of downstream multimodal generation and analysis systems. There is strong evidence supporting the necessity for new learning strategies, loss architectures, and training data schedules that simultaneously enforce both axes of alignment, such as dual-headed or multi-task training objectives.

Practically, the benchmark enables consistent model selection and ablation in evolving real-world use cases: controllable AV generation, multimodal retrieval, automatic dataset filtering/curation for highly synchronous data, and cross-modal scene understanding.

Conclusion

AV-SyncBench (2607.00726) defines a new rigorous protocol for the decoupled evaluation of audio-visual synchronization models, revealing significant gaps in the joint semantic and temporal fidelity of state-of-the-art systems. The empirical results demonstrate that no current feature extractor simultaneously achieves high accuracy on both axes, motivating a shift towards architectures and training approaches that unify fine-grained temporal and semantic representation learning. The availability of AV-SyncBench and its diagnostic analysis can be expected to set a practical and theoretical agenda for future research in general-purpose, high-fidelity multimodal learning.