DualBench: Video-to-Soundtrack Evaluation

• DualBench is a benchmark for video-to-soundtrack evaluation that jointly assesses dubbed speech and background audio generation using a curated animated test set.
• It provides separated speech and background references with transcripts and employs advanced metrics like DualScore and AV-Align for comprehensive analysis.
• The benchmark supports detailed ablation studies, highlighting the impact of curriculum learning, cross-modal alignment, and decoder strategies on synthesis quality.

Searching arXiv for the specified papers and closely related benchmark work. DualBench is a benchmark for video-to-soundtrack (V2ST) evaluation, introduced as the first open benchmark tailored to the joint generation of speech and background audio for a given video. It was designed to address a gap in prior video-to-audio evaluations, which predominantly assessed background sound realism and synchronization while not accounting for speech intelligibility, speaker identity consistency, or the acoustic harmony between speech and non-speech audio. DualBench operationalizes V2ST as the generation of two synchronous audio streams from visual content: dubbed speech aligned to provided transcripts and speaker prompts, and background audio aligned to visual events. Its central contribution is a curated test set with separated speech and background references together with a metric suite spanning generation quality, audio-video alignment, and audio-speech harmony (Tian et al., 14 Jul 2025).

1. Definition, motivation, and task framing

DualBench was introduced in the context of "DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis" (Tian et al., 14 Jul 2025). The benchmark targets the evaluation of complete soundtracks rather than background-only audio. In this formulation, the generated speech is evaluated with respect to the provided transcript, perceptual quality, and speaker similarity to a reference prompt, while the generated background audio is evaluated for realism, diversity, and temporal correspondence with visual events. A further dimension, specific to DualBench, is acoustic harmony between speech and background audio: whether the two generated streams form a coherent, non-conflicting mix in rhythm, energy, and timbre.

The benchmark focuses on videos where both speech and background audio are present and relevant. The authors specifically target animation-style content derived from V2C-Animation. This setting stresses lip and speech rhythms, foreground dialogue, and diverse background sound effects. The benchmark is therefore not simply a repackaging of earlier video-to-audio evaluations. It is structured around the premise that a V2ST system must synthesize the "whole soundtrack," and that evaluation must therefore jointly address background realism, speech intelligibility, cross-modal synchronization, and cross-track coherence (Tian et al., 14 Jul 2025).

A common misconception is to treat DualBench as a generic name for any benchmark involving two components or two controllers. In the literature provided here, DualBench specifically denotes the V2ST benchmark introduced with DualDub. By contrast, $\tau^2$-Bench is a dual-control conversational benchmark rather than a video-audio benchmark, and DuoBench is a bimanual manipulation benchmark with a different scope and official name (Barres et al., 9 Jun 2025, Jülg et al., 10 Jun 2026).

2. Dataset curation and benchmark composition

DualBench’s test set is constructed from the test split of the V2C-Animation dataset. The curation pipeline has three explicit stages: audio channel normalization by merging multi-channel audio to mono; speech/background separation using Mel-RoFormer; and energy-based filtering that discards pairs if either separated stream has energy lower than $-40$ dB, a threshold described as empirically chosen. The resulting benchmark contains 1,319 curated clips from the original 2,793 test clips, with an average duration of 2.65 seconds (Tian et al., 14 Jul 2025).

Several benchmark properties follow directly from this construction. First, each retained clip contains both speech and background audio, and the separated streams are available as references. Second, transcripts are available through inheritance from the source dataset, since speech word error rate is evaluated against the provided transcript. Third, synchronization ground truth is not provided as explicit event labels; instead, synchronization is assessed against the video content using AV-Align. DualBench is presented as a curated test-only benchmark rather than as a train/validation/test suite (Tian et al., 14 Jul 2025).

The benchmark’s data provenance also matters for interpretation. The domain is animation, described as cartoon or anime-like, and this domain emphasizes distinctive prosody patterns and rich Foley effects. A plausible implication is that benchmark scores should not be assumed to transfer directly to live-action or documentary settings. The paper itself notes domain bias and reports in-domain versus out-of-domain performance differences, indicating that DualBench is best understood as a focused testbed rather than a universal measure of soundtrack generation (Tian et al., 14 Jul 2025).

In addition to the test set, the benchmark includes an evaluation model called DualScore, trained using audio-speech pairs. The supporting corpora listed for DualDub and DualBench include approximately 100 hours of animation data, approximately 5 hours of V2C-Animation, and approximately 100 hours of private multimodal pairs, while approximately 1500 hours of private audio-speech pairs are used for DualBench’s CASP-based evaluation model only. The paper states that all corpora marked with an asterisk were separated and filtered to remove speech leakage from background audio (Tian et al., 14 Jul 2025).

3. Evaluation protocol and metric structure

DualBench evaluates three principal dimensions: generation quality, audio-video alignment, and audio-speech harmony. It also includes subjective human listening studies. The objective metric suite is heterogeneous by design, combining established audio metrics, ASR-based speech metrics, and a learned cross-track harmony metric (Tian et al., 14 Jul 2025).

Dimension	Metrics	Brief role
Background audio quality	FD, FAD, IS, KL	realism, diversity, distributional similarity
Speech quality and intelligibility	WER, SIM, UTMOS	transcript accuracy, speaker similarity, perceptual quality
Synchronization and harmony	AV-Align, DualScore	video alignment and cross-track coherence

For background audio, DualBench uses the audioldm-eval toolkit. Fréchet Distance is computed on PANN features, Fréchet Audio Distance on VGGish embeddings, and Inception Score and Kullback–Leibler divergence are computed as in audioldm-eval. The paper does not provide explicit formulas for these metrics, instead referencing the toolkit and feature encoders used (Tian et al., 14 Jul 2025).

For speech, word error rate is computed with Whisper-large-v3 against the provided transcript, speaker similarity is computed using a WavLM-based speaker verification model, and UTMOS is estimated via SpeechMOS. For audio-video synchronization, AV-Align detects peaks independently in audio and video modalities and computes Intersection over Union between the detected peak regions; the paper summarizes the method but does not provide a formal equation (Tian et al., 14 Jul 2025).

DualBench’s distinctive metric is DualScore, a learned measure of audio-speech harmony based on Contrastive Audio-Speech Pretraining (CASP). CASP uses a dual-branch encoder: the audio branch uses a pre-trained BEATs encoder, while the speech branch uses the same network architecture but is trained from scratch to adapt speech-specific representations. An attention pooling layer aggregates features, and after training DualScore is defined as cosine similarity between the speech and audio embeddings:

$$\mathrm{DualScore}=\cos(f_{\mathrm{audio}},f_{\mathrm{speech}})=\frac{f_{\mathrm{audio}}\cdot f_{\mathrm{speech}}}{\|f_{\mathrm{audio}}\|\|f_{\mathrm{speech}}\|}.$$

On retrieval validation over the 1,319-pair V2ST test set, the paper reports Top-1 accuracy 70%, Top-3 90%, and Top-5 95%, with a heatmap showing diagonal concentration of correct matches (Tian et al., 14 Jul 2025).

Human evaluation complements the automatic metrics. DualBench uses 20 raters, comprising 10 non-expert listeners and 10 with professional audio knowledge. Ratings are collected on a 5-point scale for overall audio quality (OAQ), overall speech quality (OSQ), and audio-speech correspondence and harmony (ASCH), using 16 sample pairs and reporting means with 95% confidence intervals (Tian et al., 14 Jul 2025).

4. Comparative results and empirical use

DualBench is used to compare video-to-audio-only baselines, video-to-speech-only baselines, concatenated V2A+V2S pipelines, and the joint V2ST model DualDub. In background-audio evaluation on DualBench, DualDub reports FD 23.80, FAD 5.86, KLD 1.79, IS 2.72, and AV-Align 0.25. The reported baselines are MMAudio with FD 38.15, FAD 9.44, KLD 2.20, IS 2.93, AV-Align 0.23; FoleyCrafter with FD 25.18, FAD 4.60, KLD 1.84, IS 2.45, AV-Align 0.23; and Diff-Foley with FD 41.82, FAD 6.24, KLD 2.78, IS 5.20, AV-Align 0.20. The paper summarizes this as DualDub attaining the best FD and AV-Align with competitive FAD and KL (Tian et al., 14 Jul 2025).

For speech generation, DualDub reports WER 12.74%, SIM 0.84, and UTMOS 2.70. The cited baselines are HPMDubber with WER 20.96%, SIM 0.74, UTMOS 1.31; Speaker2Dubber with WER 24.92%, SIM 0.82, UTMOS 2.28; and StyleDubber with WER 30.57%, SIM 0.82, UTMOS 1.97. GroundTruth is also reported, with WER 35.62%, SIM 1.00, and UTMOS 2.02, a result the paper presents without further formal explanation. The summary given is that DualDub substantially outperforms the V2S baselines in WER and perceptual quality while maintaining speaker similarity comparable to the best V2S baselines (Tian et al., 14 Jul 2025).

The strongest contrast appears in joint soundtrack evaluation. The concatenated pipeline Speaker2Dubber + MMAudio reports DualScore 0.19, OAQ $1.94 \pm 0.08$, OSQ $2.62 \pm 0.11$, ASCH $1.44 \pm 0.06$. StyleDubber + MMAudio reports DualScore 0.23, OAQ $1.94 \pm 0.08$, OSQ $2.81 \pm 0.13$, ASCH $1.50 \pm 0.06$. Speaker2Dubber + FoleyCrafter reports DualScore 0.23, OAQ $2.12 \pm 0.07$, OSQ $-40$0, ASCH $-40$1. StyleDubber + FoleyCrafter reports DualScore 0.30, OAQ $-40$2, OSQ $-40$3, ASCH $-40$4. DualDub, as a joint model, reports DualScore 0.59, OAQ $-40$5, OSQ $-40$6, ASCH $-40$7, while GroundTruth reports DualScore 0.84, OAQ $-40$8, OSQ $-40$9, ASCH $$\mathrm{DualScore}=\cos(f_{\mathrm{audio}},f_{\mathrm{speech}})=\frac{f_{\mathrm{audio}}\cdot f_{\mathrm{speech}}}{\|f_{\mathrm{audio}}\|\|f_{\mathrm{speech}}\|}.$$0 (Tian et al., 14 Jul 2025).

The paper interprets these results as showing that concatenated pipelines tend to produce rhythmic or volume conflicts and even unintended vocal artifacts in background audio. Formal statistical tests are not provided, although the reported confidence intervals and score gaps support the qualitative comparison (Tian et al., 14 Jul 2025).

5. Ablations, methodology sensitivity, and practical use

DualBench is also used for ablation analysis. The reported ablations isolate curriculum learning, the cross-modal aligner, and the flow-matching decoder. The base DualDub configuration reports FD 23.08, FAD 5.86, KLD 1.79, IS 2.72, AV-Align 0.25, SIM 0.84, WER 12.74%, UTMOS 2.70, and DualScore 0.59. Removing curriculum learning degrades performance to FD 31.61, FAD 9.61, KLD 2.60, IS 2.33, AV-Align 0.21, SIM 0.79, WER 15.40%, UTMOS 2.58, and DualScore 0.56. Removing the cross-modal aligner gives FD 24.26, FAD 6.78, KLD 1.87, IS 2.69, AV-Align 0.18, SIM 0.85, WER 11.96%, UTMOS 2.69, and DualScore 0.36. Removing the flow-matching decoder gives FD 42.61, FAD 13.61, KLD 6.74, IS 1.90, AV-Align 0.19, SIM 0.82, WER 13.55%, UTMOS 2.62, and DualScore 0.41 (Tian et al., 14 Jul 2025).

These ablations support three benchmark-relevant conclusions. First, curriculum learning stabilizes optimization and improves both speech and audio quality. Second, the cross-modal aligner is especially important for synchronization and harmony, as indicated by the sharp drops in AV-Align and DualScore. Third, the flow-matching decoder strongly affects background-audio quality, with consequential effects on the joint soundtrack metrics (Tian et al., 14 Jul 2025).

For practical evaluation, DualBench expects as input a silent video or a video with known original audio for reference comparisons, a transcript, and a reference speech prompt where applicable. The evaluation procedure then runs audioldm-eval for FD, FAD, IS, and KL; Whisper-large-v3 for WER; WavLM-based speaker verification for SIM; SpeechMOS for UTMOS; AV-Align for synchronization; and the CASP-based DualScore model for harmony. The benchmark resources and DualDub samples are made available through an anonymous repository, and the paper describes DualBench as an open-source benchmark, although specific license terms are not specified (Tian et al., 14 Jul 2025).

6. Limitations, biases, and terminological ambiguity

The benchmark has several explicit limitations. Its test set is built on animation content, so domain bias is inherent. Mel-RoFormer separation can produce empty segments, which motivated the $$\mathrm{DualScore}=\cos(f_{\mathrm{audio}},f_{\mathrm{speech}})=\frac{f_{\mathrm{audio}}\cdot f_{\mathrm{speech}}}{\|f_{\mathrm{audio}}\|\|f_{\mathrm{speech}}\|}.$$1 dB filtering criterion; this improves cleanliness but reduces the test set and may bias it toward higher-energy speech and audio clips. The paper does not detail language distribution, and the relatively high GroundTruth WER suggests either transcript mismatches or ASR sensitivity. DualBench also does not include explicit lip-sync metrics such as LSE-C, LSE-D, or SyncNet, relying instead on AV-Align peak IoU for synchronization assessment. Future improvements proposed in the paper include broader domains and languages, richer event-aligned annotations, improved separation models, and the inclusion of lip-sync measures (Tian et al., 14 Jul 2025).

The name "DualBench" also creates a potential literature ambiguity. In the materials considered here, the closest neighboring usage is $$\mathrm{DualScore}=\cos(f_{\mathrm{audio}},f_{\mathrm{speech}})=\frac{f_{\mathrm{audio}}\cdot f_{\mathrm{speech}}}{\|f_{\mathrm{audio}}\|\|f_{\mathrm{speech}}\|}.$$2-Bench, a dual-control benchmark for conversational agents formalized as a Dec-POMDP in a telecom technical-support domain. That work explicitly notes that it does not introduce a benchmark called "DualBench" by name, although $$\mathrm{DualScore}=\cos(f_{\mathrm{audio}},f_{\mathrm{speech}})=\frac{f_{\mathrm{audio}}\cdot f_{\mathrm{speech}}}{\|f_{\mathrm{audio}}\|\|f_{\mathrm{speech}}\|}.$$3-Bench can serve as a dual-control bench for research and development (Barres et al., 9 Jun 2025). Another nearby benchmark is DuoBench, officially named DuoBench rather than DualBench, which targets bimanual manipulation policies on the FR3 Duo platform in simulation and the real world (Jülg et al., 10 Jun 2026). These benchmarks share a concern with duality or coordination, but they address different modalities, tasks, and evaluation problems.

Within video-audio research, DualBench’s main historical significance is that it redefines evaluation around joint soundtrack generation rather than around isolated background audio. Relative to prior V2A benchmarks such as VGGSound-focused evaluation, it adds speech intelligibility, speaker identity, and cross-track harmony to the benchmark objective. That broader framing is its principal methodological contribution: it turns V2ST from a loosely specified synthesis goal into a measurable benchmark problem with separated references, a multi-axis metric protocol, and a learned harmony score (Tian et al., 14 Jul 2025).