VidChatBench: Audio-Visual Chat Benchmark

• VidChatBench is a controlled benchmark that evaluates generative models for dyadic audio-visual interactions, emphasizing identity preservation and coherent responses.
• It comprises 500 diverse conversational pairs from SpeakerVid-5M, featuring synchronized video, audio, and reference images for rigorous testing.
• The evaluation framework integrates advanced metrics like FID, FVD, SSIM, and ArcFace to measure visual quality, audio-visual consistency, and dialogue realism.

VidChatBench is a controlled, public benchmark for evaluating generative models of audio-visual dyadic video chat, designed as a held-out test set within the SpeakerVid-5M dataset. VidChatBench enables rigorous, standardized measurement of systems tasked with synthesizing both video and audio responses in virtual two-person interactions, with emphasis on identity preservation, multimodal coherence, and high-fidelity generation. The benchmark serves as a critical tool for the empirical analysis of end-to-end generative pipelines in the context of next-generation virtual human technologies, especially for single-turn, unconstrained audio-visual response synthesis without explicit textual prompt conditioning (Zhang et al., 14 Jul 2025).

1. Dataset Construction and Properties

VidChatBench is composed of 500 diverse dyadic conversational pairs sampled from the broader SpeakerVid-5M collection. Each sample includes: (1) an initiator video clip (length 3–14 s, 8 FPS, 480×768 resolution), (2) a reference image of the respondent for explicit identity control, and (3) the corresponding ground-truth responder video and audio. All speaker identities in VidChatBench are disjoint from the training split, ensuring strong generalization and robust evaluation.

The source dataset, SpeakerVid-5M, is stratified along two orthogonal axes:

• Interaction Branches:
  • Single branch: Monadic speech; 5.2M clips, 8,743 hr, 83K unique speakers.
  • Dialogue branch: Q→A dyads; 770K pairs, 1,800 hr (responses), 16K speakers.
  • Listening branch: Filtering co-present and non-co-present listening segments from dialogue data.
  • Multi-turn branch: Sequences with extended temporal context, organized in contextual and sequential extraction modes.
• Quality Stratification:
  • Supervised Fine-Tuning (SFT): 571K clips, 1,368 hr, selected via face/hand blur, DOVER, motion, and ASR confidence thresholds.
  • Pre-training: Remaining (approx. 4.6M single-speaker, plus all dialogue/listening/multi-turn), 7,375 hr.

VidChatBench samples exclusively from test-held identities. The benchmark is used solely for final evaluation; no validation splits are reported for the dyadic setting (Zhang et al., 14 Jul 2025).

2. Baseline System Architecture

The baseline system for VidChatBench evaluation is a fully autoregressive (AR) video chat model, integrating discrete token representations and high-capacity multimodal modules. The architecture comprises:

• Qwen2.5-Omni Thinker (frozen): A multimodal encoder ingesting video and audio streams.
• 3D-VAE Video Encoder/Decoder: Open-source, temporal stride 4, spatial stride 8; yields $C \times T \times H \times W$ latent maps, patched and flattened (360 patches/frame) for tokenization.
• CosyVoice2 Audio Tokenizer: Transforms raw audio waveform into 12 discrete tokens per chunk.
• Autoregressive Generation Head: A transformer operating over past visual and audio tokens with within-chunk positional encodings (1D for audio, combined 1D/2D for video); predicts next chunk of multimodal tokens.
• Spatial Transformer (NOVA-inspired): Refines visual token sequences to higher-resolution latents.
• Diffusion MLP: Denoises and enhances visual latents, conditioned on refined tokens, for high-fidelity reconstruction.
• CosyVoice Flow-Matching Vocoder: Synthesizes waveform from audio tokens.

Input representations accept tokenized video, audio, and respondent reference image (encoded by 3D-VAE). Output comprises the generated video and audio sequence for the dyadic response.

3. Training Objectives and Evaluation Protocol

The model is trained via a composite loss:

$$L_\text{total} = L_\text{audio} + \lambda_\text{diff} L_\text{diff}$$

where

• $L_\text{audio}$: Cross-entropy next-chunk prediction for autoregressive audio tokens,
• $L_\text{diff}$: Diffusion reconstruction loss over visual latents.

Pre-training is performed on the ~7,375 hr subset, with SFT on the 1,368 hr curated set. VidChatBench is reserved for held-out testing only.

Benchmarking mandates uniform preprocessing: frame rate fixed at 8 FPS, 480×768 resolution, 24 kHz audio sampling, identity crops for ArcFace metric extraction, and fixed random seeds for autoregressive sampling. All test samples feature initiator clips and respondent reference images with no direct text/ASR access, obliging generative models to infer intent and produce both modalities in a single-turn dyadic scenario (Zhang et al., 14 Jul 2025).

4. Metrics

VidChatBench supports a comprehensive multi-dimensional evaluation suite:

• Video Quality:
  • Fréchet Video Distance (FVD): Feature-distributional metric with Inception features over videos.
  • Fréchet Inception Distance (FID): Distributional similarity for image features, calculated frame-wise.
  • PSNR: Pixel-wise peak-signal-to-noise ratio.
  • SSIM: Structural similarity index.
• Identity Preservation: ArcFace cosine similarity between reference image and video frames.
• Dialogue Coherence: CLIP_dialog ranks generated responses by proximity in CLIP-embedding space to five manually or LLM-curated candidate responses, using a scoring function $S_{k^*}$ based on minimal CLIP distance.
• Audio-Visual Consistency: SyncNet confidence between lip motion and audio.
• Emotional Alignment: FID_emotion on Deep3DFaceRecon facial expression feature distributions.
• Audio Timbre Similarity: SIM-o score, cosine similarity in speaker embedding space (F5-TTS protocol).

This array collectively quantifies visual realism, speaker identity, cross-modal synchrony, linguistic alignment, affective accuracy, and audio timbre fidelity.

5. Results and Ablations

Ablation results (Table 7 of (Zhang et al., 14 Jul 2025)) reveal the following:

• Joint audio–video generation (“+A”): No degradation in video fidelity across FID/FVD/PSNR/SSIM.
• Spatial transformer (“+S”): Substantial gains in all visual and identity metrics.
• Noise injection (“+N”): Further mitigates long-term generation drift, improving visual quality.
• Dyadic context: Models with full audiovisual dyadic context (“Dyadic + A+S+N”) outperform those conditioned on text or ASR alone, demonstrating superior identity, coherence, and cross-modal results.

Representative scores for the best baseline configuration on VidChatBench include FID: 32.35, FVD: 28.82, PSNR: 17.55, SSIM: 0.66, ArcFace: 0.772, CLIP_dialog: 0.643, Sync_conf: 2.698, FID_emotion: 3.22, SIM-o: 0.65.

6. Reproducibility and Usage

Details regarding frame rate, resolution, chunking, and hardware are specified in the experimental section of (Zhang et al., 14 Jul 2025). The released resources encompass data processing scripts, model training recipes, and metric implementations, with all metric codes (FID, FVD, SyncNet, ArcFace, CLIP) adhering to standard, officially referenced toolkits.

Researchers utilizing VidChatBench are expected to preserve the test protocol—including unseen identity splits, single-turn dyadic format, and uniform data preprocessing—ensuring fair benchmarking and replicability. All data, code, and supporting materials are publicly available via the SpeakerVid-5M project page.

7. Significance and Relationship to Audio-Visual Dyadic Generation

VidChatBench is integral for advancing the evaluation and development of conversational virtual human systems, bridging video, audio, and language understanding. The focus on dyadic, single-turn, unconstrained interaction—absent explicit text conditioning—sets a high bar for multimodal reasoning and generative quality. The benchmark's multidimensional metrics reflect a holistic view of performance, supporting progress in identity coherence, audio-visual synchrony, dialogue realism, and affective expression in digital humans (Zhang et al., 14 Jul 2025). A plausible implication is that improvements on VidChatBench correspond to broader advances in realistic, responsive audiovisual avatar modeling.