---
title: 'MTAVG-Bench: Multi-Talker T2AV Evaluation'
url: https://www.emergentmind.com/topics/mtavg-bench
type: topic
---

# MTAVG-Bench: Multi-Talker T2AV Evaluation

MTAVG-Bench is a comprehensive diagnostic benchmark specifically designed to evaluate text-to-audio-video (T2AV) models in the challenging context of multi-talker, dialogue-centric generation. While previous evaluation protocols primarily addressed single-speaker or human-recorded audio-visual content, MTAVG-Bench introduces a structured, failure-focused evaluation pipeline for generated videos featuring two or more participants in interactive dialogue. It comprises 1,880 videos synthesized by state-of-the-art T2AV models, each accompanied by expert-annotated, fine-grained question–answer (QA) pairs that span nine error types across four levels: audio-visual signal fidelity, temporal attribute consistency, social interaction, and cinematic expression. This resource enables detailed error analysis, model comparison, and guides targeted refinements for next-generation multi-speaker T2AV systems [2602.00607].

## 1. Dataset Construction and Curation

MTAVG-Bench was constructed via a semi-automatic pipeline comprising three stages:

**a. Prompt Generation and Multi-Model Synthesis:**  
Seeded by thousands of short two-speaker dialogue sketches, an LLM rewrote each sketch into a structured prompt specifying character attributes (gender, age bracket, ethnicity, clothing), environment (location, time, ambience), and visual style (realistic/hyper-realistic). Speaker-centric camera instructions (e.g., "[Close shot]") were included for cinematic guidance. These prompts were input to three advanced T2AV generators—Veo 3.1, Wan 2.5, and Sora 2—producing over 3,000 diverse multi-actor, multi-turn dialogue videos represented in broad real-world scenarios.

**b. Automated Filtering for Failure Emphasis:**  
An agent-based filter discarded candidate videos with no detectable AI generation errors (e.g., perfect lip-synch and no identity drift). This guarantees the retained benchmark corpus focuses solely on videos containing at least one substantial generation failure, thus directly facilitating diagnostic evaluation.

**c. Annotation of Failures and QA Generation:**  
Automated detectors flagged candidate anomalies such as lip–audio mismatches or abrupt cinematic transitions for manual review. Human annotators confirmed true positives, mapped failures to a taxonomy of nine error classes, and, assisted by LLMs, generated corresponding QA pairs in single-choice, multiple-choice, or pairwise comparison formats. Expert reviewers validated and refined these QA items—resulting in 2,410 high-quality, human-verified QA pairs covering all nine diagnostic dimensions, with each video containing at least one error and an associated QA item.

*Data distribution:*  
- 1,880 multi-talker videos  
- 2,410 QA pairs  
- Dominant error types are lip-sync (Level 2, LS), turn-taking logic (Level 3, TT), and speaker–utterance mismatch (Level 3, SA), with coverage across all nine diagnostic dimensions [2602.00607].

## 2. Hierarchical Benchmark Organization

MTAVG-Bench provides a four-level, hierarchical taxonomy of diagnostic criteria, each capturing a distinct aspect of audio-visual multi-speaker dialogue generation:

| Level | Dimension (Abbreviation)             | Criterion Description                   |
|-------|--------------------------------------|-----------------------------------------|
| 1     | VQ (Perceptual Video Quality)        | No visual artifacts (flicker, blur)     |
| 1     | SQ (Perceptual Speech Quality)       | Speech naturalness, uninterrupted audio |
| 2     | SC (Scene Consistency)               | Coherence of background, lighting, props|
| 2     | CC (Character/Speaker Consistency)   | Identity stability: voice, appearance   |
| 2     | LS (Lip–Audio Synchronization)       | Alignment of mouth motion to audio      |
| 3     | SA (Speaker–Utterance Alignment)     | Correct attribution of utterances       |
| 3     | TT (Turn-Taking Logic)               | Proper dialogue turn boundaries         |
| 4     | EA (Expressive Alignment)            | Match of gesture/emotion with prosody   |
| 4     | CA (Camera Alignment)                | Camera follows active speaker           |

Scoring is defined as follows for each question $i$, ground truth $G_i$, and model prediction $P_i$:

$$
s_i = \begin{cases} 
\mathbb{I}[P_i = G_i], & \text{(single-choice/pairwise)} \\[4pt]
\frac{|P_i \cap G_i|}{|G_i|}, & \text{(multiple-choice)}
\end{cases}
$$

Dimension-level and overall scores:

$$
\mathrm{Score}_d = \frac{1}{|\mathcal{Q}_d|}\sum_{i\in\mathcal{Q}_d} s_i,
\qquad
\mathrm{Avg.} = \frac{1}{|\mathcal{D}|}\sum_{d\in\mathcal{D}}\mathrm{Score}_d
$$

where $\mathcal{Q}_d$ denotes the set of QA pairs for dimension $d$, and $\mathcal{D}$ the nine dimensions.

This hierarchical design enables both low-level (signal fidelity) and high-level (social, cinematic) diagnostic coverage, facilitating nuanced model analysis [2602.00607].

## 3. Diagnostic Metrics and Measurement Protocols

Each of the nine dimensions functions as a targeted evaluation metric:

- **VQ:** Fraction of artifact-free videos.
- **SQ:** Fraction of clips with natural, uninterrupted speech.
- **SC:** Proportion of videos maintaining consistent environment (lighting, props) across frames.
- **CC:** Fraction where each speaker’s appearance and voice remain consistent; Identity Drift quantifies losses here.
- **LS:** $1 - \text{Score}_{LS}$ measures prevalence of lip–audio misalignments.
- **SA:** Rate of correct speaker–utterance attributions.
- **TT:** Dialogue sequences with valid turn-taking logic (no overlaps, missing turns).
- **EA:** Fraction matching gesture/emotion to prosody.
- **CA:** Fraction where camera focuses on the active speaker.

The annotation process blends automated failure detection (candidate anomaly proposal) with human confirmation, mapping and refinement through LLM-assisted QA item generation. Model performance is quantified by model answers on these validated QAs using the aforementioned scoring formulae [2602.00607].

## 4. Model Benchmarking and Comparative Results

Twelve leading proprietary and open-source omni-modal T2AV systems were evaluated on MTAVG-Bench. The following table compares top models on each dimension (all values in percentages):

| Model          | VQ   | SQ   | SC   | CC   | LS   | SA   | TT   | EA   | CA   | Avg.  |
|----------------|------|------|------|------|------|------|------|------|------|-------|
| Gemini 3 Pro   | 70.4 | 55.3 | 53.4 | 41.2 | 52.9 | 68.6 | 60.8 | 58.0 | 50.8 | 56.8  |
| Ola-Omni (7B)  | 46.4 | 56.0 | 37.5 | 36.8 | 62.0 | 52.2 | 43.3 | 46.1 | 50.0 | 47.8  |
| Video-LLaMA2   | 48.8 | 50.0 | 48.0 | 45.1 | 47.5 | 50.9 | 39.8 | 45.9 | 51.2 | 47.5  |

Key patterns observed:
- Signal fidelity (VQ, SQ) is consistently strong (≥ 45%) with relatively low variance (≤ 10 points).
- Interaction-level dimensions (SA, TT) exhibit large performance gaps (often >20 points), revealing persistent model weaknesses in assigning "who speaks when."
- Lip-sync (LS) and character consistency (CC) remain challenging: drift rates can exceed 60%.
- Cinematic metrics (EA, CA) are lowest for all systems (<60% for EA, ~50% for CA).
- Typical errors include identity drift (CC), silent lip movement (LS), unprompted utterances (TT), and camera neglect of speaker (CA) [2602.00607].

## 5. Evaluation Methodology and Best Practices

The recommended MTAVG-Bench protocol for T2AV evaluation consists of:

- Application of the complete four-level diagnostic taxonomy for comprehensive model assessment.
- Supplying both the original generation prompt and synthesized clip to all evaluators; prompt–output misalignment can significantly degrade scores (ablation study–5 points CC/CA penalty).
- Reporting both per-dimension and overall scores to expose specific system weaknesses.
- Adhering to the provided QA item templates (single-choice, multi-choice, pairwise) and scoring formulae for robust replicability.
- Blending automated and expert-validated annotation to ensure accuracy and fine-grained coverage.

A significant implication is that low-level signal quality can mask critical high-level failures (such as incorrect speaker attribution or turn-taking flaws), necessitating a holistic metric profile in published comparisons [2602.00607].

## 6. Limitations and Prospects for Benchmark Extension

Current limitations of MTAVG-Bench include:

- Evaluator OOD: Many vision-language models were not trained on the fine-grained lip-sync and time-alignment discrepancies present in T2AV generations; future work can fine-tune evaluation modules with MTAVG-Bench.
- Imbalanced error prevalence: Lip-sync (LS) and turn-taking (TT) dominate the dataset; targeted prompt engineering in future releases could achieve more uniform coverage.
- Modal diversity: Present coverage is limited to English dialogues, standard accents, and single-scene clips under 15 seconds; future updates may integrate diverse languages, accents, crowd scenes, and longer/complex multi-shot narratives.
- Automation: Improving off-the-shelf metric proxies for automatic annotation (e.g., lip-sync detectors) may further reduce curation burden.

MTAVG-Bench serves as a foundational diagnostic framework for testing and refining multi-speaker, dialogue-driven T2AV models, enabling rigorous, failure-centric analysis and benchmarking for the evolving landscape of generative audio-visual research [2602.00607].

Source: https://www.emergentmind.com/topics/mtavg-bench