Multilingual Talking Face Benchmark (MTFB)

• MTFB is a benchmark for speech-driven talking face synthesis featuring multilingual, controlled audiovisual data and universal phoneme-viseme alignment.
• It comprises 95.04 hours of high-quality video/audio pairs across 12 diverse languages, enabling intra-language, cross-language, and zero-shot evaluations.
• The benchmark employs detailed metrics such as FVD, Sync-C, LSE-D, and TMDC to rigorously assess articulatory fidelity and perceptual realism in synthesized faces.

Searching arXiv for the benchmark papers and related multilingual talking-face work. I’m going to use the arXiv search capability to verify the cited papers before writing the article. Multilingual Talking Face Benchmark (MTFB) is a benchmark for speech-driven talking face synthesis (TFS) introduced in “A Bridge from Audio to Video: Phoneme-Viseme Alignment Allows Every Face to Speak Multiple Languages” (Su et al., 8 Oct 2025). It comprises 12 diverse languages with 95.04 hours of high-quality videos for training and evaluating multilingual TFS performance, and it was constructed to address the English-dominated training distributions and weak cross-language generalization identified in prior talking-face systems. In the same work, MTFB is paired with the Multilingual Experts (MuEx) framework, whose Phoneme-Guided Mixture-of-Experts (PG-MoE) architecture and Phoneme-Viseme Alignment Mechanism (PV-Align) use phonemes and visemes as universal intermediaries to bridge audio and video modalities (Su et al., 8 Oct 2025).

1. Corpus definition and linguistic coverage

MTFB comprises 12 typologically diverse, high-resource and low-resource languages. Each language is represented by professionally recorded news broadcasts and one-on-one interview videos collected by the authors. The total duration is 95.04 hours of curated video/audio pairs. Although the paper does not quote an exact speaker count, each language comprises dozens of unique speakers, typically 20–50, with a roughly balanced gender distribution whenever possible (Su et al., 8 Oct 2025).

Language	Approximate data source
English	United States/BBC news
Mandarin Chinese	CCTV news/interviews
Spanish	RTVE news
French	France 2/France 24 interviews
German	Deutsche Welle news
Japanese	NHK news
Italian	RAI news
Russian	RT news
Arabic	Al Jazeera news
Dutch	NOS news
Thai	Thai PBS interviews
Vietnamese	VTV news

All videos are center-cropped and resized to $512 \times 512$ pixels, standardized to 25 fps, and paired with audio resampled to 16 kHz, mono channel. Video quality filters require a single frontal speaker in shot, no background music or heavy ambient noise, and good illumination without extreme occlusions (Su et al., 8 Oct 2025).

This composition suggests that MTFB was designed to prioritize controlled audiovisual correspondence over maximal environmental diversity. A plausible implication is that the benchmark emphasizes cross-lingual articulation fidelity under relatively clean recording conditions rather than robustness to unconstrained internet video.

2. Construction pipeline and representational design

The preprocessing pipeline is specified as a sequence of face normalization, landmark extraction, audiovisual feature extraction, prototype clustering, and quality control (Su et al., 8 Oct 2025). Face Detection & Alignment uses a lightweight detector, exemplified by MediaPipe Face Mesh, to locate the face, crop it to $512 \times 512$, and align it so that eyes and mouth lie in canonical positions. Landmark Extraction then produces a 468-point facial mesh, from which 26 standardized lip landmarks are retained for LSE-D and TMDC computation.

Audio-Visual Feature Extraction is explicitly split into phoneme features and viseme features. For phoneme features, the input waveform is passed through a Wav2Vec 2.0 encoder to obtain continuous embeddings $\{z_t^p\}$. For viseme features, the cropped mouth region is processed by a CNN backbone, exemplified as ResNet-based, to obtain continuous embeddings $\{z_t^v\}$. Prototype Clustering & Annotation then uses K-means++ to form $K=40\text{–}44$ phoneme prototypes $\{c_k^p\}$ and viseme prototypes $\{c_k^v\}$, with hard and soft assignments producing pseudo-phoneme labels $v_t$ and discrete viseme codes $q_t^v$. Manual spot-checks are finally used to ensure correct mouth visibility and synchronization (Su et al., 8 Oct 2025).

Within the MuEx framework, these design choices are tied to the claim that phonemes and visemes are the basic units of speech sounds and mouth movements. The benchmark therefore does not merely collect multilingual clips; it supports a representation regime in which audiovisual structure is normalized into cross-lingual intermediaries. This suggests that MTFB is intended to measure whether multilingual TFS systems can preserve articulatory consistency even when surface language characteristics differ.

3. Splits, benchmark tasks, and evaluation procedure

For each of the 12 languages, MTFB is randomly partitioned in a speaker-disjoint manner into training, validation, and test splits: 80% of clips for training, approximately 76 hours total; 10% for validation, approximately 9.5 hours; and 10% for test, approximately 9.5 hours (Su et al., 8 Oct 2025). The benchmark also defines cross-lingual zero-shot protocols. The 12 benchmark languages are treated as “seen” languages and are used fully for training, while Korean, Burmese, and Hindi are held out as “unseen” languages for zero-shot evaluation, with no fine-tuning allowed.

Three tasks are defined. Within-language TFS trains and tests on the same language, chosen from the 12 benchmark languages. Cross-language TFS generates faces in one language voice while driving a face from another language. Zero-shot generalization synthesizes talking faces for the three unseen languages, Korean, Burmese, and Hindi (Su et al., 8 Oct 2025).

The evaluation procedure uses 5–10 s generated video clips from each test split. Four quantitative metrics are computed: FVD, Sync-C, LSE-D, and TMDC. In addition, a human study uses 300 native speakers across 15 language groups to rank systems from 1st to 6th on lip-audio synchronization and teeth-naturalness, with weighted scores of 6 points for 1st down to 1 point for 6th, averaged per language (Su et al., 8 Oct 2025).

This protocol separates three distinct generalization regimes: intra-language fidelity, cross-language transfer, and unseen-language transfer. A plausible implication is that MTFB treats multilinguality not as a single aggregate property, but as a stack of increasingly stringent tests on articulatory alignment.

4. Metric suite and associated mathematical formulations

MTFB specifies clip-level, synchronization-oriented, landmark-based, and temporal-dynamics metrics, and the paper also reports the principal MuEx loss functions alongside the benchmark definitions (Su et al., 8 Oct 2025).

Fréchet Video Distance (FVD) extends FID to video by embedding short clips via an I3D network. Let $(\mu_r, \Sigma_r)$ and $512 \times 512$0 be the means and covariances of real versus generated clip features. The formula is given as

$512 \times 512$1

SyncNet Confidence (Sync-C) uses a pre-trained SyncNet that returns a frame-wise lip-audio alignment score $512 \times 512$2, and is defined by

$512 \times 512$3

Higher means better synchronization.

Lip-Sync Error Distance (LSE-D) measures per-frame error between 26 ground-truth and generated lip landmarks $512 \times 512$4 and $512 \times 512$5:

$512 \times 512$6

and

$512 \times 512$7

Lower is better.

Temporal Mouth Dynamics Correlation (TMDC) uses five shape features per frame—width, height, area, aspect ratio, and openness. Let $512 \times 512$8 and $512 \times 512$9 be the real and generated values. The per-feature Pearson coefficient is

$\{z_t^p\}$0

and

$\{z_t^p\}$1

Higher is better.

The Phoneme-Viseme Alignment loss is defined as

$\{z_t^p\}$2

where

$\{z_t^p\}$3

It encourages matching discrete codes while penalizing spurious raw-feature correlations.

The PG-MoE router loss is

$\{z_t^p\}$4

The generation loss is given by

$\{z_t^p\}$5

Taken together, these definitions indicate that the benchmark is tightly coupled to an articulatory account of multilingual talking-face synthesis: clip realism, frame-level synchronization, geometric lip accuracy, and temporal mouth dynamics are measured separately rather than collapsed into a single scalar.

5. Baselines, reported performance, and ablation evidence

The baselines re-trained on MTFB are AniTalker (CVPR’24), SadTalker (CVPR’23), EchoMimic (arXiv 2024), AniPortrait (arXiv 2024), and Hallo2 (ICLR 2025) (Su et al., 8 Oct 2025). On the 12 seen languages, the paper reports metric averages over all languages as follows: FVD $\{z_t^p\}$6: MuEx 171.28 versus the best baseline, Hallo2, 158.63; Sync-C $\{z_t^p\}$7: MuEx 7.536 versus Hallo2 7.469; LSE-D $\{z_t^p\}$8: MuEx 0.0437 versus baseline best 0.0447; and TMDC $\{z_t^p\}$9: MuEx 0.756 versus baseline best 0.712.

The human evaluation reports that MuEx ranks first in both lip-sync and teeth-naturalness for all 12 training languages, with average scores of 5.68 and 5.37 respectively. On the three unseen languages, MuEx still leads, with average scores of 5.57 and 5.52 (Su et al., 8 Oct 2025). In the ablations, removing PG-MoE or PV-Align degrades LSE-D by up to 40% and TMDC by up to 30%.

These results place the strongest emphasis on synchronization, mouth-shape accuracy, temporal articulation, and human preference. The reported average FVD value differs from the best baseline average, while Sync-C, LSE-D, TMDC, and the human studies favor MuEx. This suggests that, within MTFB, the paper treats multilingual articulatory correctness and perceptual mouth realism as the primary axes of improvement.

6. Relation to adjacent multilingual datasets and stated limitations

A related line of work is “MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Dataset” (Sung-Bin et al., 2024). That work introduces a multilingual 2D video dataset comprising over 420 hours of talking videos in 20 languages, presents a multilingually enhanced model with language-specific style embeddings, and introduces a metric for assessing lip-sync accuracy in multilingual settings. Its emphasis is on 3D talking head generation and multilingual 2D video-to-3D supervision, whereas MTFB in (Su et al., 8 Oct 2025) is presented as a benchmark for multilingual TFS with phoneme-viseme alignment. This suggests that multilingual talking-face research has developed along both 2D video-centric and 3D mesh-centric trajectories.

The limitations explicitly noted for MTFB are that speaker demographics skew toward news-anchor profiles, with limited age range and expressive variability; all data are formal speech, so conversational or emotional speech is under-represented; and there is an absence of low-resource African and Indic languages beyond Hindi in zero-shot evaluation (Su et al., 8 Oct 2025). The suggested expansions are to incorporate conversational, emotional, and multi-person dialogues; increase speaker diversity in age, accent, and spontaneous speech; extend coverage to additional language families such as Bantu, Dravidian, and Austronesian; add noisy or in-the-wild conditions including background music and reverberation; and provide finer speaker metadata such as gender, age, and dialect.

These constraints matter for interpreting benchmark scores. Strong results on MTFB indicate performance under curated multilingual broadcast and interview conditions, but they do not by themselves establish robustness to spontaneous dialogue, emotional prosody, multi-person interaction, or broader demographic coverage.