---
title: 'MAV-Celeb: Multilingual Face–Voice Benchmark'
url: https://www.emergentmind.com/topics/multilingual-audio-visual-mav-celeb
type: topic
---

# MAV-Celeb: Multilingual Face–Voice Benchmark

Searching arXiv for MAV-Celeb and closely related papers to ground the article.
Multilingual Audio-Visual (MAV-Celeb) is a multilingual audio-visual celebrity dataset and benchmark introduced to study whether face–voice association and speaker recognition remain stable when the spoken language changes. It was created because earlier face–voice datasets such as VoxCeleb, FVCeleb, and FVMatching did not provide language-level annotations, which prevented direct analysis of language effects in cross-modal biometric matching [2004.13780]. Subsequent work positioned MAV-Celeb as the data foundation for the Face-voice Association in Multilingual Environments (FAME) challenge series, where the central task is cross-modal verification under heard and unheard language conditions rather than monolingual matching alone [2404.09342].

## 1. Origins and problem formulation

MAV-Celeb was introduced in "Cross-modal Speaker Verification and Recognition: A Multilingual Perspective" to address two explicit questions: whether face-voice association is language independent, and whether a speaker can be recognized irrespective of the spoken language [2004.13780]. The dataset was motivated by the observation that a large fraction of the world’s population is bilingual or trilingual, while most prior biometric corpora either lacked language annotations, were not audio-visual, or were not collected in sufficiently unconstrained wild settings [2004.13780].

The benchmark is framed around multilingual biometric analysis in the wild. Rather than treating language as irrelevant nuisance variation, MAV-Celeb makes language an experimental variable by collecting the same celebrity identities across multiple languages and evaluating performance when training and test languages differ [2004.13780]. This design was later formalized in the FAME 2024 evaluation plan as a multilingual cross-modal verification problem in which a face image and a voice recording must be matched under language mismatch, including evaluation on a completely unheard language for the same identity [2404.09342].

A common misconception is to conflate MAV-Celeb with general multilingual audio-visual learning datasets for instructional videos or speech recognition. The paper "Cascaded Multilingual Audio-Visual Learning from Videos" does not mention MAV-Celeb; its multilingual setting is based on YouCook-Japanese and Places Audio Caption rather than celebrity face–voice association [2111.04823]. MAV-Celeb is therefore not a generic multilingual AV pretraining corpus, but a benchmark centered on multilingual cross-modal biometric verification.

## 2. Dataset composition, languages, and collection pipeline

The original MAV-Celeb release contains 154 identities, 3 languages—English, Hindi, and Urdu—2,182 language-annotated human speech clips, and 41,674 utterances collected from online videos, primarily YouTube, with additional online image sources for static faces [2004.13780]. The choice of English, Hindi, and Urdu was justified by their representation of roughly 1.4 billion bilingual/trilingual speakers and the abundance of online video content in those languages [2004.13780].

The recordings come from unconstrained settings including political debates, press conferences, outdoor interviews, quiet studio interviews, and drama and movie clips [2004.13780]. Both modalities exhibit substantial nuisance variation: pose variation, motion blur, background clutter, occlusions, lighting changes, background chatter, music, overlapping speech, and compression artifacts [2004.13780]. This makes MAV-Celeb closer to in-the-wild biometric data than to studio-style corpora.

The collection pipeline is described as semi-automated and inspired by VoxCeleb/VGGFace-style acquisition. It begins with scraping Wikipedia to build a list of persons of interest, uses crowd-sourcing to gather YouTube queries and links, applies MTCNN for face detection and alignment, detects shot boundaries via color histogram comparison, uses SyncNet for active speaker verification, downloads static celebrity images with Google Custom Search API, uses DBSCAN to remove false positives and cluster faces, and trains an SVM classifier over Inception ResNet V1 embeddings trained on VGGFace2 with center loss to determine whether a face track belongs to a person of interest [2004.13780]. This pipeline reflects an explicit attempt to preserve multilingual coverage while retaining reliable identity labels under unconstrained web-video conditions.

## 3. Splits, statistics, and benchmark structure

The original dataset is divided into two multilingual splits: EU for English–Urdu and EH for English–Hindi [2004.13780]. These splits are constructed to test whether a model trained on one language generalizes to another language for the same identities. The train and test identities are disjoint; for the main verification setup, EU has 64 train identities and 6 test identities, while EH has 78 train identities and 6 test identities [2004.13780].

The original paper reports the following split statistics.

| Split | Core composition | Reported statistics |
|---|---|---|
| EU | English–Urdu | 70 celebrities; 43 male, 27 female; 560 / 406 / 966 videos; 59 / 32 / 91 hours; 11,835 / 6,550 / 18,385 utterances |
| EH | English–Hindi | 84 celebrities; 56 male, 28 female; 546 / 668 / 1,214 videos; 48 / 60 / 109 hours; 9,974 / 13,313 / 23,287 utterances |

The corresponding averages are also reported. For EU, the average videos per celebrity are 8 / 6 / 14, average utterances per celebrity are 169 / 94 / 263, and average utterance length is 17.9 / 17.8 / 17.8 seconds [2004.13780]. For EH, the average videos per celebrity are 6 / 8 / 14, average utterances per celebrity are 119 / 158 / 277, and average utterance length is 17.4 / 16.5 / 16.9 seconds [2004.13780].

Later FAME documentation preserved the same core logic while reformatting the benchmark as language-pair versions. The FAME 2024 evaluation plan describes MAV-Celeb as containing 3 languages—English, Hindi, and Urdu—and two language-pair splits, V1-EU and V2-EH, with 70 celebrities in V1-EU and 84 celebrities in V2-EH [2404.09342]. The FAME 2026 evaluation plan extends the benchmark with a new English-German split, V3-EG, and reports that the newer release includes an additional language and bilingual speakers [2508.04592]. This suggests an evolution from the original research dataset into a challenge-oriented benchmark with expanded language coverage and more explicit protocol control.

## 4. Tasks and evaluation protocol

The primary MAV-Celeb task is cross-modal verification: given a face image and a voice clip, the system predicts whether they belong to the same identity [2004.13780]. In the original paper, this task corresponds to the first research question, namely whether face–voice association transfers across languages [2004.13780]. The multilingual aspect is operationalized through an unseen-unheard protocol: training is done on identities from one language, and testing is performed on the same language, termed heard, and on a different language never seen during training, termed unheard [2004.13780].

The FAME challenge series formalizes this setup. The 2024 plan describes a heard/unheard configuration in which a model is trained on one language for a celebrity and evaluated both on the heard language and on a completely unheard language for the same identity [2404.09342]. The final evaluation is explicitly stated to use cross-modal verification on a completely unheard language [2404.09342]. The 2026 plan further specifies that each speaker appears in at least two languages, each video is in only one language, and the model is trained on one language and evaluated on both the heard and unheard language [2508.04592].

The evaluation metric is Equal Error Rate (EER), defined as the threshold where false acceptance rate and false rejection rate are equal [2004.13780]. The FAME plans retain EER as the official ranking measure and define the overall score as
$$
\text{Overall Score} = (\text{Sum of all EERs}) / 4
$$
across the four heard/unheard language conditions [2404.09342]. Lower EER indicates better verification performance. Because submissions consist of confidence scores rather than hard class decisions, EER is used as a threshold-independent verification criterion [2404.09342].

The FAME submission protocol also makes the multilingual setting operationally explicit through condition-specific score files such as `sub_score_English_heard.txt`, `sub_score_English_unheard.txt`, `sub_score_Urdu_heard.txt`, and `sub_score_Urdu_unheard.txt` for V1-EU, with analogous naming for V2-EH [2404.09342]. This converts the dataset from a research resource into a standardized challenge benchmark.

## 5. Baseline models and reported performance

The original MAV-Celeb paper evaluates a two-stream cross-modal verification network for face–voice association. The face branch uses a VGG-Face CNN descriptor, the voice branch uses VGG-Vox, and the embeddings are passed into a shallow two-branch network [2004.13780]. The face input is an RGB face crop resized to 256 × 256, while the voice stream uses a VGG-Vox configuration with embedding output from the fc7 layer [2004.13780]. The joint verification head contains fully connected layers, ReLU nonlinearities, and \(L2\) normalization [2004.13780].

The loss is a hinge-triplet-style metric loss enforcing that a positive face–voice pair be closer than a negative pair by a margin \(m\):
$$
d(f_i, y_j) + m < d(f_i, y_k) \quad \forall y_j \in Y_i^+, \forall y_k \in Y_i^-,
$$
with a symmetric constraint in the voice-to-face direction and additional neighborhood constraints weighted by \(\lambda_1 = 2\), \(\lambda_2 = 0.1\) or \(0.2\), and \(\lambda_3 = 0.1\) or \(0.2\) [2004.13780]. Triplets are selected within the mini-batch only [2004.13780].

The main empirical findings are that face–voice association degrades under language mismatch. On the EU split, the reported EERs are 45.1 for English train to English test, 48.3 for English train to Urdu test, 47.0 for Urdu train to English test, and 44.3 for Urdu train to Urdu test [2004.13780]. On the EH split, the corresponding EERs are 35.7, 36.7, 38.9, and 37.3 [2004.13780]. The paper concludes from these results that face–voice association is not language independent [2004.13780].

The same paper also studies speaker identification and speaker verification under language mismatch. For speaker identification, the baselines are VGGVox, an Utterance Level model using NetVLAD or GhostVLAD, and SincNet; the metric is Top-1 accuracy [2004.13780]. For speaker verification, the metric is again EER [2004.13780]. Across these settings, performance drops when the test language is unheard, leading the authors to conclude that speaker recognition is also not language independent [2004.13780].

Later FAME baselines use Fusion and Orthogonal Projection for Improved Face-Voice Association (FOP), described as a two-stream pipeline with pretrained face and voice encoders, multimodal fusion, and orthogonal constraints to learn discriminative joint face–voice embeddings [2404.09342]. The FAME 2024 plan reports for V2-EH EERs of 20.8 for English train to English test, 24.0 for English train to Hindi test, 24.0 for Hindi train to English test, 19.3 for Hindi train to Hindi test, and an overall score of 22.0 [2404.09342]. For V1-EU, it reports 29.3, 37.9, 40.4, 25.8, and an overall score of 33.4 [2404.09342]. The FAME 2026 plan reports the same V1-EU values and, for the new V3-EG split, 34.5 for English train to English test, 43.7 for English train to German test, 43.2 for German train to English test, 39.6 for German train to German test, and an overall score of 40.2 [2508.04592].

These later baselines do not overturn the original conclusion. They reinforce it under a more formal challenge setting: heard-language performance is consistently better than unheard-language performance, and language mismatch remains a significant source of degradation [2404.09342][2508.04592].

## 6. Research significance, benchmark evolution, and scope

MAV-Celeb fills a specific gap in multilingual biometric evaluation. Earlier datasets such as VoxCeleb, FVCeleb, and FVMatching could support face–voice association or speaker recognition, but not direct analysis of language effects because they lacked language annotations [2004.13780]. MAV-Celeb made language-dependent evaluation possible while preserving unconstrained web-video realism [2004.13780].

The benchmark’s subsequent reuse in FAME indicates that the dataset became more than a one-off corpus. In FAME 2024, MAV-Celeb is presented explicitly as a public multilingual audio-visual dataset of celebrity recordings designed for face–voice association research under multilingual conditions [2404.09342]. In FAME 2026, the benchmark is extended with English–German data and distributed with `.wav` audio, `.jpg` images, text files specifying face–voice pairs, and pre-extracted features from state-of-the-art pre-trained encoders [2508.04592]. This progression suggests a transition from dataset introduction to shared evaluation infrastructure.

MAV-Celeb also anchors a broader research question: whether cross-modal identity cues are language-specific or language-invariant. The original results indicate that they are not fully language invariant [2004.13780]. The challenge framing then recasts this as a multilingual generalization problem rather than a purely biometric matching problem [2404.09342][2508.04592]. A plausible implication is that models trained on face–voice identity cues capture a mixture of person-specific information and language-conditioned acoustic structure, so multilingual robustness cannot be assumed even when the face modality is unchanged.

At the same time, MAV-Celeb should not be generalized beyond its stated scope. The benchmark focuses on celebrities, only a limited set of languages and bilingual pairings, and verification rather than open-set identification or retrieval [2404.09342]. The data are collected from online media and thus inherit noise and bias from uncontrolled acquisition conditions [2404.09342]. These characteristics make the benchmark realistic for multilingual face–voice association, but they also delimit the claims that can be made from it.

## 7. Distinctions from adjacent datasets and recurring confusions

The name MAV-Celeb can be confused with other multilingual audio-visual resources. One source of confusion is the Multilingual Audio-Visual Smartphone dataset, which is a different corpus built for smartphone biometric verification. That dataset contains 103 subjects, 3 sessions, 5 smartphones, and 3 languages—English, Hindi, and Bengali—and emphasizes device variation, session variation, and presentation attacks rather than celebrity face–voice matching under heard/unheard language protocols [2109.04138]. Despite thematic overlap in multilingual audio-visual biometrics, it is not MAV-Celeb.

A second confusion concerns multilingual audio-visual representation learning and speech recognition datasets. "Cascaded Multilingual Audio-Visual Learning from Videos" studies multilingual transfer from English instructional videos to Japanese and Hindi but does not define or use MAV-Celeb [2111.04823]. "Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model" addresses multilingual AVSR on MuAViC, a nine-language TED/TEDx corpus, rather than multilingual face–voice association [2310.14946]. These works are relevant to multilingual AV modeling in general, but not to the definition of MAV-Celeb itself.

Within the biometric literature, the distinctive feature of MAV-Celeb is therefore not merely that it is multilingual and audio-visual, but that it was designed to test whether cross-modal identity matching survives language change for the same speaker [2004.13780]. The FAME challenge series preserves exactly this emphasis by treating the unheard language as a first-class evaluation condition rather than incidental metadata [2404.09342][2508.04592].

Source: https://www.emergentmind.com/topics/multilingual-audio-visual-mav-celeb