FAME 2026 Challenge

• FAME 2026 Challenge is a benchmark designed to evaluate cross-modal biometric association by verifying face-voice pairs under both language-matched and unseen conditions.
• It quantifies the degradation in performance due to language mismatch using metrics like Equal Error Rate (EER), ROC, and mAP on a bilingual dataset.
• Key methodologies include fusion strategies, orthogonal projection losses, and attention-based multimodal alignment for robust cross-lingual generalization.

The Face-Voice Association in Multilingual Environments (FAME) 2026 Challenge is a benchmark designed to advance methodology in cross-modal biometric association, specifically targeting the verification and retrieval of person identity from paired facial images and voice utterances under multilingual and cross-lingual conditions. The challenge addresses the open research question of how language mismatch between training and test data impacts the robustness of face-voice association, reflecting the reality that a majority of the global population regularly communicates in multiple languages. The FAME 2026 challenge was organized as a special event at ICASSP 2026 and constitutes a rigorous testbed for both algorithmic innovation and systematic error analysis in multimodal person verification and retrieval across unseen languages (Moscati et al., 23 Dec 2025, Moscati et al., 6 Aug 2025, Simic et al., 4 Dec 2025).

1. Objectives and Task Definition

The principal objective is to develop and evaluate cross-modal verification methods that can associate a face image with a speech segment and ascertain whether they belong to the same identity—even when the language at test time has not been encountered during training (so-called "unheard" languages). The challenge quantifies the degradation in association performance incurred by a language mismatch.

Two primary task scenarios structure the competition:

• Development Phase: Methods are trained on a multi-speaker, bilingual dataset (English + German) and evaluated on languages seen during training, creating an in-language test scenario.
• Evaluation Phase: Models are required to generalize to evaluation on a split of speakers who appear exclusively in an "unheard" language, holding out both identity and language to stress-test cross-language robustness.

An optional retrieval extension shifts the setting from binary verification to ranking by similarity for face–voice matching queries (Moscati et al., 23 Dec 2025).

Formally, the core verification problem can be described as learning a function $s(x_f^i, x_v^j)$ that, given a face embedding $x_f^i$ and a voice embedding $x_v^j$, outputs a similarity score indicating whether samples $i$ and $j$ are of the same identity, with higher scores suggesting greater confidence in identity match (Moscati et al., 6 Aug 2025, Simic et al., 4 Dec 2025).

2. Dataset and Multilingual Protocol

The official dataset is MAV-Celeb ("Multilingual Audio–Visual Celebrities"), sourced from YouTube interviews with robust variation in pose, lighting, background conditions, and artifacts. Key dataset facts (Moscati et al., 6 Aug 2025, Moscati et al., 23 Dec 2025, Simic et al., 4 Dec 2025):

• Composition: For the English–German split (V3-EG), 58 bilingual speakers contributed multiple face images (from video frames) and speech utterances (≥1 s duration); utterances totaled 3,812 across 428 bilingual videos.
• Splits:
  • 50 training speakers (used for development/fine-tuning)
  • 8 test speakers (used solely for final evaluation, disjoint from train identities)
• Language Annotation: Each video is strictly monolingual; bilingual speakers ensure robust variance in cross-language appearance.
• Preprocessing:
  • Face images undergo detection, alignment, resizing (224×224), and pixel normalization.
  • Voice clips are resampled to 16 kHz, voice-activity detected, features extracted as 64-dimensional mel-filterbanks, and normalized.
• Auxiliary Datasets: VoxCeleb2 and multilingual CommonVoice subsets are used for both pre-training (with language-exclusion to preserve "unheard" split protocol) and for training auxiliary age/gender predictors (Simic et al., 4 Dec 2025).

A pivotal protocol constraint forbids any pre-training or fine-tuning on data from "unheard" languages in the held-out split, enforcing a pure cross-lingual generalization regime in evaluation (Moscati et al., 6 Aug 2025).

3. Evaluation Metrics and Protocol

Evaluation focuses on the ability of models to correctly verify identity under both language-matched and language-mismatched conditions. The principal metrics are as follows:

• Equal Error Rate (EER): This is the threshold $\tau^*$ where the false acceptance rate (FAR) and false rejection rate (FRR) are equal:

$$\text{FAR}(\tau) = \frac{\#\{\text{impostor pairs with } s \geq \tau\}}{\#\{\text{impostor trials}\}}, \qquad \text{FRR}(\tau) = \frac{\#\{\text{genuine pairs with } s < \tau\}}{\#\{\text{genuine trials}\}}$$

$$\text{EER} = \text{FAR}(\tau^*) = \text{FRR}(\tau^*)$$

• Retrieval@K (optional): Computes the proportion of queries where the correct match is ranked within the top $K$ returned items:

$$\text{M@K} = \frac{1}{N} \sum_{i=1}^N \mathbb{I}(r_i \le K)$$

where $x_f^i$0 is the rank of the correct item for query $x_f^i$1, and $x_f^i$2 is the number of queries (Moscati et al., 23 Dec 2025).

• Area under ROC Curve (AUC) and mean Average Precision (mAP) are also tracked in baseline reports (Moscati et al., 6 Aug 2025).

Challenge protocol enforces strict evaluation phases:

• Progress Phase: Public dev leaderboard with submissions per language split, up to 100 submissions (max 10/day).
• Evaluation Phase: Held out test on “unheard” language, up to 5 submissions (Moscati et al., 6 Aug 2025).

4. Baseline Systems and Leading Approaches

The baseline, available as public code, is a "Fusion & Orthogonal Projection" (FOP) pipeline (Moscati et al., 23 Dec 2025, Moscati et al., 6 Aug 2025):

• Architecture: Two-branch network; face images processed via CNN (e.g., Parkhi et al.’s Deep Face/ResNet-50), voices via an utterance-level audio encoder (e.g., Xie et al.’s ResNet).
• Embedding: Each modality embedding is projected to a 512-dimensional joint space, then fused (concatenation or learned gating).
• Loss: Combines cross-entropy for speaker classification with an orthogonal projection loss:

$x_f^i$3

where $x_f^i$4 and $x_f^i$5 are projection matrices for each modality, and $x_f^i$6 is the identity (Moscati et al., 23 Dec 2025).

Top challenge entries adopted a range of advanced architectures:

Rank	Team	Key Methods	EER (%)
1	Simicch	Dual transformers (face/voice), shared embedding, AAM/SupCon loss	23.99
2	Areffarhadi	Foundation models fine-tuned for cross-modal/lingual invariance	24.73
3	Alpha_code	Attention-based fusion, refined orthogonality loss	33.11
4	LTINI	Contrastive gated fusion (dynamic channel gating)	33.18
5	Punkmale	Margin-based cross-modal alignment	33.51
-	Baseline (FOP)	Fusion & Orthogonal Projection	41.57

• Simicch et al. (Simic et al., 4 Dec 2025): Used ECAPA-TDNN and VGGFace/ViT for uni-modal embeddings, concatenated with age/gender features. Mapped 7680-dim audio and 4864-dim visual features by joint linear projections into a 192-dim shared embedding. Aggressive dropout ($x_f^i$7) was applied before linear maps due to high base dimensionality. Trained with Adaptive Angular Margin (AAM) loss, which encourages small angular separation for same-identity and orthogonality for different identities. Pre-trained on VoxCeleb2, fine-tuned on MAV-Celeb under split constraints.
• Areffarhadi et al.: Fine-tuned large CLIP-style audio-visual encoders, leveraging language-agnostic pooling to mitigate language mismatch.
• Alpha_code: Refined FOP via attention-based multimodal fusion and subspace-weighted orthogonality.
• LTINI and Punkmale: Employed dynamic, contrastive, and margin-based objectives to explicitly structure cross-modal embedding space for robustness.

5. Results and Comparative Analysis

Aggregate results demonstrate a clear hierarchy in cross-lingual generalization:

• Baseline (FOP) EER: 41.57% (nearly doubled from in-language to unheard language within the same protocol).
• Top method (Simicch et al.) EER: 23.99%; runner-up achieved 24.73%.
• Variance: Simicch et al. also displayed low submission-to-submission variance, indicating methodological stability.
• Per-language performance (from Simicch et al. (Simic et al., 4 Dec 2025) Table 1): English test set—30.6% (English-trained), 30.1% (German-trained); German test set—17.4% (English-trained), 17.9% (German-trained).
• Reduction in language-mismatch gap: Leading entries were able to reduce the degradation under language mismatch by 40–45% relative to the baseline, demonstrating effectiveness of contrastive and embedding-alignment objectives.

A plausible implication is that joint embedding spaces and leveraged pretraining on large, diverse audio-visual corpora are crucial for robust cross-language identity association.

6. Technical and Methodological Insights

Three methodological themes emerge:

• Robustness to Language Mismatch: Naively trained models experience severe accuracy degradation under language mismatch. Incorporating modality-shared embedding spaces, supervised contrastive losses (e.g., SupCon, AAM), and language-agnostic pooling/attention proved effective for cross-lingual generalization (Moscati et al., 23 Dec 2025, Simic et al., 4 Dec 2025).
• Role of Pretraining and Regularization: Pretraining on large-scale, multi-language corpora (e.g., VoxCeleb2) improves generalization, while high-dimensional base embeddings require aggressive regularization (dropout $x_f^i$8) when adapting moderately sized task-specific mapping heads.
• Fusion and Orthogonality: Explicitly designed orthogonality losses and channel-wise gating encourage complementary information across modalities, avoiding over-reliance on a single modality's bias in the presence of shifting linguistic content (Moscati et al., 23 Dec 2025).

7. Open Problems and Future Directions

The FAME 2026 challenge highlights several avenues for future research:

• Extension to more languages, particularly low-resource and tonal languages, to test scalability of proposed methodologies.
• Modality-robust retrieval and identification, including performance on tasks where only one modality is available or is significantly degraded.
• Dialectal variation, code-switching, and sequential/temporal modeling are future evaluation targets.
• Dataset augmentation with further annotation, domain shift variables, and adversarial factors (e.g., synthetic samples, occlusions).
• Refinement of loss functions and architecture modularity to further minimize cross-modal, cross-lingual mismatch (Moscati et al., 23 Dec 2025, Simic et al., 4 Dec 2025).

Interdisciplinary research may further benefit by incorporating findings from related biometric and speech recognition competitions, particularly concerning the disentanglement of identity from linguistic content, robustness to real-world acoustic and visual nuisances, and advancement of cross-modal self-supervised pretraining (Moscati et al., 6 Aug 2025).

---

References:

(Moscati et al., 23 Dec 2025, Moscati et al., 6 Aug 2025, Simic et al., 4 Dec 2025)