---
title: FAME 2026 Challenge
url: https://www.emergentmind.com/topics/fame-2026-challenge
type: topic
---

# FAME 2026 Challenge

The Face-Voice Association in Multilingual Environments (FAME) 2026 Challenge is a benchmark designed to advance methodology in cross-modal biometric association, specifically targeting the verification and retrieval of person identity from paired facial images and voice utterances under multilingual and cross-lingual conditions. The challenge addresses the open research question of how language mismatch between training and test data impacts the robustness of face-voice association, reflecting the reality that a majority of the global population regularly communicates in multiple languages. The FAME 2026 challenge was organized as a special event at ICASSP 2026 and constitutes a rigorous testbed for both algorithmic innovation and systematic error analysis in multimodal person verification and retrieval across unseen languages [2512.20376, 2508.04592, 2512.04814].

## 1. Objectives and Task Definition

The principal objective is to develop and evaluate cross-modal verification methods that can associate a face image with a speech segment and ascertain whether they belong to the same identity—even when the language at test time has not been encountered during training (so-called "unheard" languages). The challenge quantifies the degradation in association performance incurred by a language mismatch.

Two primary task scenarios structure the competition:
- **Development Phase**: Methods are trained on a multi-speaker, bilingual dataset (English + German) and evaluated on languages seen during training, creating an in-language test scenario.
- **Evaluation Phase**: Models are required to generalize to evaluation on a split of speakers who appear exclusively in an "unheard" language, holding out both identity and language to stress-test cross-language robustness.

An optional retrieval extension shifts the setting from binary verification to ranking by similarity for face–voice matching queries [2512.20376].

Formally, the core verification problem can be described as learning a function $s(x_f^i, x_v^j)$ that, given a face embedding $x_f^i$ and a voice embedding $x_v^j$, outputs a similarity score indicating whether samples $i$ and $j$ are of the same identity, with higher scores suggesting greater confidence in identity match [2508.04592, 2512.04814].

## 2. Dataset and Multilingual Protocol

The official dataset is MAV-Celeb ("Multilingual Audio–Visual Celebrities"), sourced from YouTube interviews with robust variation in pose, lighting, background conditions, and artifacts. Key dataset facts [2508.04592, 2512.20376, 2512.04814]:

- **Composition**: For the English–German split (V3-EG), 58 bilingual speakers contributed multiple face images (from video frames) and speech utterances (≥1 s duration); utterances totaled 3,812 across 428 bilingual videos.
- **Splits**:
  - 50 training speakers (used for development/fine-tuning)
  - 8 test speakers (used solely for final evaluation, disjoint from train identities)
- **Language Annotation**: Each video is strictly monolingual; bilingual speakers ensure robust variance in cross-language appearance.
- **Preprocessing**:
  - Face images undergo detection, alignment, resizing (224×224), and pixel normalization.
  - Voice clips are resampled to 16 kHz, voice-activity detected, features extracted as 64-dimensional mel-filterbanks, and normalized.
- **Auxiliary Datasets**: VoxCeleb2 and multilingual CommonVoice subsets are used for both pre-training (with language-exclusion to preserve "unheard" split protocol) and for training auxiliary age/gender predictors [2512.04814].

A pivotal protocol constraint forbids any pre-training or fine-tuning on data from "unheard" languages in the held-out split, enforcing a pure cross-lingual generalization regime in evaluation [2508.04592].

## 3. Evaluation Metrics and Protocol

Evaluation focuses on the ability of models to correctly verify identity under both language-matched and language-mismatched conditions. The principal metrics are as follows:

- **Equal Error Rate (EER)**: This is the threshold $\tau^*$ where the false acceptance rate (FAR) and false rejection rate (FRR) are equal:
  \[
    \text{FAR}(\tau) = \frac{\#\{\text{impostor pairs with } s \geq \tau\}}{\#\{\text{impostor trials}\}}, \qquad
    \text{FRR}(\tau) = \frac{\#\{\text{genuine pairs with } s < \tau\}}{\#\{\text{genuine trials}\}}
  \]
  \[
    \text{EER} = \text{FAR}(\tau^*) = \text{FRR}(\tau^*)
  \]
- **Retrieval@K** (optional): Computes the proportion of queries where the correct match is ranked within the top $K$ returned items:
  \[
  \text{M@K} = \frac{1}{N} \sum_{i=1}^N \mathbb{I}(r_i \le K)
  \]
  where $r_i$ is the rank of the correct item for query $i$, and $N$ is the number of queries [2512.20376].
- **Area under ROC Curve (AUC)** and mean Average Precision (mAP) are also tracked in baseline reports [2508.04592].

Challenge protocol enforces strict evaluation phases:
- **Progress Phase**: Public dev leaderboard with submissions per language split, up to 100 submissions (max 10/day).
- **Evaluation Phase**: Held out test on “unheard” language, up to 5 submissions [2508.04592].

## 4. Baseline Systems and Leading Approaches

The baseline, available as public code, is a "Fusion & Orthogonal Projection" (FOP) pipeline [2512.20376, 2508.04592]:
- **Architecture**: Two-branch network; face images processed via CNN (e.g., Parkhi et al.’s Deep Face/ResNet-50), voices via an utterance-level audio encoder (e.g., Xie et al.’s ResNet).
- **Embedding**: Each modality embedding is projected to a 512-dimensional joint space, then fused (concatenation or learned gating).
- **Loss**: Combines cross-entropy for speaker classification with an orthogonal projection loss:
  \[
    L_{\text{OP}} = \lVert (W_f^\top W_v) - I \rVert_F^2
  \]
  where $W_f$ and $W_v$ are projection matrices for each modality, and $I$ is the identity [2512.20376].

Top challenge entries adopted a range of advanced architectures:
  
| Rank | Team           | Key Methods                                                      | EER (%)   |
|------|----------------|------------------------------------------------------------------|-----------|
| 1    | Simicch        | Dual transformers (face/voice), shared embedding, AAM/SupCon loss| 23.99     |
| 2    | Areffarhadi    | Foundation models fine-tuned for cross-modal/lingual invariance  | 24.73     |
| 3    | Alpha_code     | Attention-based fusion, refined orthogonality loss               | 33.11     |
| 4    | LTINI          | Contrastive gated fusion (dynamic channel gating)                | 33.18     |
| 5    | Punkmale       | Margin-based cross-modal alignment                              | 33.51     |
| -    | Baseline (FOP) | Fusion & Orthogonal Projection                                  | 41.57     |

- **Simicch et al. [2512.04814]**: Used ECAPA-TDNN and VGGFace/ViT for uni-modal embeddings, concatenated with age/gender features. Mapped 7680-dim audio and 4864-dim visual features by joint linear projections into a 192-dim shared embedding. Aggressive dropout ($p=0.9$) was applied before linear maps due to high base dimensionality. Trained with Adaptive Angular Margin (AAM) loss, which encourages small angular separation for same-identity and orthogonality for different identities. Pre-trained on VoxCeleb2, fine-tuned on MAV-Celeb under split constraints.
- **Areffarhadi et al.**: Fine-tuned large CLIP-style audio-visual encoders, leveraging language-agnostic pooling to mitigate language mismatch.
- **Alpha_code**: Refined FOP via attention-based multimodal fusion and subspace-weighted orthogonality.
- **LTINI and Punkmale**: Employed dynamic, contrastive, and margin-based objectives to explicitly structure cross-modal embedding space for robustness.

## 5. Results and Comparative Analysis

Aggregate results demonstrate a clear hierarchy in cross-lingual generalization:

- **Baseline (FOP) EER: 41.57%** (nearly doubled from in-language to unheard language within the same protocol).
- **Top method (Simicch et al.) EER: 23.99%**; runner-up achieved 24.73%.
- **Variance**: Simicch et al. also displayed low submission-to-submission variance, indicating methodological stability.
- **Per-language performance** (from Simicch et al. [2512.04814] Table 1): English test set—30.6% (English-trained), 30.1% (German-trained); German test set—17.4% (English-trained), 17.9% (German-trained).
- **Reduction in language-mismatch gap**: Leading entries were able to reduce the degradation under language mismatch by 40–45% relative to the baseline, demonstrating effectiveness of contrastive and embedding-alignment objectives.

A plausible implication is that joint embedding spaces and leveraged pretraining on large, diverse audio-visual corpora are crucial for robust cross-language identity association.

## 6. Technical and Methodological Insights

Three methodological themes emerge:
- **Robustness to Language Mismatch**: Naively trained models experience severe accuracy degradation under language mismatch. Incorporating modality-shared embedding spaces, supervised contrastive losses (e.g., SupCon, AAM), and language-agnostic pooling/attention proved effective for cross-lingual generalization [2512.20376, 2512.04814].
- **Role of Pretraining and Regularization**: Pretraining on large-scale, multi-language corpora (e.g., VoxCeleb2) improves generalization, while high-dimensional base embeddings require aggressive regularization (dropout $p=0.9$) when adapting moderately sized task-specific mapping heads.
- **Fusion and Orthogonality**: Explicitly designed orthogonality losses and channel-wise gating encourage complementary information across modalities, avoiding over-reliance on a single modality's bias in the presence of shifting linguistic content [2512.20376].

## 7. Open Problems and Future Directions

The FAME 2026 challenge highlights several avenues for future research:
- **Extension to more languages**, particularly low-resource and tonal languages, to test scalability of proposed methodologies.
- **Modality-robust retrieval and identification**, including performance on tasks where only one modality is available or is significantly degraded.
- **Dialectal variation, code-switching, and sequential/temporal modeling** are future evaluation targets.
- **Dataset augmentation** with further annotation, domain shift variables, and adversarial factors (e.g., synthetic samples, occlusions).
- **Refinement of loss functions and architecture modularity** to further minimize cross-modal, cross-lingual mismatch [2512.20376, 2512.04814].

Interdisciplinary research may further benefit by incorporating findings from related biometric and speech recognition competitions, particularly concerning the disentanglement of identity from linguistic content, robustness to real-world acoustic and visual nuisances, and advancement of cross-modal self-supervised pretraining [2508.04592].

---

**References:**  
[2512.20376]  
[2508.04592]  
[2512.04814]

Source: https://www.emergentmind.com/topics/fame-2026-challenge