---
title: Faetar Automatic Speech Recognition Benchmark
url: https://www.emergentmind.com/topics/faetar-automatic-speech-recognition-benchmark
type: topic
---

# Faetar Automatic Speech Recognition Benchmark

Searching arXiv for the specified benchmark and follow-up normalization paper to ground the article in the current literature.
Faetar Automatic Speech Recognition Benchmark is a phone-level automatic speech recognition benchmark for Faetar, an endangered and very under-resourced Franco‑Provençal variety spoken primarily in Faeto and Celle di San Vito in southern Italy, with a small diaspora in Canada. It was introduced to study low-resource ASR under conditions that are atypical of standard benchmarks: very limited labelled data, noisy field recordings, heterogeneous recording conditions, variable forced alignment quality, no standardized orthography, and quasi-phonetic transcriptions that mix phonetic and phonemic or lexical decisions [2409.08103]. A subsequent analysis examined whether transcription inconsistencies are the principal source of difficulty and concluded that, although such inconsistencies exist, they are not the main obstacle; normalization to canonical headwords can in fact make the task harder, while finite-lexicon decoding may help more than word-level bigram language modelling [2508.11771].

## 1. Linguistic setting and benchmark rationale

Faetar is described as a variety of Franco‑Provençal, within the southern Gallo‑Romance branch, and not as a variety of Italian, despite intense contact with Italian and its being spoken in Italy [2409.08103]. The speech community is small: the combined population of Faeto and Celle is reported as less than 1000, and the variety is endangered due to population loss and language shift to Italian. Heritage speakers are also present in the Greater Toronto Area [2409.08103].

The benchmark is motivated by the fact that Faetar has virtually no existing textual or speech resources beyond the benchmark itself, no standard orthography, and no large public dictionaries or ASR resources [2409.08103]. The available transcriptions are quasi-phonetic and somewhat inconsistent, partly because they were produced for linguistic analysis rather than ASR. The benchmark therefore targets what the literature characterizes as “dirty” low-resource ASR: settings with little labelled data, noisy and heterogeneous audio, substantial phonetic and phonological variability, and non-standardized transcription practice [2508.11771].

This design distinguishes Faetar from more conventional ASR benchmarks. Because Faetar lacks a codified spelling system and the available labels are primarily phonetic, the task is framed mainly as phone recognition and evaluated chiefly with phone error rate rather than standard orthographic word recognition [2409.08103]. This suggests that the benchmark is intended not only as an engineering task but also as a test case for documentation-oriented ASR in endangered-language settings.

## 2. Corpus composition, recording conditions, and annotation pipeline

The benchmark is built from the Faetar portion of the Heritage Language Variation and Change in Toronto corpus [2409.08103]. The corpus comprises approximately 5 hours of labelled speech with matching transcriptions and approximately 20 hours of unlabelled speech. After alignment, filtering, and dataset splitting, the detailed durations are reported as 4:30:17 for labelled train, 0:11:49 for dev, 0:46:54 for test, and 19:55:21 for the unlabelled set [2409.08103].

The recordings come from field data rather than studio speech. The homeland subset consists of 184 recordings of native speakers in Faeto from 1992–1994, while the heritage subset consists of 37 recordings of first- and second-generation heritage speakers in Toronto from 2009–2010 [2409.08103]. The recordings include interview speech and a “Words” condition elicited from a picture book. They are saved at 44.1 kHz, and homeland recordings were digitized from analog cassette tapes. The recordings are often noisy, with background noise, back-channels, overlapping speech, and interruptions [2409.08103].

The benchmark uses only interview speech in dev and test in order to avoid the over-simplified Words condition, which consists largely of single words or short phrases. Training includes both Interview and Words material. Additional 1h and 10min subsets are carved from train for comparability with ML‑SUPERB-style low-resource evaluation [2409.08103].

Transcriptions come from ELAN annotation files and Microsoft Word documents using specialized IPA fonts. These were converted to UTF‑8 by mapping font-specific glyph codes to IPA or Unicode and removing non-transcription material [2409.08103]. For recordings without pre-existing alignment, the benchmark creators trained an initial monophone speaker-independent GMM‑HMM in Kaldi, composed with a character-level 5-gram language model over phonetic transcripts, then used diarization, segmentation, alignment, stitching, and PyAnnote 3.0 voice activity detection to refine utterance boundaries and speaker labels [2409.08103]. The test set received an additional manual correction pass for boundaries and obvious transcript problems.

Filtering removed interviewer speech, utterances shorter than 500 ms, and utterances judged substantially Italian or English using closed-word-list criteria. The stated goal was to produce a more homogeneous Faetar corpus with minimal code-switching, thereby simplifying ASR evaluation and interpretation [2409.08103]. The alignment procedure is explicitly described as error-prone, and alignment quality is variable, which is itself part of the benchmark’s realism.

## 3. Label space, transcription regime, and evaluation protocol

The benchmark is defined as a single-language, phone-level ASR task. Systems take Faetar audio segments from dev or test and produce a 1-best phone sequence over a 68-phone inventory, with no spaces in the output [2409.08103]. The main evaluation metric is phone error rate. The benchmark paper defines PER over a set of utterances using Levenshtein alignment with unit-cost substitutions, deletions, and insertions:

$$
\text{PER}_{\mathcal{U}} = \frac{\sum_{u \in \mathcal{U}} e_u}{\sum_{u \in \mathcal{U}} n_u} \times 100\%
$$

and equivalently in aggregate form as

$$
\text{PER} = \frac{S + D + I}{N} \times 100\%
$$

where $S$, $D$, and $I$ are total substitutions, deletions, and insertions, and $N$ is the total number of reference phones [2409.08103]. The normalization paper also presents PER in the standard form and additionally uses word error rate on normalized word sequences [2508.11771].

The 68-phone inventory is derived from a phoneme inventory based on Nagy (2000), adapted to reflect the phonetic richness of the transcriptions [2409.08103]. The inventory includes consonants such as /p b m f v t d n r s ts l ʃ tʃ dʒ ɲ j ʎ k g ŋ w/ and vowels /i u e ə o a/, together with geminate consonants marked by a length symbol, doubled vowels used to represent hiatus, and additional phones introduced in transcription, including [h], [2], and [2:] [2409.08103]. All consonants can geminate, vowel length is not phonemic, and diphthongs are transcribed as vowel-plus-glide sequences [2409.08103].

The transcription regime is mixed phonetic and phonemic. Some words are transcribed in ways that reflect observed pronunciation, while others are transcribed more abstractly, closer to a phonemic or lexical form [2409.08103; 2508.11771]. The normalization study gives this mixed-level representation a central role: the labels are not standard orthography, not purely lexical, and not consistently standardized, so the benchmark simultaneously exposes acoustic modelling difficulties and representational ambiguities [2508.11771].

Secondary metrics include WER and CER, but the benchmark paper states that WER is too harsh and unstable because the transcripts are quasi-phonetic and include phonetic variation, while CER correlates strongly with PER and differs only by a few percentage points [2409.08103]. The emphasis on PER is therefore methodological as well as practical.

## 4. Baseline systems and reported performance

The benchmark paper provides both constrained and unconstrained baselines [2409.08103]. Constrained systems do not use external data or multilingual pretrained models. These include a monophone speaker-independent GMM‑HMM and a triphone speaker-dependent GMM‑HMM in Kaldi, both composed with a 5-gram character-level modified Kneser‑Ney language model trained on Faetar transcriptions. A small neural baseline in ESPnet uses the ML‑SUPERB ASR recipe with filterbank input features, a one-layer convolutional front-end, two transformer layers, CTC loss, and effective batch size 4 [2409.08103].

Unconstrained baselines use multilingual self-supervised foundation models. The paper fine-tunes MMS, based on wav2vec 2.0 with language-specific adapters and heads, initializing Faetar from an Italian-fine-tuned MMS model, and also fine-tunes the third iteration of mHuBERT‑147 [2409.08103]. For both, the fine-tuning configuration uses Faetar train only, a linear scheduler, peak learning rate $1 \times 10^{-5}$, 200 epochs, 10 warmup epochs, 10% dropout, effective batch size 8, and phone classification with CTC decoding [2409.08103].

The unlabelled Faetar audio is used in three MMS configurations: continued self-supervised pre-training on the 19:55:21 unlabelled set, self-training with pseudo-labels decoded from unlabelled data, and a combined PT + ST setup [2409.08103]. Continued pre-training uses effective batch size 32, 300,000 steps, 30% warmup, and peak learning rate $3 \times 10^{-4}$ before fine-tuning [2409.08103].

The following table summarizes the principal test-set PER figures reported in the benchmark paper.

| System | Setting | Test PER |
|---|---|---:|
| HMM–GMM monophone + 5-gram LM | Constrained | 62.6% ± 0.8 |
| HMM–GMM triphone + 5-gram LM | Constrained | 56.7% ± 0.9 |
| ESPnet CTC | Constrained | 35.8% ± 0.8 |
| MMS fine-tuning | Unconstrained | 33.0% ± 0.8 |
| mHuBERT fine-tuning | Unconstrained | 33.6% ± 0.8 |
| MMS PT + FT | Unconstrained + unlabelled audio | 31.5% ± 0.8 |
| MMS ST | Unconstrained + unlabelled audio | 31.0% ± 0.8 |
| MMS PT + ST | Unconstrained + unlabelled audio | 30.4% ± 0.8 |

The best reported result is a test PER of 30.4% ± 0.8 with MMS PT + ST [2409.08103]. The 1h and 10min ESPnet conditions obtain 37.4% ± 0.8 and 45.1% ± 0.8 test PER respectively [2409.08103]. The triphone GMM‑HMM reaches 9.6% ± 0.4 train PER but 56.7% ± 0.9 test PER, which the paper interprets as severe overfitting [2409.08103].

These results support three conclusions stated in the benchmark paper. First, small neural CTC models substantially outperform classical HMM‑GMM systems even in very low-resource conditions. Second, multilingual self-supervised pre-training yields an additional improvement of roughly 2–3 absolute PER points over the constrained ESPnet model. Third, the 20 hours of unlabelled Faetar speech are useful both for continued pre-training and for self-training, with the combination performing best [2409.08103].

## 5. Transcription inconsistency, normalization, and lexical abstraction

The normalization study re-examines the benchmark by asking whether transcription inconsistency is the fundamental cause of the difficulty [2508.11771]. It characterizes the benchmark transcriptions as phonetic rather than lexical, mixing phonetic detail with more abstract word-level or phonemic decisions, and lacking systematic standardization. “Transcription normalization” is defined as mapping heterogeneous surface transcriptions to canonical headwords, thereby collapsing multiple phonetic variants of the same lexical item into a single more abstract representation [2508.11771].

To study this, the authors construct a small, expert-curated lexicon for the test set and the 1-hour training subset. They begin with a tentative lexicon produced by iterated clustering that combines Word2Vec embeddings of transcriptions with HuBERT acoustic embeddings, then manually edit it to define headwords [2508.11771]. Tokens are mapped to lexical classes wherever possible, yielding both normalized transcriptions and the original “dirty phonetic” transcriptions.

The lexicon statistics show heavy variant proliferation. In the test set, 9119 tokens and 2509 unique word forms are grouped into 1742 variants, of which 1269 are singletons. In the 1-hour train subset, 10445 tokens and 2769 unique word forms are grouped into 1910 variants, of which 1414 are singletons [2508.11771]. The paper interprets this as evidence of substantial token-level variability and many variants appearing only once.

Using MMS‑1B with continued pretraining and fine-tuning on Faetar, the study compares training and evaluation on original versus normalized transcripts without a language model. The simplified results are as follows.

| Train / Test transcription regime | Test PER | Test WER |
|---|---:|---:|
| Original / Original | 31.6 | 79.1 |
| Normalized / Normalized | 36.2 | 82.3 |
| Original / Normalized | 33.8 | 80.4 |

These figures lead to the central conclusion that transcription inconsistencies are not the main source of difficulty in the Faetar benchmark [2508.11771]. Training and testing on normalized transcriptions is worse than training and testing on the original transcriptions, and training on original transcriptions while evaluating against normalized ones is better than training directly on normalized data. The stated interpretation is that fine-grained phonetic transcriptions provide richer supervision, whereas normalized labels require learning many-to-one mappings from acoustically distinct realizations to a single canonical representation in a severely data-limited regime [2508.11771].

A common misconception would be to treat the quasi-phonetic and inconsistent labels as mere annotation noise whose removal should lower error rates. The normalization results argue against that view. They indicate that some of what appears as “noise” is informative phonetic detail for model training, while lexical normalization introduces an additional abstraction burden [2508.11771].

## 6. Language modelling, lexicon-constrained decoding, and what remains difficult

The normalization paper also evaluates hybrid-style decoding with a finite lexicon and a word-based bigram language model using modified Kneser–Ney smoothing [2508.11771]. The design separates acoustic model training on the 1-hour training subset from language model training on the test-set transcripts, which is described as approximating an oracle language model. The paper writes the bigram model as $P(w_t \mid w_{t-1})$ and gives the standard hybrid decoding objective in terms of combined acoustic and language scores [2508.11771].

Three main decoding conditions are reported. With a full variant lexicon and bigram LM, the PER on the LM train set is 30.3 and the WER is 84.3; with no LM, PER on the LM train set is 31.6 and WER is 80.4; with a reduced lexicon containing only headwords and using fully normalized data, PER on the LM train set drops to 28.9 while WER remains 84.3 [2508.11771]. The paper concludes that bigram word-based language modelling provides no added benefit under these conditions, even when trained in this oracle-like fashion.

The reported explanation is that the modest PER gains come chiefly from constraining decoding to a finite lexicon rather than from modelling word-sequence predictability [2508.11771]. Word-level language modelling does not help choose the correct lexical item and can worsen WER. The paper attributes this to the very small amount of available text and poor overlap between the acoustic-model and language-model vocabularies, noting that 70% of words in the LM train set are not in the AM train set [2508.11771].

The reduced lexicon condition is particularly informative. Headwords are described as somewhat arbitrary, but they are the most frequent variant in about 80% of lexical entries with at least two variants [2508.11771]. Surprisingly, keeping only one canonical form per lexical entry lowers PER even though some actually occurring surface variants are excluded. The paper interprets this as evidence that variant-rich lexicons increase ambiguity, and that choosing among many near-synonymous forms often produces the wrong variant and therefore more phone mismatches [2508.11771].

After normalization and lexicon-based decoding, the task remains extremely difficult. The benchmark paper reports an unconstrained best test PER of 30.4% ± 0.8 and a constrained triphone HMM‑GMM test PER of 56.7% ± 0.9 [2409.08103], while the normalization study reports MMS-based systems hovering around PER 31–36 and WER around 80–82 in its evaluation settings [2508.11771]. The normalization paper explicitly states that error rates remain very high and speculates that audio quality plays a major role, alongside limited labelled data, acoustic variability, lexical sparsity, and the abstractness of lexical-level recognition [2508.11771].

This suggests that the benchmark’s difficulty is structural rather than incidental. The main obstacles are not simply inconsistent labels but the conjunction of minimal supervision, noisy field audio, phonetic and phonological variability, sparse lexical evidence, and the absence of substantial external text resources.

## 7. Significance, access model, and research directions

The benchmark is positioned as a depth-oriented complement to breadth-oriented multilingual evaluation suites such as SUPERB, ML‑SUPERB, FLEURS, and MLS [2409.08103]. Rather than averaging performance across many languages, it provides a single harsh test case in which the behavior of adaptation strategies, semi-supervised methods, and robust front ends can be examined under severe scarcity [2409.08103].

The intended uses include ASR research for very low-resource languages, comparison of continued pre-training and self-training, testing of speech enhancement and diarization methods, transfer learning from related languages such as Italian or French, and automatic transcription to support linguistic documentation and archival indexing [2409.08103]. The project page is reported as `https://perceptimatic.github.io/faetarspeech`, and release occurs in two phases: an embargoed challenge period in which test references are held by the organizers, followed by a full release including test transcriptions. Access is free but subject to an online data-sharing agreement [2409.08103].

The benchmark paper leaves several directions open to the research community: improved speaker modelling, speech enhancement, hybrid models, better alignment and annotation, more data collection, advanced semi-supervised methods, and cross-lingual transfer from related varieties beyond Italian-only initialization [2409.08103]. The normalization study adds further implications. It recommends caution toward assuming that transcription normalization will simplify the problem, suggests that simple phone-focused models may currently be more suitable than more elaborate two-level systems, and highlights explicit handling of the phone-recognition versus word-recognition split as an open problem [2508.11771].

Taken together, the two papers define the Faetar Automatic Speech Recognition Benchmark as a rigorous testbed for ASR in an endangered, orthography-less, very under-resourced language. Its empirical profile is unusually clear. Multilingual foundation models help; unlabelled in-domain audio helps more; finite-lexicon constraints can reduce phone errors; bigram word-level language modelling does not provide clear benefit in the reported setting; and transcription inconsistency, although real, is not the dominant explanation for the benchmark’s difficulty [2409.08103; 2508.11771]. The benchmark’s enduring value lies in making these interacting constraints measurable within a single, realistic corpus.

Source: https://www.emergentmind.com/topics/faetar-automatic-speech-recognition-benchmark