Faetar Automatic Speech Recognition Benchmark

• The paper introduces a benchmark for Faetar ASR that tackles challenges like limited labeled data, noisy field recordings, and quasi-phonetic transcriptions.
• It employs methods ranging from classical GMM-HMM to modern self-supervised models, achieving PER improvements from over 60% down to around 30%.
• The study reveals that transcription inconsistencies are less critical than audio quality and data scarcity, emphasizing realistic conditions in endangered language ASR.

Searching arXiv for the specified benchmark and follow-up normalization paper to ground the article in the current literature. Faetar Automatic Speech Recognition Benchmark is a phone-level automatic speech recognition benchmark for Faetar, an endangered and very under-resourced Franco‑Provençal variety spoken primarily in Faeto and Celle di San Vito in southern Italy, with a small diaspora in Canada. It was introduced to study low-resource ASR under conditions that are atypical of standard benchmarks: very limited labelled data, noisy field recordings, heterogeneous recording conditions, variable forced alignment quality, no standardized orthography, and quasi-phonetic transcriptions that mix phonetic and phonemic or lexical decisions (Ong et al., 2024). A subsequent analysis examined whether transcription inconsistencies are the principal source of difficulty and concluded that, although such inconsistencies exist, they are not the main obstacle; normalization to canonical headwords can in fact make the task harder, while finite-lexicon decoding may help more than word-level bigram language modelling (Peckham et al., 15 Aug 2025).

1. Linguistic setting and benchmark rationale

Faetar is described as a variety of Franco‑Provençal, within the southern Gallo‑Romance branch, and not as a variety of Italian, despite intense contact with Italian and its being spoken in Italy (Ong et al., 2024). The speech community is small: the combined population of Faeto and Celle is reported as less than 1000, and the variety is endangered due to population loss and language shift to Italian. Heritage speakers are also present in the Greater Toronto Area (Ong et al., 2024).

The benchmark is motivated by the fact that Faetar has virtually no existing textual or speech resources beyond the benchmark itself, no standard orthography, and no large public dictionaries or ASR resources (Ong et al., 2024). The available transcriptions are quasi-phonetic and somewhat inconsistent, partly because they were produced for linguistic analysis rather than ASR. The benchmark therefore targets what the literature characterizes as “dirty” low-resource ASR: settings with little labelled data, noisy and heterogeneous audio, substantial phonetic and phonological variability, and non-standardized transcription practice (Peckham et al., 15 Aug 2025).

This design distinguishes Faetar from more conventional ASR benchmarks. Because Faetar lacks a codified spelling system and the available labels are primarily phonetic, the task is framed mainly as phone recognition and evaluated chiefly with phone error rate rather than standard orthographic word recognition (Ong et al., 2024). This suggests that the benchmark is intended not only as an engineering task but also as a test case for documentation-oriented ASR in endangered-language settings.

2. Corpus composition, recording conditions, and annotation pipeline

The benchmark is built from the Faetar portion of the Heritage Language Variation and Change in Toronto corpus (Ong et al., 2024). The corpus comprises approximately 5 hours of labelled speech with matching transcriptions and approximately 20 hours of unlabelled speech. After alignment, filtering, and dataset splitting, the detailed durations are reported as 4:30:17 for labelled train, 0:11:49 for dev, 0:46:54 for test, and 19:55:21 for the unlabelled set (Ong et al., 2024).

The recordings come from field data rather than studio speech. The homeland subset consists of 184 recordings of native speakers in Faeto from 1992–1994, while the heritage subset consists of 37 recordings of first- and second-generation heritage speakers in Toronto from 2009–2010 (Ong et al., 2024). The recordings include interview speech and a “Words” condition elicited from a picture book. They are saved at 44.1 kHz, and homeland recordings were digitized from analog cassette tapes. The recordings are often noisy, with background noise, back-channels, overlapping speech, and interruptions (Ong et al., 2024).

The benchmark uses only interview speech in dev and test in order to avoid the over-simplified Words condition, which consists largely of single words or short phrases. Training includes both Interview and Words material. Additional 1h and 10min subsets are carved from train for comparability with ML‑SUPERB-style low-resource evaluation (Ong et al., 2024).

Transcriptions come from ELAN annotation files and Microsoft Word documents using specialized IPA fonts. These were converted to UTF‑8 by mapping font-specific glyph codes to IPA or Unicode and removing non-transcription material (Ong et al., 2024). For recordings without pre-existing alignment, the benchmark creators trained an initial monophone speaker-independent GMM‑HMM in Kaldi, composed with a character-level 5-gram LLM over phonetic transcripts, then used diarization, segmentation, alignment, stitching, and PyAnnote 3.0 voice activity detection to refine utterance boundaries and speaker labels (Ong et al., 2024). The test set received an additional manual correction pass for boundaries and obvious transcript problems.

Filtering removed interviewer speech, utterances shorter than 500 ms, and utterances judged substantially Italian or English using closed-word-list criteria. The stated goal was to produce a more homogeneous Faetar corpus with minimal code-switching, thereby simplifying ASR evaluation and interpretation (Ong et al., 2024). The alignment procedure is explicitly described as error-prone, and alignment quality is variable, which is itself part of the benchmark’s realism.

3. Label space, transcription regime, and evaluation protocol

The benchmark is defined as a single-language, phone-level ASR task. Systems take Faetar audio segments from dev or test and produce a 1-best phone sequence over a 68-phone inventory, with no spaces in the output (Ong et al., 2024). The main evaluation metric is phone error rate. The benchmark paper defines PER over a set of utterances using Levenshtein alignment with unit-cost substitutions, deletions, and insertions:

$$\text{PER}_{\mathcal{U}} = \frac{\sum_{u \in \mathcal{U}} e_u}{\sum_{u \in \mathcal{U}} n_u} \times 100\%$$

and equivalently in aggregate form as

$\text{PER} = \frac{S + D + I}{N} \times 100\%$

where $S$, $D$, and $I$ are total substitutions, deletions, and insertions, and $N$ is the total number of reference phones (Ong et al., 2024). The normalization paper also presents PER in the standard form and additionally uses word error rate on normalized word sequences (Peckham et al., 15 Aug 2025).

The 68-phone inventory is derived from a phoneme inventory based on Nagy (2000), adapted to reflect the phonetic richness of the transcriptions (Ong et al., 2024). The inventory includes consonants such as /p b m f v t d n r s ts l ʃ tʃ dʒ ɲ j ʎ k g ŋ w/ and vowels /i u e ə o a/, together with geminate consonants marked by a length symbol, doubled vowels used to represent hiatus, and additional phones introduced in transcription, including [h], [2], and 2:. All consonants can geminate, vowel length is not phonemic, and diphthongs are transcribed as vowel-plus-glide sequences (Ong et al., 2024).

The transcription regime is mixed phonetic and phonemic. Some words are transcribed in ways that reflect observed pronunciation, while others are transcribed more abstractly, closer to a phonemic or lexical form (Ong et al., 2024, Peckham et al., 15 Aug 2025). The normalization study gives this mixed-level representation a central role: the labels are not standard orthography, not purely lexical, and not consistently standardized, so the benchmark simultaneously exposes acoustic modelling difficulties and representational ambiguities (Peckham et al., 15 Aug 2025).

Secondary metrics include WER and CER, but the benchmark paper states that WER is too harsh and unstable because the transcripts are quasi-phonetic and include phonetic variation, while CER correlates strongly with PER and differs only by a few percentage points (Ong et al., 2024). The emphasis on PER is therefore methodological as well as practical.

4. Baseline systems and reported performance

The benchmark paper provides both constrained and unconstrained baselines (Ong et al., 2024). Constrained systems do not use external data or multilingual pretrained models. These include a monophone speaker-independent GMM‑HMM and a triphone speaker-dependent GMM‑HMM in Kaldi, both composed with a 5-gram character-level modified Kneser‑Ney LLM trained on Faetar transcriptions. A small neural baseline in ESPnet uses the ML‑SUPERB ASR recipe with filterbank input features, a one-layer convolutional front-end, two transformer layers, CTC loss, and effective batch size 4 (Ong et al., 2024).

Unconstrained baselines use multilingual self-supervised foundation models. The paper fine-tunes MMS, based on wav2vec 2.0 with language-specific adapters and heads, initializing Faetar from an Italian-fine-tuned MMS model, and also fine-tunes the third iteration of mHuBERT‑147 (Ong et al., 2024). For both, the fine-tuning configuration uses Faetar train only, a linear scheduler, peak learning rate $1 \times 10^{-5}$, 200 epochs, 10 warmup epochs, 10% dropout, effective batch size 8, and phone classification with CTC decoding (Ong et al., 2024).

The unlabelled Faetar audio is used in three MMS configurations: continued self-supervised pre-training on the 19:55:21 unlabelled set, self-training with pseudo-labels decoded from unlabelled data, and a combined PT + ST setup (Ong et al., 2024). Continued pre-training uses effective batch size 32, 300,000 steps, 30% warmup, and peak learning rate $3 \times 10^{-4}$ before fine-tuning (Ong et al., 2024).

The following table summarizes the principal test-set PER figures reported in the benchmark paper.

System	Setting	Test PER
HMM–GMM monophone + 5-gram LM	Constrained	62.6% ± 0.8
HMM–GMM triphone + 5-gram LM	Constrained	56.7% ± 0.9
ESPnet CTC	Constrained	35.8% ± 0.8
MMS fine-tuning	Unconstrained	33.0% ± 0.8
mHuBERT fine-tuning	Unconstrained	33.6% ± 0.8
MMS PT + FT	Unconstrained + unlabelled audio	31.5% ± 0.8
MMS ST	Unconstrained + unlabelled audio	31.0% ± 0.8
MMS PT + ST	Unconstrained + unlabelled audio	30.4% ± 0.8

The best reported result is a test PER of 30.4% ± 0.8 with MMS PT + ST (Ong et al., 2024). The 1h and 10min ESPnet conditions obtain 37.4% ± 0.8 and 45.1% ± 0.8 test PER respectively (Ong et al., 2024). The triphone GMM‑HMM reaches 9.6% ± 0.4 train PER but 56.7% ± 0.9 test PER, which the paper interprets as severe overfitting (Ong et al., 2024).

These results support three conclusions stated in the benchmark paper. First, small neural CTC models substantially outperform classical HMM‑GMM systems even in very low-resource conditions. Second, multilingual self-supervised pre-training yields an additional improvement of roughly 2–3 absolute PER points over the constrained ESPnet model. Third, the 20 hours of unlabelled Faetar speech are useful both for continued pre-training and for self-training, with the combination performing best (Ong et al., 2024).

5. Transcription inconsistency, normalization, and lexical abstraction

The normalization study re-examines the benchmark by asking whether transcription inconsistency is the fundamental cause of the difficulty (Peckham et al., 15 Aug 2025). It characterizes the benchmark transcriptions as phonetic rather than lexical, mixing phonetic detail with more abstract word-level or phonemic decisions, and lacking systematic standardization. “Transcription normalization” is defined as mapping heterogeneous surface transcriptions to canonical headwords, thereby collapsing multiple phonetic variants of the same lexical item into a single more abstract representation (Peckham et al., 15 Aug 2025).

To study this, the authors construct a small, expert-curated lexicon for the test set and the 1-hour training subset. They begin with a tentative lexicon produced by iterated clustering that combines Word2Vec embeddings of transcriptions with HuBERT acoustic embeddings, then manually edit it to define headwords (Peckham et al., 15 Aug 2025). Tokens are mapped to lexical classes wherever possible, yielding both normalized transcriptions and the original “dirty phonetic” transcriptions.

The lexicon statistics show heavy variant proliferation. In the test set, 9119 tokens and 2509 unique word forms are grouped into 1742 variants, of which 1269 are singletons. In the 1-hour train subset, 10445 tokens and 2769 unique word forms are grouped into 1910 variants, of which 1414 are singletons (Peckham et al., 15 Aug 2025). The paper interprets this as evidence of substantial token-level variability and many variants appearing only once.

Using MMS‑1B with continued pretraining and fine-tuning on Faetar, the study compares training and evaluation on original versus normalized transcripts without a LLM. The simplified results are as follows.

Train / Test transcription regime	Test PER	Test WER
Original / Original	31.6	79.1
Normalized / Normalized	36.2	82.3
Original / Normalized	33.8	80.4

These figures lead to the central conclusion that transcription inconsistencies are not the main source of difficulty in the Faetar benchmark (Peckham et al., 15 Aug 2025). Training and testing on normalized transcriptions is worse than training and testing on the original transcriptions, and training on original transcriptions while evaluating against normalized ones is better than training directly on normalized data. The stated interpretation is that fine-grained phonetic transcriptions provide richer supervision, whereas normalized labels require learning many-to-one mappings from acoustically distinct realizations to a single canonical representation in a severely data-limited regime (Peckham et al., 15 Aug 2025).

A common misconception would be to treat the quasi-phonetic and inconsistent labels as mere annotation noise whose removal should lower error rates. The normalization results argue against that view. They indicate that some of what appears as “noise” is informative phonetic detail for model training, while lexical normalization introduces an additional abstraction burden (Peckham et al., 15 Aug 2025).

6. Language modelling, lexicon-constrained decoding, and what remains difficult

The normalization paper also evaluates hybrid-style decoding with a finite lexicon and a word-based bigram LLM using modified Kneser–Ney smoothing (Peckham et al., 15 Aug 2025). The design separates acoustic model training on the 1-hour training subset from LLM training on the test-set transcripts, which is described as approximating an oracle LLM. The paper writes the bigram model as $P(w_t \mid w_{t-1})$ and gives the standard hybrid decoding objective in terms of combined acoustic and language scores (Peckham et al., 15 Aug 2025).

Three main decoding conditions are reported. With a full variant lexicon and bigram LM, the PER on the LM train set is 30.3 and the WER is 84.3; with no LM, PER on the LM train set is 31.6 and WER is 80.4; with a reduced lexicon containing only headwords and using fully normalized data, PER on the LM train set drops to 28.9 while WER remains 84.3 (Peckham et al., 15 Aug 2025). The paper concludes that bigram word-based language modelling provides no added benefit under these conditions, even when trained in this oracle-like fashion.

The reported explanation is that the modest PER gains come chiefly from constraining decoding to a finite lexicon rather than from modelling word-sequence predictability (Peckham et al., 15 Aug 2025). Word-level language modelling does not help choose the correct lexical item and can worsen WER. The paper attributes this to the very small amount of available text and poor overlap between the acoustic-model and language-model vocabularies, noting that 70% of words in the LM train set are not in the AM train set (Peckham et al., 15 Aug 2025).

The reduced lexicon condition is particularly informative. Headwords are described as somewhat arbitrary, but they are the most frequent variant in about 80% of lexical entries with at least two variants (Peckham et al., 15 Aug 2025). Surprisingly, keeping only one canonical form per lexical entry lowers PER even though some actually occurring surface variants are excluded. The paper interprets this as evidence that variant-rich lexicons increase ambiguity, and that choosing among many near-synonymous forms often produces the wrong variant and therefore more phone mismatches (Peckham et al., 15 Aug 2025).

After normalization and lexicon-based decoding, the task remains extremely difficult. The benchmark paper reports an unconstrained best test PER of 30.4% ± 0.8 and a constrained triphone HMM‑GMM test PER of 56.7% ± 0.9 (Ong et al., 2024), while the normalization study reports MMS-based systems hovering around PER 31–36 and WER around 80–82 in its evaluation settings (Peckham et al., 15 Aug 2025). The normalization paper explicitly states that error rates remain very high and speculates that audio quality plays a major role, alongside limited labelled data, acoustic variability, lexical sparsity, and the abstractness of lexical-level recognition (Peckham et al., 15 Aug 2025).

This suggests that the benchmark’s difficulty is structural rather than incidental. The main obstacles are not simply inconsistent labels but the conjunction of minimal supervision, noisy field audio, phonetic and phonological variability, sparse lexical evidence, and the absence of substantial external text resources.

7. Significance, access model, and research directions

The benchmark is positioned as a depth-oriented complement to breadth-oriented multilingual evaluation suites such as SUPERB, ML‑SUPERB, FLEURS, and MLS (Ong et al., 2024). Rather than averaging performance across many languages, it provides a single harsh test case in which the behavior of adaptation strategies, semi-supervised methods, and robust front ends can be examined under severe scarcity (Ong et al., 2024).

The intended uses include ASR research for very low-resource languages, comparison of continued pre-training and self-training, testing of speech enhancement and diarization methods, transfer learning from related languages such as Italian or French, and automatic transcription to support linguistic documentation and archival indexing (Ong et al., 2024). The project page is reported as https://perceptimatic.github.io/faetarspeech, and release occurs in two phases: an embargoed challenge period in which test references are held by the organizers, followed by a full release including test transcriptions. Access is free but subject to an online data-sharing agreement (Ong et al., 2024).

The benchmark paper leaves several directions open to the research community: improved speaker modelling, speech enhancement, hybrid models, better alignment and annotation, more data collection, advanced semi-supervised methods, and cross-lingual transfer from related varieties beyond Italian-only initialization (Ong et al., 2024). The normalization study adds further implications. It recommends caution toward assuming that transcription normalization will simplify the problem, suggests that simple phone-focused models may currently be more suitable than more elaborate two-level systems, and highlights explicit handling of the phone-recognition versus word-recognition split as an open problem (Peckham et al., 15 Aug 2025).

Taken together, the two papers define the Faetar Automatic Speech Recognition Benchmark as a rigorous testbed for ASR in an endangered, orthography-less, very under-resourced language. Its empirical profile is unusually clear. Multilingual foundation models help; unlabelled in-domain audio helps more; finite-lexicon constraints can reduce phone errors; bigram word-level language modelling does not provide clear benefit in the reported setting; and transcription inconsistency, although real, is not the dominant explanation for the benchmark’s difficulty (Ong et al., 2024, Peckham et al., 15 Aug 2025). The benchmark’s enduring value lies in making these interacting constraints measurable within a single, realistic corpus.