Syllable Homonym Error Rate (SHER) in ASR

• SHER is a metric that evaluates syllable recognition in ASR by considering syllables with the same articulatory attribute sequence as equivalent.
• It differentiates true pronunciation recognition errors from ambiguities due to inherent syllable homonymy in bottom-up ASR systems.
• SHER serves as a diagnostic tool to assess attribute-based models and guide improvements in disambiguation strategies.

Syllable Homonym Error Rate (SHER) is an evaluation metric for syllable-based automatic speech recognition (ASR) that measures recognition performance under syllable homonymy by treating syllables with the same underlying articulatory attribute sequence as equivalent. It was introduced in a bottom-up ASR framework that unifies language-universal articulatory attribute modeling with syllable-level prediction, where ambiguity arises because a single attribute sequence can correspond to multiple syllables. In that setting, SHER functions as a relaxed version of Syllable Error Rate (SER) and as a lower bound on the achievable SER of an attribute-based system, because it discounts errors attributable solely to ambiguity in the mapping from attributes to surface syllables rather than to failures of pronunciation recognition itself (Yen et al., 9 Sep 2025).

1. Origin and conceptual scope

SHER was introduced alongside Pronunciation Error Rate (PrER) in a bottom-up framework for ASR in syllable-based languages. The system first recognizes sequences or lattices of articulatory attributes as a language-universal, interpretable representation of pronunciation, and then transforms them into syllables through a structured knowledge integration process. Within that architecture, SHER is designed specifically to evaluate the model’s ability to handle syllable ambiguities that emerge when attribute posteriors are mapped to syllable sequences (Yen et al., 9 Sep 2025).

The metric is defined at the syllable level rather than at the word level. Its central premise is that multiple syllables may share identical underlying pronunciation attributes, especially when the attribute inventory lacks full discriminative power. Standard strict metrics count such cases as errors whenever the predicted syllable token differs from the reference token. SHER instead regards all syllables sharing the same underlying attribute sequence as equivalent, even if their surface orthographic forms differ (Yen et al., 9 Sep 2025).

This placement gives SHER a distinctive role in ASR evaluation. It is neither a generic replacement for syllable-level edit-distance metrics nor a purely phonetic metric. Rather, it quantifies syllable recognition under an equivalence relation induced by attribute sequences. In that sense, SHER is tailored to bottom-up and attribute-based syllable recognition systems, where the system may correctly recover pronunciation-relevant structure while remaining unable to resolve surface-form ambiguity without downstream knowledge (Yen et al., 9 Sep 2025).

2. Motivating ambiguity and diagnostic interpretation

The motivation for SHER is rooted in syllable ambiguity. In syllable-based languages such as Mandarin and Japanese, several syllables may share identical articulatory attribute sequences. The paper introducing SHER characterizes these as homonyms in the attribute space and observes that standard metrics such as SER or WER count them as errors even when they are indistinguishable from the ASR system’s perspective (Yen et al., 9 Sep 2025).

This ambiguity is especially salient in attribute-based modeling. A bottom-up system that predicts articulatory attributes and then maps them to syllables may be inherently unable to distinguish between homonymous syllables unless additional prosodic or contextual information is available. SHER therefore reveals how much error is due to actual attribute misrecognition versus ambiguity in converting correctly recognized attributes into surface syllables. The paper explicitly states that SHER serves as “the ideal case in which homonymous syllables are perfectly disambiguated in a downstream rescoring or mapping stage” (Yen et al., 9 Sep 2025).

The diagnostic value of SHER follows directly from this interpretation. If SHER remains high, then the attribute recognizer itself is weak. If SHER is substantially lower than SER, then a nontrivial fraction of apparent syllable errors is attributable to homonym mapping rather than acoustic failure. This makes SHER useful for model diagnosis, for assessing the theoretical lower bound of error given an attribute inventory, and for identifying when rescoring or contextual disambiguation could reduce surface-level error without improving the underlying pronunciation recognizer (Yen et al., 9 Sep 2025).

A common misconception is to treat SHER as a relaxed scoring heuristic with no substantive interpretation. The formulation in the introducing paper supports a more precise view: SHER is intended to isolate a specific source of irreducible ambiguity in attribute-based syllable recognition. It is therefore best understood as an analytic metric for systems constrained by many-to-one mappings from syllables to attribute sequences, not as a universal substitute for strict token accuracy (Yen et al., 9 Sep 2025).

3. Formal definition and computation

The formal definition of SHER is given using syllable sequences, attribute mappings, and edit distance. Let $S = (s_1, s_2, ..., s_N)$ denote the ground-truth sequence of syllables and $\hat{S} = (\hat{s}_1, \hat{s}_2, ..., \hat{s}_M)$ the hypothesis sequence produced by the ASR system. Let $a(s)$ denote the attribute sequence corresponding to syllable $s$, and let $\mathcal{E}$ denote the edit-distance function (Yen et al., 9 Sep 2025).

Two syllables are defined as equivalent if they have the same attribute sequence, that is, if $a(s) = a(\hat{s})$. SHER is then computed by mapping both the reference and hypothesis syllable sequences into their attribute-sequence representations, aligning those mapped sequences with standard Levenshtein edit distance, and normalizing by the number of reference syllables (Yen et al., 9 Sep 2025).

The metric is written as

$$\mathrm{SHER} = \frac{\mathcal{E}\left( [a(s_1), ..., a(s_N)],\, [a(\hat{s}_1), ..., a(\hat{s}_M)] \right)}{N}$$

where the compared unit is the attribute sequence associated with each syllable rather than the surface syllable label itself (Yen et al., 9 Sep 2025).

The computational procedure described for SHER consists of four steps. First, map each reference syllable and each hypothesis syllable to its corresponding attribute sequence. Second, define equivalence classes by exact equality of attribute sequences. Third, compute alignment and errors over these mapped sequences rather than over the surface syllable tokens. Fourth, normalize the resulting edit distance by the number of reference syllables. Under this definition, a substitution between two different written syllables does not count as an error if both syllables share the same attribute mapping (Yen et al., 9 Sep 2025).

This formulation places the burden of distinguishability on the attribute representation. SHER therefore evaluates syllable recovery subject to the information preserved by the attribute inventory, rather than subject to strict orthographic identity. A plausible implication is that the metric is especially sensitive to the granularity and discriminative capacity of the attribute design, because coarser inventories induce larger homonym equivalence classes.

4. Relation to SER, WER, PrER, and phonetic alignment metrics

The primary contrast for SHER is with strict syllable-level edit-distance metrics. In the introducing paper, SER is defined as

$$\mathrm{SER} = \frac{\mathcal{E}([s_1,\ldots,s_N], [\hat{s}_1, ..., \hat{s}_M])}{N}$$

and counts any surface-form mismatch as an error. By construction, SHER is always less than or equal to SER because surface distinctions that collapse to the same attribute sequence are not penalized under SHER (Yen et al., 9 Sep 2025).

PrER occupies a different level of analysis. It evaluates at the pronunciation or attribute level and measures errors between predicted and reference attribute sequences, potentially at a finer temporal or phoneme-like granularity. The introducing paper positions SHER between PrER and SER: PrER asks whether pronunciation attributes were recognized correctly, SHER asks whether syllables were reconstructed correctly while allowing for homonyms, and SER asks whether the exact surface syllable labels were recovered (Yen et al., 9 Sep 2025).

A broader comparison is useful:

Metric	Compared representation	Treatment of homonymous syllables
SER / syllable-level WER	Surface syllable tokens	Different tokens are errors
SHER	Attribute-mapped syllable sequences	Same attribute sequence is correct
PrER	Pronunciation / attribute sequences	Surface syllable mapping is not primary

The need for such distinctions is visible in earlier syllable recognition work. A comparison of hybrid and end-to-end models for German syllable recognition evaluates only “WER w.r.t. the syllables,” using the standard edit-distance formula $(S + D + I)/N$, and explicitly does not define or compute SHER. In that evaluation regime, all substitutions, deletions, and insertions are counted equally, regardless of whether the mismatch reflects homonymic ambiguity or other acoustic or modeling causes (Bayerl et al., 2019).

At the word-alignment level, the POWER metric provides a related but distinct perspective. POWER augments conventional WER with phoneme alignment and syllable boundaries, recombining aligned phonemes into aligned words to improve word alignment labels in error regions. It is designed to capture one-to-many word alignments corresponding to homophonic errors in speech recognition hypotheses, such as aligning “a day” with “today.” The summary accompanying that work explicitly states that POWER “can be seen as an operationalization of SHER” in the sense that it identifies phonetically and syllabically overlapping sequences and treats them as substitution spans rather than unrelated insertions and deletions (Ruiz et al., 2019).

The conceptual difference is that POWER operates through phonetic realignment of word spans, whereas SHER is defined directly on syllable sequences after mapping them into articulatory attribute equivalence classes. Both address homophony or homonymy, but they do so at different representational levels and for different diagnostic purposes.

5. Empirical behavior in syllable-based ASR

The introducing paper reports SHER on the AISHELL-1 Mandarin corpus for two systems: a direct syllable baseline (WavLM_syl) and a bottom-up attribute-based ASR system (WavLM_BU). The reported values are as follows (Yen et al., 9 Sep 2025):

System	SER (%)	SHER (%)
WavLM_syl	2.36	2.33
WavLM_BU (M+P+H+B+A)	2.47	2.12

These results are interpreted in the paper as showing that, for the direct syllable baseline, SER and SHER are nearly identical, implying that most errors are true recognition mistakes rather than homonym ambiguities. For the bottom-up system, SHER is lower than SER, indicating that a nontrivial fraction of surface-level syllable errors arises because the system captures the correct pronunciation attributes but cannot uniquely map them to the correct syllable label (Yen et al., 9 Sep 2025).

The paper also reports behavior under low-resource conditions. In that regime, the bottom-up approach exhibits lower SHER than SER consistently, and the gap between the two grows as training resources decrease. The stated interpretation is that more of the remaining errors could in principle be resolved by additional knowledge through biasing or rescoring, rather than by improving attribute recognition itself (Yen et al., 9 Sep 2025).

A qualitative example further clarifies the metric. The bottom-up system outputs the syllable xuan instead of the correct xian, but the two are treated as homonyms at the attribute level. Consequently, SER counts the substitution as an error, whereas SHER does not. This is presented as a case where the system has recovered the relevant pronunciation-level structure but not the exact surface syllable identity (Yen et al., 9 Sep 2025).

Taken together, these findings establish SHER as an empirical instrument for separating pronunciation recovery from surface disambiguation. In the reported experiments, its main value is not to replace SER, but to reveal the portion of strict syllable error attributable to ambiguity induced by the attribute representation.

6. Broader evaluation context and extension to speech-error analysis

SHER was introduced explicitly in the context of syllable-based ASR with articulatory attributes, but related work suggests how homonym-aware syllable evaluation can be extended to other ASR settings. In a study of WhisperX using the Simon Fraser University Speech Error Database (SFUSED), the database is described as systematically labeling speech errors involving sub-lexical sounds for both word position and syllable position, with syllable positions encoded as Onset, Nucleus, and Coda. It also distinguishes ASR outcomes as Faithful, Corrected, and Incorrect, and records intended and produced forms for each error (Alderete et al., 18 Aug 2025).

That paper does not explicitly mention homonym errors as a separate annotation label, and “Syllable Homonym Error Rate (SHER)” does not appear directly in the manuscript. Nevertheless, the provided summary states that the combination of intended versus produced forms, syllable-level error positions, and ASR output classifications provides the scaffolding for post hoc extraction of homonym confusions. In particular, cases in the Incorrect category could be examined to determine whether the ASR output is a syllabic or phonemic homonym of the intended or error word (Alderete et al., 18 Aug 2025).

The same summary notes that syllable position did not reach significance in determining transcription outcome for the analyzed error types, even though transcription accuracy was tabulated for Onset, Nucleus, and Coda positions. This is important because it shows that syllable-level annotation and syllable-level homonym analysis are not the same thing. A system can support fine-grained positional diagnostics without yet defining a formal homonym-aware metric (Alderete et al., 18 Aug 2025).

Within this broader context, SHER can be understood as one formal answer to a general evaluative problem: how to score ASR outputs when the recognized unit is ambiguous under the representational assumptions of the model. In attribute-based syllable ASR, the equivalence relation is induced by articulatory attribute sequences. In phonetic realignment approaches such as POWER, the relevant notion is phonetic and syllabic overlap between word spans. In conventional syllable-recognition benchmarks based only on syllable-level WER, such ambiguity is left unmodeled (Yen et al., 9 Sep 2025).

This suggests a broader methodological role for SHER-like evaluation. Where the representational layer of an ASR system collapses distinctions among surface syllables, strict token-level metrics alone can conflate modeling limitations, acoustic confusions, and unavoidable ambiguity. SHER separates these factors by design and thereby provides a technically precise lower-bound view of syllable recognition performance under homonymy.