- The paper introduces a constrained, character-width-aware Needleman–Wunsch algorithm that preserves REF-HYP-EVAL alignment across Tamil, Hindi, Kannada, Russian, Greek, and Arabic, correcting cascading sclite misalignments.
- The corrected pipeline reveals language-specific PoS error patterns, including high noun and proper-noun error rates in Tamil and Russian and adjective and noun errors in Arabic, with substitutions dominating most errors.
- The paper applies empirical PoS error weights to Transformer cross-attention, reducing Tamil WER from 23.3% to 22.0%, Arabic WER from 41.6% to 41.4%, and Russian WER from 9.7% to 9.2%, while noting limitations for unsegmented scripts and unreliable taggers.
Motivation and problem statement
Aggregate metrics such as Word Error Rate (WER) summarize ASR performance without revealing which grammatical categories of tokens are being misrecognized. Part-of-Speech (PoS)-wise error analysis addresses this, but it presupposes a correct three-way alignment between the reference transcription (REF), the ASR hypothesis (HYP), and an evaluation string (EVAL) encoding substitution (S), insertion (I), and deletion (D) operations. The standard tool for this task, NIST's sclite, assumes uniform character widths and fixed padding conventions. The authors demonstrate that this assumption fails for non-Latin scripts: after any insertion or deletion, sclite's REF, HYP, and EVAL streams shift out of correspondence for Tamil, Hindi, Kannada, Russian, Greek, and Arabic, producing cascading misalignments that make automated PoS tagging of errors unreliable. For Latin-script languages such as English, Spanish, and German, the default alignment remains adequate. The paper's central contribution is a character-spacing-aware modified Needleman–Wunsch alignment algorithm that restores reliable REF-HYP-EVAL correspondence across both Latin and non-Latin scripts, enabling scalable, reproducible PoS-wise error analysis and, ultimately, PoS-aware model training.
Proposed alignment algorithm
The method operates at the word level. Given reference sequence R=(r1,…,rN) and hypothesis H=(h1,…,hM), a dynamic programming matrix is filled using a weighted Levenshtein recurrence with unit costs for substitution, insertion, and deletion, and zero cost for exact matches. Crucially, the search is constrained to admissible alignments consistent with the externally observed S/I/D tag sequence from the evaluation string; backtracking then recovers the minimum-cost path, with tie-breaking priority match > I > D > S.
Two modifications distinguish the approach from vanilla Needleman–Wunsch:
- Post-processing of edit operations: consecutive insertion-deletion pairs (I,D) or (D,I) are merged into a single substitution S, avoiding artificial inflation of error counts.
- Character-width-based spacing: instead of padding by raw character count, each aligned column is padded to max(width(r),width(h),width(o))+δ with δ=1, where width reflects rendered display size. This accounts for combining characters, vowel modifiers, contextual glyph shaping in Arabic cursive, and non-uniform Cyrillic widths — phenomena characteristic of Abugida, Abjad, and Alphabetic systems alike.
An ablation shows that adaptive spacing is essential: pure Needleman–Wunsch alignment on a Tamil example causes right-shift drift after a long placeholder token, collapsing later substitutions into invalid None/UNK mappings, whereas the proposed method preserves token boundaries throughout. Worked examples across Tamil, Russian, Arabic, Hindi, Kannada, and Greek show that sclite misalignments (e.g., aligning a deleted Tamil pronoun against an unrelated noun, or pairing Russian "Латинская" with a deletion) are corrected, yielding accurate per-error PoS assignments.
PoS-wise error analysis across writing systems
Using the corrected alignments, the authors tag errors with language-appropriate taggers (spaCy-Stanza for Tamil/Hindi, spaCy for Russian, AI4Bharat IndicBERTv2 for Kannada, CAMeL Tools for Arabic, gr-nlp-toolkit for Greek) over Universal PoS categories. Representative findings across the three segmented writing systems include:
| Language (system) |
Most error-prone categories |
Approximate error rates |
| Tamil (Abugida) |
proper nouns, nouns |
24.05% / 20.98% |
| Russian (Alphabetic) |
proper nouns, nouns |
6.88% / 4.56% |
| Arabic (Abjad) |
adjectives, nouns |
23.9% / 21.5% |
The authors attribute high nominal error rates to agglutinative morphology in Tamil, rich inflection in Russian, and root-pattern affixation in Arabic. Substitutions dominate nearly everywhere; deletions and insertions are comparatively rare. A cross-model comparison on Tamil reveals architecture-specific profiles: Transformer and LSTM produce balanced, substitution-dominated distributions, while the Conformer exhibits unusually high deletion rates (e.g., 61–72% deletion percentages for major categories) offset by near-zero insertions, indicating that stronger acoustic modeling does not guarantee better lexical retention in morphologically rich languages. On LibriSpeech test-clean versus test-other, content-bearing categories (nouns, verbs, adjectives) degrade disproportionately under acoustic difficulty, while functional words remain relatively stable.
The paper then demonstrates a concrete use of the error statistics: decoder cross-attention scores are reweighted by PoS-dependent scalars wPOS(j) derived from empirical error rates, so that attention assigns higher importance to error-prone syntactic categories during decoding. Training uses the standard joint CTC-attention objective with label smoothing (λCTC=0.3).
The reported gains are consistent but modest:
| Dataset |
Baseline WER/CER/TER/S.Err |
PoS-aware |
| Tamil |
23.3 / 8.7 / 16.2 / 73.9 |
22.0 / 8.3 / 15.2 / 67.1 |
| Arabic |
41.6 / 17.4 / 22.8 |
41.4 / 16.6 / 21.9 |
| Russian |
9.7 / 2.9 / 4.1 / 37.9 |
9.2 / 2.7 / 3.8 / 36.0 |
At the category level, proper-noun substitutions in Tamil drop from 2630 to 2419 (~8% relative reduction), verb substitutions fall ~5%, and Russian propn total errors decrease ~3.9%. Gains are not uniform: some low-frequency or function-word categories (e.g., adj, conj_sub, part_interrog in Arabic; adj, verb, sconj in Russian) show slight increases. The authors explicitly characterize the weighting scheme as an early demonstration rather than a tuned method.
Limitations
Three limitations are acknowledged directly. First, the algorithm presumes whitespace-delimited tokens and therefore does not apply to non-segmented scripts such as Chinese, Japanese, or Thai; extending it would require a segmentation stage whose own errors could propagate into alignment quality. Second, the framework inherits the accuracy of its PoS taggers — in Kannada, a large fraction of tokens were assigned the X category, indicating unreliable tagging that contaminates both the error statistics and the downstream PoS-aware training. Third, the PoS weighting strategy is presented as preliminary, leaving open how weights should be selected or learned rather than hand-set from empirical error counts.
Conclusion
This work identifies a concrete failure mode of sclite-based alignment for non-Latin scripts and supplies a general remedy via a constrained, character-width-aware Needleman–Wunsch alignment with post-hoc edit merging. The resulting pipeline enables automated PoS-wise ASR error analysis across Abugida, Alphabetic, and Abjad writing systems, and the derived statistics yield measurable WER improvements when injected into decoder cross-attention. Open questions include extension to unsegmented scripts, robustness to weak PoS taggers in low-resource settings, and principled optimization of the PoS weighting scheme beyond heuristic assignment.