Stringalign: Moving beyond summary statistics with a transparent Unicode-aware tool for evaluating automatic transcription models

Abstract: Comparing text strings is crucial when evaluating and understanding the performance of various text processing tasks such as document recognition and audio transcription. With an increasingly complex landscape of AI-based handwritten text recognition (HTR), optical character recognition (OCR) and automatic speech recognition (ASR) models, there is a need for tools that facilitate evaluation in a flexible and reproducible way. This paper presents Stringalign, a Python library designed to simplify the evaluation process for automatic transcription projects and facilitate transparent evaluation. Stringalign's tools to examine and visualise both the rate of errors and the types of errors a model makes, give insights into possible improvements and help inform model selection for a particular task. Widely used string comparison metrics, such as the character and word error rates (CER and WER), although useful, can be ambiguous due to varying definitions of what constitutes a character and a word. Stringalign addresses this challenge by ensuring all preprocessing (i.e. normalisation and tokenisation) is transparent and easily replicable, and by providing tools to move beyond summary statistics and analyse common model errors. Moreover, Stringalign adheres to FAIR (Findable, Accessible, Interoperable, and Reusable) principles for research software while staying lightweight and easy to adapt into researchers existing workflows. In this paper, we discuss challenges with character and word level string comparisons and show through examples that where existing tools can yield opaque and sometimes confusing results, Stringalign provides an easy-to-use and unambiguous alternative.

• The paper introduces Stringalign, a tool that transparently handles Unicode complexities and tokenization for accurate evaluation in ATR, OCR, and ASR tasks.
• It employs rigorous alignment methodologies with explicit normalization and tokenization to ensure reproducible error analysis and reliable benchmarking.
• The tool offers diagnostic visualizations and error inspection features that improve the analysis of complex Unicode and whitespace variations in transcriptions.

Stringalign: Transparent, Unicode-Aware Evaluation for Automatic Transcription Models

Motivation and Context

Evaluating output alignment quality in automatic transcription tasks—including Handwritten Text Recognition (HTR), Optical Character Recognition (OCR), and Automatic Speech Recognition (ASR)—is foundational for benchmarking, system selection, and diagnostic error analysis. Standard metrics such as Character Error Rate (CER) and Word Error Rate (WER) are widespread but frequently suffer from ambiguities related to character definitions (particularly for Unicode), preprocessing transparency, and tokenization specifics. These ambiguities propagate to model development, cross-system comparison, annotation process auditing, and downstream data-centric AI improvements. Existing tools and libraries addressing CER/ WER computation often lack transparent handling of Unicode complexities, reproduceability, or flexible tokenization.

Stringalign addresses these limitations by providing a fully transparent, Unicode-aware, and reproducible Python-based evaluation framework tailored to the contemporary landscape of ATR and ASR research.

Evaluation Metrics and String Ambiguity

Underlying string comparison—central to ATR and ASR evaluations—is the edit-based alignment paradigm. Using operations such as insertion, deletion, and replacement, various optimal alignments (e.g., Levenshtein, Needleman-Wunsch) are computed for reference and hypothesis strings. However, alignment is generally non-unique: multiple minimal edit alignments can exist, leading to variations in token-specific statistics like TPR, FDR, and $f_1$.

Stringalign implements:

• Canonical and user-defined normalization based on Unicode recommendations
• Support for grapheme cluster segmentation (beyond naive codepoint splitting)
• Extensible tokenization strategies (beyond whitespace or custom boundaries)
• Handling of confusables and normalization of compatibility equivalents

These features directly address biases and inconsistencies endemic in many published results.

Figure 1: Inputs and most commonly used attributes, methods and properties for the AlignmentAnalyzer.

Limitations of Existing Tooling

A comparative analysis of tools such as Calamari, Dinglehopper, ISRI, Jiwer, Meeteval, and ocrevalUAtion reveals significant limitations: lack of Unicode-awareness, closed or opaque processing pipelines, inflexible APIs, and minimal support for token-level error breakdown or custom segmentation. Many frameworks aggregate errors with undocumented normalization or tokenization, impeding reproducibility, particularly for non-Latin scripts, composite Unicode points, and corpora with extensive whitespace variation. Integration in arbitrary pipelines is often hampered by rigid dependencies and non-modular architecture.

Stringalign Design and Implementation

Stringalign’s API is rooted in explicit, standardized Unicode handling via modules for tokenization, normalization, statistics, visualization, and alignment. Notably, it heavily leverages robust Unicode segmentation and normalization specifications (NFC, NFD, NFKC, NFKD), and exposes all preprocessing steps, facilitating rigorous experiments and FAIR (Findable, Accessible, Interoperable, Reusable) research software practices.

Core design attributes include:

• Lightweight/minimal dependencies (only NumPy at runtime)
• Transparent, modular architecture with comprehensive unit tests and in-depth documentation
• Support for stochastic and exhaustive enumeration of multiple optimal alignments with quantification of uncertainty in edit statistics
• Native Python API, with built-in visualization and introspection, designed for seamless use in data-centric research workflows

Illustrative Examples: Resolving Ambiguities

Stringalign is empirically compared to six widely-used tools on both synthetic and real-world (IMPACT dataset) examples. Discrepancies in error rates across tools are shown to arise from differences in Unicode handling, segmentation policy, and reporting format.

Results demonstrate:

• For modern Unicode (e.g., emojis, combining characters) and whitespace edge cases, only Stringalign and Dinglehopper yield correct CER/ WER consistent with grapheme/ word boundaries.
• For large-scale real-world datasets, macro- versus micro-averaging and undetected confusable handling induce nontrivial deviations in CER/ WER; Stringalign’s documentation and output traceability enable researchers to reproduce and explain such divergences explicitly.

Advanced Diagnostic and Visualization Tools

Error inspection is facilitated through token-specific error counts, indices for false positive/negative alignments for targeted entity analysis, and integration with Jupyter and common image-processing libraries (for OCR RT). Stringalign’s visualization layer allows direct mapping of error operations to original transcript or image, supporting both qualitative and quantitative auditing.

Figure 2: The visual output of Stringalign analysis showing alignment errors for particular model shortcomings, e.g., error confounds due to similar-looking characters or dialectal variants.

Practical and Theoretical Implications

On the practical front, widespread adoption of Stringalign could standardize reporting and diagnostic practices in ATR/ASR research, enhancing reproducibility and interpretability, particularly for languages and scripts with complex Unicode representations. Improved handling of alignment ambiguity and tokenization supports robust data-centric workflows—informing data collection, annotation, and error-driven active learning.

Theoretically, Stringalign’s modular architecture lends itself to extending evaluation paradigms towards subword-level, morpheme-level, or non-Latin script settings, and facilitates integration in benchmarking studies aiming to assess the true generalization gap across models and domains.

Prospects for Future Development

Potential extensions include further optimization of alignment algorithms (e.g., Ukkonen’s or SIMD-based variants), expansion into languages or domains requiring custom segmentation (e.g., Chinese, Japanese, Korean), and community-driven enrichment for new tokenization schemes or error typologies. Addressing these directions is likely to further advance the reproducibility and rigor of ATR and ASR evaluation.

Conclusion

Stringalign provides a comprehensive, transparent, and Unicode-compliant toolbox for evaluation of automatic transcription models. By strictly adhering to open FAIR principles and making all preprocessing explicit and reproducible, it substantially mitigates the ambiguity in CER/ WER benchmarking. Its extensibility, minimal dependencies, and diagnostic orientation position it as a recommended standard for the research community working on ATR, ASR, and related string alignment tasks.