VietNormalizer1: Vietnamese Text Normalization

• VietNormalizer1 is a family of Vietnamese text normalization systems designed to convert non-standard inputs into canonical forms for TTS, ASR, and NLP applications.
• It encompasses diverse methodologies, including rule-based Python libraries, sequence-to-sequence models, and weakly supervised frameworks tailored for social media and dialect normalization.
• The systems demonstrate measurable improvements in downstream tasks such as hate speech detection, sentiment analysis, and machine translation through enhanced accuracy and reduced manual interventions.

Searching arXiv for the cited Vietnamese normalization papers to ground the article in current records. VietNormalizer1 denotes a cluster of Vietnamese normalization systems rather than a single immutable artifact. In the literature, the label is used most explicitly for an open-source, zero-dependency Python library for Vietnamese text normalization in TTS and NLP applications (Nguyen et al., 4 Mar 2026), but closely related work also uses it as a faithful shorthand for a Vietnamese sequence-to-sequence normalizer trained on 2,181 annotated social-media comment pairs (Nguyen et al., 2023), a weakly supervised lexical normalization framework for Vietnamese social media (Nguyen et al., 2024), and a streaming inverse text normalization design integrated with ASR (Ho et al., 30 May 2025). Across these usages, the common objective is the transformation of non-standard Vietnamese text into forms that are more canonical, more pronounceable, or more useful for downstream processing.

1. Terminological scope and research landscape

Across the cited work, VietNormalizer1 appears in multiple technical senses. This suggests that the term functions less as a single canonical model name than as a family designation for Vietnamese normalization pipelines spanning social-media lexical normalization, spoken-text postprocessing, and rule-based TTS front ends.

Interpretation	Task	Source
S2SMulti-based Vietnamese seq2seq normalizer trained on 2,181 comment pairs	Textual normalization for social media and HSD preprocessing	(Nguyen et al., 2023)
Weakly supervised data labeling and normalization framework	Lexical normalization in Vietnamese social media	(Nguyen et al., 2024)
Streaming ITN system for Vietnamese ASR	Spoken-form to written-form normalization	(Ho et al., 30 May 2025)
Open-source, zero-dependency Python library	Rule-based normalization for TTS and NLP	(Nguyen et al., 4 Mar 2026)

Two corpora are especially important for situating these systems. "ViLexNorm" is the first dedicated corpus for Vietnamese lexical normalization on social media, with 10,467 sentence pairs from Facebook and TikTok (Nguyen et al., 2024). "ViDia2Std" extends the normalization problem to dialectal variation through 13,657 dialect-to-standard sentence pairs covering all 63 provinces of Vietnam (Ta et al., 10 Mar 2026). A plausible implication is that VietNormalizer1 should be understood as part of a broader Vietnamese normalization ecosystem rather than a narrowly bounded software package.

2. Sequence-to-sequence normalization for social-media comments

The 2023 work on automatic textual normalization formulates Vietnamese normalization as sequence-to-sequence machine translation from non-standard text to standard Vietnamese, with Vietnamese monosyllabic tokens as the modeling unit (Nguyen et al., 2023). The authors curate a dataset from the ViHSD training split by selecting the first 3,400 comments, annotating them with five fluent Vietnamese annotators, resolving ambiguous cases by a third reviewer, and dropping comments with no NSW. The final dataset contains 2,181 pairs of original and normalized comments in an 8:1:1 train/validation/test split. The full data contain 33,988 tokens, 6,131 NSW, 4,103 unique tokens, and 1,638 unique NSW tokens. Inter-annotator agreement is reported with Cohen’s kappa for the decision “this token should be normalized,” with $\kappa = 0.9014$, and “Same candidates” at 0.7449.

The system family includes three encoder-decoder variants at token level. S2S uses a Bi-LSTM encoder-decoder with Luong-style attention. S2SSelf adds a special @self marker during training to indicate tokens that should remain unchanged. S2SMulti is a two-stage arrangement that prioritizes tokens with a single normalized form first, then translates tokens with multiple possible adjustments via the base S2S. The attention-based encoder-decoder uses a Bi-LSTM encoder, an LSTM decoder, and Luong attention with general score; the training objective is standard cross-entropy with scheduled sampling:

$$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$

Tokenization is defined at the level of Vietnamese monosyllables: spaces separate syllables; punctuation, brackets, emoji, and emoticons are treated as single tokens; and special strings such as URL, email, hashtag, mention, time, date, and currency are preserved unchanged. Representative mappings include in4 → thông tin, ng → người, iu → yêu, đc → được, and ko → không. The paper also emphasizes context-dependent ambiguity, such as t → tôi or tao or tớ, and vs → với.

On the test set, S2SMulti is the best normalizer, with F1 = 58.30% and BLEU-4 = 69.71%, compared with S2S at F1 = 56.16% and BLEU-4 = 68.71%, and S2SSelf at F1 = 37.65% and BLEU-4 = 58.40%. The abstract’s statement that normalization accuracy is “less than 70%” reflects the fact that BLEU-4 peaks at about 69.7% while F1 peaks at about 58%. Error analysis identifies sequence length sensitivity, ambiguity, OOV and evolving slang, diacritics restoration issues, and labeling inconsistencies as recurrent failure modes.

The paper’s principal downstream result is that even moderate normalization quality improves Hate Speech Detection. Using 438 comments with HATE/OFFENSIVE/CLEAN labels from ViHSD, S2SMulti-normalized inputs improve Text-CNN from 77.27% to 79.55% in F1-micro and Accuracy, and from 35.61% to 41.45% in F1-macro. GRU improves from 75.00% to 77.27% in F1-micro and Accuracy, and from 28.57% to 35.61% in F1-macro. No statistical significance tests are reported. The comparison with rule-based and multi-stage Vietnamese normalization is explicit: seq2seq reduces manual rule engineering and handles variable-length and context-sensitive mappings, but mature rule-based systems can still outperform it on specific constrained error types.

3. Corpora, alignment, and weakly supervised lexical normalization

"ViLexNorm" provides the first corpus-scale foundation for Vietnamese lexical normalization on social media (Nguyen et al., 2024). It contains 10,467 sentence pairs sourced from Facebook and TikTok public comments, with an 8:1:1 split. The annotation protocol uses six native Vietnamese speakers, detailed guidelines, training on a 100-sentence subset, manual quality checks, and re-annotation when agreement fell below 75% on complete-sentence agreement with the authors’ normalization. Averaged agreement during the main phase is 88.46% between authors and other annotators and 93.54% between the two authors. After annotation, sentences without any in-scope normalization are removed. The corpus contains 20,061 word pairs overall and 3,489 distinct pairs, and the standard word không alone has 53 variants. Its normalization scope includes abbreviations and slang expansions, diacritic restoration, dialectal variants rendered in writing, misspellings, repeated characters, spacing and tokenization issues, and obfuscation with punctuation or symbol insertion.

The evaluated systems include LSTM, BiGRU with attention, a vanilla Transformer, and BARTpho in syllable-level and word-level variants. The best-performing system is BARTphosyllable, with ERR 0.5774, precision 0.9332, and recall 0.9193. The paper’s extrinsic evaluation shows that normalization improves F1-macro on UIT-VSMEC and ViHSD across TextCNN, BiLSTM, GRU, and PhoBERT. For example, PhoBERT on UIT-VSMEC improves from 59.15 to 62.03 F1, and on ViHSD from 65.91 to 66.54 F1. Under 50% diacritic removal, ViHSD improves from 53.92 to 58.81 F1 after normalization. Downstream gains are not universal: results on ViSPAM are mixed or slightly negative, which the paper attributes to the possibility that non-standard features may themselves be predictive for spam detection.

The 2024 weakly supervised framework turns VietNormalizer1 into a data labeling and normalization pipeline tailored to Vietnamese social media (Nguyen et al., 2024). It uses labeled normalization data $D_L$ from ViLexNorm, with 10,467 parallel comment pairs and 10,463 entries after preprocessing, and unlabeled social media data $D_U$ comprising 121,087 entries from ViHSD, UIT-VSMEC, ViHOS, ViSpamReviews, and UIT-ViSFD. Preprocessing includes lowercasing, emoji/emoticon separation, punctuation spacing normalization, NER anonymization using NlpHUST viELECTRA NER, and word segmentation with VnCoreNLP.

Its weak supervision layer is built from dictionary labeling functions, regex labeling functions, and contextual PLM pseudo-labels. Conflict resolution is handled by RAN, a Rule Attention Network, which combines LF outputs and Student predictions into soft labels:

$$q_i = \frac{1}{Z_i}\left(\sum_{j:r^j\in R_i} a_i^j q_i^j + a_i^S p_\theta(y \mid x_i) + a_i^u u\right).$$

Student models include ViSoBERT, PhoBERT-base, and BARTpho_syllable. The framework also introduces token-level alignment tokenization with <mask> and <space> to preserve 1–n and n–1 alignments without reordering.

With BARTpho as Student on $D_L$, the baseline reaches Precision 87.52%, Recall 78.94%, F1-score 82.72%, and Sentence accuracy 95.48%. The weakly supervised Teacher+Student framework reaches F1-score 84.94% with BARTpho and sentence accuracy 96.06%. Vocabulary integrity reaches up to 99.22% in the Student baseline and remains high at 99.04% in the weakly supervised BARTpho setting. Under undiacritic augmentation, BARTpho reaches F1 up to 85.64% at $p=1.0$. Extrinsically, the framework reports average accuracy increases of 1–3% across HSD, emotion recognition, hate/offensive span detection, spam review detection/classification, and ABSA.

4. Streaming inverse text normalization for ASR

In streaming ASR, VietNormalizer1 is framed as a streaming inverse text normalization system that converts spoken-form ASR output into written-form Vietnamese while respecting latency and limited-context constraints (Ho et al., 30 May 2025). The task is formalized over variable-size chunks $C_k$ with fixed left context $L=16$ words and small right context $r_k \in \{1,2\}$, with per-chunk mapping $$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$0. The semiotic domains handled in the reported system are Number-Case categories—Case, Number, Date, Phone—plus Punctuation categories—Comma, Period, Question, Exclamation.

The architecture has two phases. The tagging phase uses PhoBERT, described as BERT-based and pretrained on Vietnamese, with 12 encoder layers, 12 attention heads, hidden size 768, 135M parameters, and a BPE tokenizer. It has two multitask heads: a Number-Case head with 4 classes and a Punctuation head with 4 classes. Labels are assigned only to the first subword of each word, while subsequent subwords are masked with $$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$1 so they do not contribute to loss or logits; all subwords of the same word share a single positional index. The transduction phase uses a WFST grammar to convert tagged spans into canonical written form, with punctuation inserted in postprocessing.

The defining methodological innovation is Dynamic Context-Aware training and inference. During training, Dynamic Right Context masking randomizes chunk sizes $$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$2 and right context $$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$3 while fixing left context $$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$4. During inference, chunk sizes vary naturally with ASR output, often 3–5 words, and a small right-context buffer is used to resolve punctuation and phone-number boundaries. The multitask objective is

$$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$5

The paper also emphasizes tail buffering for incomplete words, which is particularly important for Vietnamese multi-syllable words split across chunk boundaries.

The reported comparison includes non-streaming and streaming settings. NS2, the non-streaming PhoBERT full-context system, reaches Precision 0.86, Recall 0.85, F1 0.86, I-WER 6.30, and NI-WER 1.63. S4, the PhoBERT streaming system with Dynamic Context Mask and small right context, reaches Precision 0.81, Recall 0.76, F1 0.78, I-WER 8.24, and NI-WER 1.79. Per-class F1 in S4 is 0.80 for Case, 0.86 for Date, 0.89 for Number, and 0.70 for Phone. For punctuation, S4 reaches 0.75 for Comma, 0.46 for Exclamation, 0.56 for Period, and 0.58 for Question. The paper states that S4 is superior to other streaming baselines and close to NS2 in NI-WER.

Latency is explicitly measured for a 400 ms ASR chunk with right context = 1 and CCU = 100. ITN latency is 6.53 ms for chunk size 3, 6.85 ms for chunk size 4, and 6.93 ms for chunk size 5. ASR latency is reported as 48.32 ms for chunk size 4, and RTF is about 0.137–0.138. The system’s stated value lies in achieving accuracy comparable to non-streaming ITN while maintaining low latency suitable for streaming ASR integration.

5. Rule-based library architecture and TTS-oriented normalization

The 2026 paper "VietNormalizer: An Open-Source, Dependency-Free Python Library for Vietnamese Text Normalization in TTS and NLP Applications" presents VietNormalizer1 most explicitly as a software artifact (Nguyen et al., 4 Mar 2026). It is an open-source, zero-dependency Python library, installable via pip install vietnormalizer, available on PyPI and GitHub, and released under the MIT license. The implementation is organized around two classes: VietnameseTextProcessor for rules and VietnameseNormalizer for pipeline orchestration and public API.

Its pipeline applies transformations in a fixed order: Unicode normalization and noise filtering, date normalization, time normalization, currency normalization, percentage normalization, general number normalization, and dictionary replacement. NFC normalization is applied first; emojis and non-printable characters are removed; more specific patterns run before general numbers; and dictionary replacement is done in a single pass through a compiled alternation regex. The stated objective is to expand numbers, dates, times, currency amounts, percentages, acronyms, and foreign-language terms into pronounceable Vietnamese words for TTS and downstream NLP.

The number verbalizer uses positional decomposition into three-digit blocks with scales $$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$6 (nghin), $$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$7 (trieu), and $$L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).$$8 (ty). The rules include linh as the zero-tens filler and lam after tens, producing examples such as 105 -> "mot tram linh nam" and 1,234,567 -> "mot trieu hai tram ba muoi bon nghin nam tram sau muoi bay". Dates such as 01/05/2024 are normalized to ngay mot thang nam nam hai nghin khong tram hai muoi bon, while times such as 09:05:07 become chin gio linh nam phut bay giay. VND and USD are supported, with $12.50 -> "muoi hai do la my phay nam muoi" and 1.000.000 VND -> "mot trieu dong". Percentages are expanded as X phan tram.

Acronyms and foreign terms are handled by user-extensible CSV dictionaries. The paper gives examples such as NASA -> "na-sa", GDP -> "tong san pham quoc noi", container -> "cong-te-no", and Singapore -> "xin-ga-po". Unknown foreign tokens are left unchanged unless added to the dictionary. The library deliberately does not use NFKC compatibility folding and does not attempt automatic transliteration for unseen foreign terms.

Its implementation claims zero dependencies, pure Python 3.8+, pre-compiled regexes, O(n) runtime per stage, and high-throughput CPU-only batch processing. The comparison with existing Vietnamese normalization tools is framed around deployability: neural approaches require heavyweight runtimes and model files; broader NLP toolkits may provide Unicode or diacritic normalization but not NSW verbalization; and VietNormalizer offers determinism, auditability, and CSV-based extensibility. The limitations are equally explicit: context-dependent disambiguation remains difficult, proper noun handling is outside scope, code-switching coverage depends on the dictionary, and an inverse text normalization module is planned for a future release.

6. Dialect normalization, adjacent resources, and broader significance

Vietnamese normalization cannot be reduced to abbreviations, diacritics, or spoken-number formatting alone. "ViDia2Std" broadens the agenda to dialect-to-standard Vietnamese translation through 13,657 sentence pairs from authentic Facebook comments, with all 63 provinces represented and explicit coverage of Central, Southern, and non-standard Northern varieties (Ta et al., 10 Mar 2026). The dataset is split into 10,870 training, 1,184 validation, and 1,603 test pairs. Quality control uses Strict Semantic Group Agreement, with region-wise agreement rates of 86% for North, 82% for Central, and 85% for South.

The benchmark models are BARTpho-word-base, BARTpho-syllable-base, ViT5-base, vietnamese-correction-v2, and mBART-large-50. On the test set, mBART-large-50 is best, with ROUGE-L 0.9384, BLEU 0.8166, METEOR 0.8925, WER 0.1226, and CER 0.0754; ViT5-base is second, with ROUGE-L 0.9300, BLEU 0.7934, and METEOR 0.8802. The paper’s downstream evaluations further show that normalization materially improves machine translation and sentiment analysis on dialectal inputs. For sentiment analysis, Accuracy improves from 50.59% to 62.13% and Macro F1 from 0.48 to 0.58 after normalization.

A plausible implication is that VietNormalizer1 belongs to a continuum of Vietnamese text standardization technologies whose exact form depends on the application domain. In social-media lexical normalization, the dominant challenges are NSW diversity, ambiguity, and evolving slang. In streaming ITN, the central problems are chunking, right-context management, and latency. In TTS-oriented rule-based normalization, the priority is deterministic verbalization of semiotic classes. In dialect normalization, the problem becomes sentence-level meaning preservation across regional variants. The literature therefore does not support a single “best” VietNormalizer1 in the abstract; it supports domain-specific normalizers optimized for distinct notions of standardization, distinct error surfaces, and distinct downstream objectives.