---
title: 'VietNormalizer1: Vietnamese Text Normalization'
url: https://www.emergentmind.com/topics/vietnormalizer1
type: topic
---

# VietNormalizer1: Vietnamese Text Normalization

Searching arXiv for the cited Vietnamese normalization papers to ground the article in current records.
VietNormalizer1 denotes a cluster of Vietnamese normalization systems rather than a single immutable artifact. In the literature, the label is used most explicitly for an open-source, zero-dependency Python library for Vietnamese text normalization in TTS and NLP applications [2603.04145], but closely related work also uses it as a faithful shorthand for a Vietnamese sequence-to-sequence normalizer trained on 2,181 annotated social-media comment pairs [2311.06851], a weakly supervised lexical normalization framework for Vietnamese social media [2409.20467], and a streaming inverse text normalization design integrated with ASR [2505.24229]. Across these usages, the common objective is the transformation of non-standard Vietnamese text into forms that are more canonical, more pronounceable, or more useful for downstream processing.

## 1. Terminological scope and research landscape

Across the cited work, VietNormalizer1 appears in multiple technical senses. This suggests that the term functions less as a single canonical model name than as a family designation for Vietnamese normalization pipelines spanning social-media lexical normalization, spoken-text postprocessing, and rule-based TTS front ends.

| Interpretation | Task | Source |
|---|---|---|
| S2SMulti-based Vietnamese seq2seq normalizer trained on 2,181 comment pairs | Textual normalization for social media and HSD preprocessing | [2311.06851] |
| Weakly supervised data labeling and normalization framework | Lexical normalization in Vietnamese social media | [2409.20467] |
| Streaming ITN system for Vietnamese ASR | Spoken-form to written-form normalization | [2505.24229] |
| Open-source, zero-dependency Python library | Rule-based normalization for TTS and NLP | [2603.04145] |

Two corpora are especially important for situating these systems. "ViLexNorm" is the first dedicated corpus for Vietnamese lexical normalization on social media, with 10,467 sentence pairs from Facebook and TikTok [2401.16403]. "ViDia2Std" extends the normalization problem to dialectal variation through 13,657 dialect-to-standard sentence pairs covering all 63 provinces of Vietnam [2603.10211]. A plausible implication is that VietNormalizer1 should be understood as part of a broader Vietnamese normalization ecosystem rather than a narrowly bounded software package.

## 2. Sequence-to-sequence normalization for social-media comments

The 2023 work on automatic textual normalization formulates Vietnamese normalization as sequence-to-sequence machine translation from non-standard text to standard Vietnamese, with Vietnamese monosyllabic tokens as the modeling unit [2311.06851]. The authors curate a dataset from the ViHSD training split by selecting the first 3,400 comments, annotating them with five fluent Vietnamese annotators, resolving ambiguous cases by a third reviewer, and dropping comments with no NSW. The final dataset contains 2,181 pairs of original and normalized comments in an 8:1:1 train/validation/test split. The full data contain 33,988 tokens, 6,131 NSW, 4,103 unique tokens, and 1,638 unique NSW tokens. Inter-annotator agreement is reported with Cohen’s kappa for the decision “this token should be normalized,” with $\kappa = 0.9014$, and “Same candidates” at 0.7449.

The system family includes three encoder-decoder variants at token level. S2S uses a Bi-LSTM encoder-decoder with Luong-style attention. S2SSelf adds a special `@self` marker during training to indicate tokens that should remain unchanged. S2SMulti is a two-stage arrangement that prioritizes tokens with a single normalized form first, then translates tokens with multiple possible adjustments via the base S2S. The attention-based encoder-decoder uses a Bi-LSTM encoder, an LSTM decoder, and Luong attention with general score; the training objective is standard cross-entropy with scheduled sampling:
$$
L(\theta) = - \sum_{(x, y)\in D} \sum_{j=1}^{|L|} \log p_\theta(y_j \mid y_{<j}, x).
$$

Tokenization is defined at the level of Vietnamese monosyllables: spaces separate syllables; punctuation, brackets, emoji, and emoticons are treated as single tokens; and special strings such as URL, email, hashtag, mention, time, date, and currency are preserved unchanged. Representative mappings include `in4 → thông tin`, `ng → người`, `iu → yêu`, `đc → được`, and `ko → không`. The paper also emphasizes context-dependent ambiguity, such as `t → tôi` or `tao` or `tớ`, and `vs → với`.

On the test set, S2SMulti is the best normalizer, with F1 = 58.30% and BLEU-4 = 69.71%, compared with S2S at F1 = 56.16% and BLEU-4 = 68.71%, and S2SSelf at F1 = 37.65% and BLEU-4 = 58.40%. The abstract’s statement that normalization accuracy is “less than 70%” reflects the fact that BLEU-4 peaks at about 69.7% while F1 peaks at about 58%. Error analysis identifies sequence length sensitivity, ambiguity, OOV and evolving slang, diacritics restoration issues, and labeling inconsistencies as recurrent failure modes.

The paper’s principal downstream result is that even moderate normalization quality improves Hate Speech Detection. Using 438 comments with HATE/OFFENSIVE/CLEAN labels from ViHSD, S2SMulti-normalized inputs improve Text-CNN from 77.27% to 79.55% in F1-micro and Accuracy, and from 35.61% to 41.45% in F1-macro. GRU improves from 75.00% to 77.27% in F1-micro and Accuracy, and from 28.57% to 35.61% in F1-macro. No statistical significance tests are reported. The comparison with rule-based and multi-stage Vietnamese normalization is explicit: seq2seq reduces manual rule engineering and handles variable-length and context-sensitive mappings, but mature rule-based systems can still outperform it on specific constrained error types.

## 3. Corpora, alignment, and weakly supervised lexical normalization

"ViLexNorm" provides the first corpus-scale foundation for Vietnamese lexical normalization on social media [2401.16403]. It contains 10,467 sentence pairs sourced from Facebook and TikTok public comments, with an 8:1:1 split. The annotation protocol uses six native Vietnamese speakers, detailed guidelines, training on a 100-sentence subset, manual quality checks, and re-annotation when agreement fell below 75% on complete-sentence agreement with the authors’ normalization. Averaged agreement during the main phase is 88.46% between authors and other annotators and 93.54% between the two authors. After annotation, sentences without any in-scope normalization are removed. The corpus contains 20,061 word pairs overall and 3,489 distinct pairs, and the standard word `không` alone has 53 variants. Its normalization scope includes abbreviations and slang expansions, diacritic restoration, dialectal variants rendered in writing, misspellings, repeated characters, spacing and tokenization issues, and obfuscation with punctuation or symbol insertion.

The evaluated systems include LSTM, BiGRU with attention, a vanilla Transformer, and BARTpho in syllable-level and word-level variants. The best-performing system is BARTphosyllable, with ERR 0.5774, precision 0.9332, and recall 0.9193. The paper’s extrinsic evaluation shows that normalization improves F1-macro on UIT-VSMEC and ViHSD across TextCNN, BiLSTM, GRU, and PhoBERT. For example, PhoBERT on UIT-VSMEC improves from 59.15 to 62.03 F1, and on ViHSD from 65.91 to 66.54 F1. Under 50% diacritic removal, ViHSD improves from 53.92 to 58.81 F1 after normalization. Downstream gains are not universal: results on ViSPAM are mixed or slightly negative, which the paper attributes to the possibility that non-standard features may themselves be predictive for spam detection.

The 2024 weakly supervised framework turns VietNormalizer1 into a data labeling and normalization pipeline tailored to Vietnamese social media [2409.20467]. It uses labeled normalization data $D_L$ from ViLexNorm, with 10,467 parallel comment pairs and 10,463 entries after preprocessing, and unlabeled social media data $D_U$ comprising 121,087 entries from ViHSD, UIT-VSMEC, ViHOS, ViSpamReviews, and UIT-ViSFD. Preprocessing includes lowercasing, emoji/emoticon separation, punctuation spacing normalization, NER anonymization using NlpHUST viELECTRA NER, and word segmentation with VnCoreNLP.

Its weak supervision layer is built from dictionary labeling functions, regex labeling functions, and contextual PLM pseudo-labels. Conflict resolution is handled by RAN, a Rule Attention Network, which combines LF outputs and Student predictions into soft labels:
$$
q_i = \frac{1}{Z_i}\left(\sum_{j:r^j\in R_i} a_i^j q_i^j + a_i^S p_\theta(y \mid x_i) + a_i^u u\right).
$$
Student models include ViSoBERT, PhoBERT-base, and BARTpho_syllable. The framework also introduces token-level alignment tokenization with `<mask>` and `<space>` to preserve 1–n and n–1 alignments without reordering.

With BARTpho as Student on $D_L$, the baseline reaches Precision 87.52%, Recall 78.94%, F1-score 82.72%, and Sentence accuracy 95.48%. The weakly supervised Teacher+Student framework reaches F1-score 84.94% with BARTpho and sentence accuracy 96.06%. Vocabulary integrity reaches up to 99.22% in the Student baseline and remains high at 99.04% in the weakly supervised BARTpho setting. Under undiacritic augmentation, BARTpho reaches F1 up to 85.64% at $p=1.0$. Extrinsically, the framework reports average accuracy increases of 1–3% across HSD, emotion recognition, hate/offensive span detection, spam review detection/classification, and ABSA.

## 4. Streaming inverse text normalization for ASR

In streaming ASR, VietNormalizer1 is framed as a streaming inverse text normalization system that converts spoken-form ASR output into written-form Vietnamese while respecting latency and limited-context constraints [2505.24229]. The task is formalized over variable-size chunks $C_k$ with fixed left context $L=16$ words and small right context $r_k \in \{1,2\}$, with per-chunk mapping $y_k = f_\theta(C_k, R_k)$. The semiotic domains handled in the reported system are Number-Case categories—Case, Number, Date, Phone—plus Punctuation categories—Comma, Period, Question, Exclamation.

The architecture has two phases. The tagging phase uses PhoBERT, described as BERT-based and pretrained on Vietnamese, with 12 encoder layers, 12 attention heads, hidden size 768, 135M parameters, and a BPE tokenizer. It has two multitask heads: a Number-Case head with 4 classes and a Punctuation head with 4 classes. Labels are assigned only to the first subword of each word, while subsequent subwords are masked with $-\infty$ so they do not contribute to loss or logits; all subwords of the same word share a single positional index. The transduction phase uses a WFST grammar to convert tagged spans into canonical written form, with punctuation inserted in postprocessing.

The defining methodological innovation is Dynamic Context-Aware training and inference. During training, Dynamic Right Context masking randomizes chunk sizes $c \in \{3,4,5,6,7\}$ and right context $r \in \{1,2\}$ while fixing left context $L=16$. During inference, chunk sizes vary naturally with ASR output, often 3–5 words, and a small right-context buffer is used to resolve punctuation and phone-number boundaries. The multitask objective is
$$
L_{total} = L_{numcase} + L_{punct}.
$$
The paper also emphasizes tail buffering for incomplete words, which is particularly important for Vietnamese multi-syllable words split across chunk boundaries.

The reported comparison includes non-streaming and streaming settings. NS2, the non-streaming PhoBERT full-context system, reaches Precision 0.86, Recall 0.85, F1 0.86, I-WER 6.30, and NI-WER 1.63. S4, the PhoBERT streaming system with Dynamic Context Mask and small right context, reaches Precision 0.81, Recall 0.76, F1 0.78, I-WER 8.24, and NI-WER 1.79. Per-class F1 in S4 is 0.80 for Case, 0.86 for Date, 0.89 for Number, and 0.70 for Phone. For punctuation, S4 reaches 0.75 for Comma, 0.46 for Exclamation, 0.56 for Period, and 0.58 for Question. The paper states that S4 is superior to other streaming baselines and close to NS2 in NI-WER.

Latency is explicitly measured for a 400 ms ASR chunk with right context = 1 and CCU = 100. ITN latency is 6.53 ms for chunk size 3, 6.85 ms for chunk size 4, and 6.93 ms for chunk size 5. ASR latency is reported as 48.32 ms for chunk size 4, and RTF is about 0.137–0.138. The system’s stated value lies in achieving accuracy comparable to non-streaming ITN while maintaining low latency suitable for streaming ASR integration.

## 5. Rule-based library architecture and TTS-oriented normalization

The 2026 paper "VietNormalizer: An Open-Source, Dependency-Free Python Library for Vietnamese Text Normalization in TTS and NLP Applications" presents VietNormalizer1 most explicitly as a software artifact [2603.04145]. It is an open-source, zero-dependency Python library, installable via `pip install vietnormalizer`, available on PyPI and GitHub, and released under the MIT license. The implementation is organized around two classes: `VietnameseTextProcessor` for rules and `VietnameseNormalizer` for pipeline orchestration and public API.

Its pipeline applies transformations in a fixed order: Unicode normalization and noise filtering, date normalization, time normalization, currency normalization, percentage normalization, general number normalization, and dictionary replacement. NFC normalization is applied first; emojis and non-printable characters are removed; more specific patterns run before general numbers; and dictionary replacement is done in a single pass through a compiled alternation regex. The stated objective is to expand numbers, dates, times, currency amounts, percentages, acronyms, and foreign-language terms into pronounceable Vietnamese words for TTS and downstream NLP.

The number verbalizer uses positional decomposition into three-digit blocks with scales $10^3$ (`nghin`), $10^6$ (`trieu`), and $10^9$ (`ty`). The rules include `linh` as the zero-tens filler and `lam` after tens, producing examples such as `105 -> "mot tram linh nam"` and `1,234,567 -> "mot trieu hai tram ba muoi bon nghin nam tram sau muoi bay"`. Dates such as `01/05/2024` are normalized to `ngay mot thang nam nam hai nghin khong tram hai muoi bon`, while times such as `09:05:07` become `chin gio linh nam phut bay giay`. VND and USD are supported, with `$12.50 -> "muoi hai do la my phay nam muoi"` and `1.000.000 VND -> "mot trieu dong"`. Percentages are expanded as `X phan tram`.

Acronyms and foreign terms are handled by user-extensible CSV dictionaries. The paper gives examples such as `NASA -> "na-sa"`, `GDP -> "tong san pham quoc noi"`, `container -> "cong-te-no"`, and `Singapore -> "xin-ga-po"`. Unknown foreign tokens are left unchanged unless added to the dictionary. The library deliberately does not use NFKC compatibility folding and does not attempt automatic transliteration for unseen foreign terms.

Its implementation claims zero dependencies, pure Python 3.8+, pre-compiled regexes, O(n) runtime per stage, and high-throughput CPU-only batch processing. The comparison with existing Vietnamese normalization tools is framed around deployability: neural approaches require heavyweight runtimes and model files; broader NLP toolkits may provide Unicode or diacritic normalization but not NSW verbalization; and VietNormalizer offers determinism, auditability, and CSV-based extensibility. The limitations are equally explicit: context-dependent disambiguation remains difficult, proper noun handling is outside scope, code-switching coverage depends on the dictionary, and an inverse text normalization module is planned for a future release.

## 6. Dialect normalization, adjacent resources, and broader significance

Vietnamese normalization cannot be reduced to abbreviations, diacritics, or spoken-number formatting alone. "ViDia2Std" broadens the agenda to dialect-to-standard Vietnamese translation through 13,657 sentence pairs from authentic Facebook comments, with all 63 provinces represented and explicit coverage of Central, Southern, and non-standard Northern varieties [2603.10211]. The dataset is split into 10,870 training, 1,184 validation, and 1,603 test pairs. Quality control uses Strict Semantic Group Agreement, with region-wise agreement rates of 86% for North, 82% for Central, and 85% for South.

The benchmark models are BARTpho-word-base, BARTpho-syllable-base, ViT5-base, vietnamese-correction-v2, and mBART-large-50. On the test set, mBART-large-50 is best, with ROUGE-L 0.9384, BLEU 0.8166, METEOR 0.8925, WER 0.1226, and CER 0.0754; ViT5-base is second, with ROUGE-L 0.9300, BLEU 0.7934, and METEOR 0.8802. The paper’s downstream evaluations further show that normalization materially improves machine translation and sentiment analysis on dialectal inputs. For sentiment analysis, Accuracy improves from 50.59% to 62.13% and Macro F1 from 0.48 to 0.58 after normalization.

A plausible implication is that VietNormalizer1 belongs to a continuum of Vietnamese text standardization technologies whose exact form depends on the application domain. In social-media lexical normalization, the dominant challenges are NSW diversity, ambiguity, and evolving slang. In streaming ITN, the central problems are chunking, right-context management, and latency. In TTS-oriented rule-based normalization, the priority is deterministic verbalization of semiotic classes. In dialect normalization, the problem becomes sentence-level meaning preservation across regional variants. The literature therefore does not support a single “best” VietNormalizer1 in the abstract; it supports domain-specific normalizers optimized for distinct notions of standardization, distinct error surfaces, and distinct downstream objectives.

Source: https://www.emergentmind.com/topics/vietnormalizer1