Lexical Normalization in NLP

• Lexical normalization is the process of converting informal, noisy text—such as social media input—into a standardized, canonical form for reliable NLP analysis.
• Modern approaches frame normalization as a conditional sequence-to-sequence problem using methods like word-level candidate ranking, transformer models, and byte-level architectures.
• Evaluation metrics, error pattern analyses, and adaptations for code-switched and low-resource languages are key to advancing normalization techniques and improving downstream task accuracy.

Lexical normalization is the computational process of converting non-standard text—often produced in informal, spontaneous digital communication such as social media—into a more standardized, canonical form suitable for downstream NLP applications. The task addresses orthographic irregularities, creative abbreviations, misspellings, expressive character repetitions, and code-switching phenomena that degrade the performance of NLP models fine-tuned on formal corpora. In contemporary research, lexical normalization is widely treated as a conditional sequence-to-sequence problem, where an input token sequence $x = (x_1, …, x_{T_x})$ is mapped to a normalized output sequence $y = (y_1, …, y_{T_y})$ by a parameterized model $p_\theta(y|x)$, typically autoregressively factorized as $$p_\theta(y|x) = \prod_{t=1}^{T_y}p_\theta(y_t\,|\,y_{<t}, x)$$.

1. Formal Definition and Problem Scope

Lexical normalization operates on noisy inputs that differ systematically from canonical text due to typographical errors, abbreviations, dialectal variants, code-mixed language contexts, and user-driven writing strategies—especially in Twitter, Reddit, and other social media platforms. This discrepancy leads to markedly reduced performance for models trained on clean, benchmark corpora, necessitating pre-processing steps that restore standard orthography and segmentation prior to downstream analysis (Bucur et al., 2021). The normalization process encompasses:

• Canonical replacement: Correct mapping of OOV tokens to their dictionary forms.
• Grammatical corrections: Regularization of character-case and diacritics.
• Structural normalization: Splitting/merging of joined or separated tokens (handling 1→N and N→1 replacements).
• Context-sensitive selection: Particularly for code-switched data, where language identification precedes normalization.

The sequence-to-sequence framing treats normalization as a conditional language generation problem, with systems minimizing the cross-entropy loss:

$$L(\theta) = -\sum_{(x,y) \in D}\sum_{t=1}^{T_y}\log p_\theta(y_t | y_{<t},x)$$

For code-switched data, normalization complexity increases, requiring models to incorporate language-ID signals, context-sensitive candidate generation, and multilingual resources (Goot et al., 2020).

2. Model Architectures and Representative Systems

Recent advances in lexical normalization have adopted both word-level and sentence-level modeling strategies:

• Word-level candidate ranking: MoNoise integrates candidate generation (Aspell, embedding lookup, lexicon search) with feature-based ranking. For code-switched data, MoNoise variants segment input via language ID, duplicating or restricting features by language, and applying random forest classifiers to candidate sets (Goot et al., 2020).
• Transformer-based sequence-to-sequence models: Multilingual pretrained models such as mBART-large-50 (12-layer encoder and decoder, 1,024 hidden units, 16 attention heads, trained on 50 languages) frame normalization as a denoising autoencoder or machine-translation task. Fine-tuning occurs over annotated noisy–canonical pairs (Bucur et al., 2021).
• Byte-level architectures: ByT5 (byte-tokenized T5) models robustly generalize across languages and out-of-vocabulary spelling, winning the MultiLexNorm shared task (Samuel et al., 2021).
• Discriminative context modeling: Context-dependent word embeddings (ELMo, BERT), Bi-GRU encoders, and attention-based models have shown gains for normalization of domain-specific and social-media data (Stewart et al., 2019).
• Regular-expression cascades: Rule-based systems and category-specific expansions are widely used in lower-resource or morphologically rich languages, where semiotic-class taxonomies and cascaded regex replace patterns perform high-accuracy expansions for numbers, dates, abbreviations, and domain-specific codes (Kasparaitis, 2023).

Sentence-level seq2seq models in multilingual settings (mBART, ByT5) have demonstrated increased robustness to surface-form variation, but historically trail word-level baselines in word-aligned error reduction (Bucur et al., 2021). Byte-level tokenization is critical for typologically diverse scripts and code-mixed data.

3. Corpora, Annotation Standards, and Data Characteristics

Multiple normalization benchmarks have been established to measure system performance on authentic noisy text:

• MultiLexNorm (W-NUT 2021): Covers 12 languages, annotated for normalization targets, casing, splits/merges, and code-mixing. Example: Turkish-German code-switch data with 24.25% normalized tokens (Bucur et al., 2021).
• ViLexNorm (Vietnamese): 10,467 manually annotated social-media sentence pairs, 88.46% inter-annotator agreement, rich OOV forms (e.g., "không" with 53 variants), and detailed error taxonomy. 57.74% ERR with BARTpho-syllable (Nguyen et al., 2024).
• Other domain corpora: Danish DaN+, Japanese JMLN, Roman Urdu SMS/Web/CFMP datasets, and extensive gold-standard mining accident and Twitter corpora (Stewart et al., 2019, Plank et al., 2021, Higashiyama et al., 28 May 2025, Khan et al., 2020).

Annotation guidelines emphasize precise definitions for “canonical” vs “non-canonical,” both at word and subtokensegment boundaries, and require statistical measurement of agreement (e.g., Cohen’s $\kappa$, pairwise accuracy), capturing 1→N and N→1 mapping phenomena.

4. Evaluation Metrics and Comparative Results

Lexical normalization systems are evaluated along both intrinsic (token-level) and extrinsic (task-level) axes:

• Intrinsic metrics:
  • Error Reduction Rate (ERR):

  $ERR = \frac{TP - FP}{TP + FN}$

  Where TP: true positives; FP: false positives; FN: false negatives. ERR quantifies fraction of possible improvement over leave-as-is baseline (ERR=0). - Precision/Recall/F1:

  $$P = \frac{TP}{TP+FP}, \quad R = \frac{TP}{TP+FN}, \quad F_1 = \frac{2PR}{P+R}$$

System	Language(s)	ERR (%)	F1 (%)
ByT5 (ÚFAL ensemble)	11 (+2 code-mix)	67.3	N/A
mBART	12	10.65	N/A
BARTpho_syllable	Vietnamese	57.74	93.3
MoNoise (word-level)	MultiLexNorm	49.02	N/A

• Extrinsic metrics:
  • Dependency parsing LAS: Improved by ~1.7 points after normalization in noisy Twitter treebanks (Bucur et al., 2021).
  • Sentiment/offensive-language macro-F1: Sentence-level normalization yields small but consistent downstream gains versus raw input (Bucur et al., 2021, Nguyen et al., 2024).
  • POS tagging (code-switch): Normalization yields 5.4% relative accuracy increase for Turkish-German (Goot et al., 2020).

Notably, sentence-level multilingual normalization delivers more robust improvements in complex downstream tasks (e.g., parsing, NER), even when token-level error reduction lags behind specialized word-level systems.

5. Approaches for Code-Switched and Low-Resource Languages

Code-switching introduces nontrivial challenges for normalization—models must first identify token language before selecting appropriate normalization machinery. Multilingual MoNoise variants, coupled with explicit LID features and independent candidate ranking pipelines, outperform monolingual baselines and facilitate double-digit POS tagging accuracy increases for Turkish-German and Indonesian-English Twitter datasets (Goot et al., 2020).

In low-resource settings (Vietnamese, Roman Urdu), innovative weakly supervised learning frameworks combine minimal gold annotation with rule-based and neural pseudo-label generation, leveraging teacher-student architectures and attention-based aggregation to expand training data coverage and refine normalization quality. For Vietnamese social media (ViSoLex, ViLexNorm), F1 reaches 85% and vocabulary integrity 99% (Nguyen et al., 13 Jan 2025, Nguyen et al., 2024).

Rule-based approaches remain effective for semiotic expansion of numbers, dates, abbreviations, and codes in languages lacking large annotated corpora (Kasparaitis, 2023). Clustering methods (Lex-Var) employing phonetic and contextual features are demonstrated for Roman Urdu vocabularies (Khan et al., 2020).

6. Error Patterns, Limitations, and Open Challenges

Persistent normalization errors stem from:

• Spelling variants / unseen typos: LLM pipelines and transformer models struggle with low-frequency creative variants, unseen loanwords, and deep code-mixing.
• Under-/over-normalization: Sentence-level models may over-correct tokens left unchanged by annotators, incurring precision penalties.
• Token alignment: Aggressive splitting/merging and expressive punctuation introduce alignment ambiguities penalized by word-level evaluation metrics.
• Script and tokenization bottlenecks: Byte-level models (ByT5) outperform subword models on non-Latin scripts, but struggle with multi-byte segmentation and syntactic complexity (Buaphet et al., 23 Jan 2026, Higashiyama et al., 28 May 2025).
• Resource limitations: Manual annotation in new domains is labor-intensive and costly; semi-supervised, weakly supervised, and crowd-sourced expansion approaches are in active development.

7. Prospects for Future Research and Integration

Significant open directions include:

• Sub-character and grapheme-level modeling: Enhanced tokenization and neural architectures to handle morphologically rich, non-segmented scripts (Buaphet et al., 23 Jan 2026).
• Sentence-level normalization: Models capable of context-sensitive reordering, multi-token expansion, and joint learning with downstream tasks such as parsing and NER (Bucur et al., 2021, Higashiyama et al., 28 May 2025).
• Corpus extension and annotation: Larger, more diverse benchmarks, and synthetic noise generation to improve coverage of real-world phenomena in underrepresented languages (Nguyen et al., 2024, Samuel et al., 2021).
• Plug-in normalization for end-to-end systems: Seamless normalization layers integrated with neural pipelines for joint learning and propagation of normalization errors (Plank et al., 2021).
• Cross-domain and cross-lingual generalization: Transfer learning strategies and task-agnostic normalization modules robust to domain and language shift.

Lexical normalization thus remains a foundational NLP process, enabling the reliable analysis of informal, multi-lingual text at scale. Its evolution continues alongside advances in sequence modeling, cross-lingual transfer, and weakly supervised learning paradigms.