MultiLexNorm++: Multilingual Lexical Benchmark

• MultiLexNorm++ is a multilingual benchmark and generative model suite designed to normalize noisy digital text, with a focus on Asian languages and diverse scripts.
• It extends previous benchmarks by incorporating five Asian languages, addressing script-specific challenges such as tokenization and orthographic variations.
• The hybrid pipeline combines detection, lookup, and LLM prompting to achieve significant error reductions, particularly in non-Latin scripts like Thai and Japanese.

MultiLexNorm++ is a multilingual benchmark and generative model suite designed to address the challenges of lexical normalization in informal digital text, with a particular emphasis on Asian languages with diverse scripts and morphologies. Lexical normalization—transforming non-standard or noisy text (from sources such as social media) into canonical forms—directly impacts the performance of core NLP systems including dependency parsing, sentiment analysis, POS tagging, and machine translation. The MultiLexNorm++ initiative extends prior normalization resources and methodologies by introducing new languages, more representative phenomena, and new LLM-based modeling approaches (Buaphet et al., 23 Jan 2026).

1. Motivation and Scope

Lexical normalization corrects varied non-standard phenomena such as abbreviations, orthographic variants, phonetic spellings, diacritic modifications, code-switching, and script-level merges/splits (e.g., Thai’s lack of word boundaries). While the original MultiLexNorm benchmark provided a unified normalization evaluation for Indo-European, Latin-script languages, it lacked linguistic and script diversity (Bucur et al., 2021). MultiLexNorm++ addresses this gap by adding five Asian languages—Indonesian (Latn), Japanese (Kanji/Hiragana/Katakana), Korean (Hangul), Thai (Thai script), and Vietnamese (Latn with diacritics)—thus broadening the coverage to four scripts and five language families. These languages capture a much wider typology of informal linguistic phenomena, such as script-specific tokenization, orthographic variation, transliteration, and morphological irregularity (Buaphet et al., 23 Jan 2026).

2. Dataset Construction and Analysis

Data for MultiLexNorm++ is assembled from preexisting social media corpora, subsequently realigned and reannotated to the MultiLexNorm word-level normalization schema. Annotation is executed by teams of native speakers, with kappa scores (e.g., κ ≈ 0.91 for Japanese, κ = 0.916 for Korean) confirming high inter-annotator agreement. The five new languages are represented as follows:

Language	Code	# Tokens	% Normalized	Script	Family
Indonesian	ID	48,716	47.47	Latn	Austronesian
Japanese	JA	95,411	7.03	Jpan	Japonic
Korean	KO	16,618	7.52	Kore	Koreanic
Thai	TH	169,751	4.72	Thai	Tai–Kadai
Vietnamese	VI	128,685	15.98	Latn	Austroasiatic

Normalization rates span from 4–48%, dependent on the informality of the source and the richness of script-specific phenomena. Annotation guidelines ensure coverage of orthographic variants, slang, merges, splits, and transliterations, with all splits (train/validation/test) conforming to those of the original MultiLexNorm (Buaphet et al., 23 Jan 2026).

3. Modeling Architectures

3.1 Previous and Baseline Systems

The baseline Most-Frequent-Replacement (MFR) dictionary replaces words with their most common normalization observed in training. The ÚFAL system, state-of-the-art on MultiLexNorm, fine-tunes ByT5 (byte-level encoder-decoder) in a contextual, token-marked setup to map noisy tokens to normalized forms, using a standard cross-entropy objective.

Notably, model errors with byte-level segmenters (such as ByT5 in JA, KO, and TH) reveal significant tokenization bottlenecks—average characters per subword can fall to 0.34, increasing sequence lengths and degrading normalization accuracy (Buaphet et al., 23 Jan 2026).

3.2 LLM-Based Multi-Stage Pipeline

The MultiLexNorm++ generative pipeline features three core stages:

1. Detection: An XLM-RoBERTa (via MaChAmp) sequence-labeler flags tokens for normalization.
2. Lookup: Flagged tokens are passed to a dictionary. If there are multiple normalization candidates, selection is based on minimum Miller–Madow entropy.
3. LLM Prompting: For ambiguous or unresolved tokens, few-shot in-context prompts (8 examples) are constructed and posed to LLMs, including Llama 3 70B, Qwen 2.5 72B, Gemma 3 27B-it, or GPT-4o.

This hybrid design leverages the robustness of pre-trained LLMs while controlling for known and frequent token-level mappings, maximizing efficiency and accuracy across scripts (Buaphet et al., 23 Jan 2026).

4. Training Regimes and Evaluation Metrics

The detection model is trained with a learning rate of $1\times10^{-4}$, batch size 32, and dropout 0.2 over 20 epochs. The ÚFAL baseline uses original ByT5 fine-tuning configurations. LLMs are utilized in zero- or few-shot inference without additional fine-tuning.

Evaluation relies on:

• Error Reduction Rate (ERR):

$$\mathrm{ERR} = \frac{E_{\text{base}} - E_{\text{model}}}{E_{\text{base}}} \times 100\%$$

where $E_{\text{base}}$ and $E_{\text{model}}$ are token-level errors. ERR $>0$ indicates reduction in errors relative to baseline; negative values indicate performance degradation.

• Precision, Recall, F₁:

$$\text{Precision} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}} \qquad \text{Recall} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} \qquad F_1 = 2\,\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}$$

• Token-level Accuracy: Proportion of correctly normalized tokens (Buaphet et al., 23 Jan 2026).

5. Empirical Results and Error Analysis

Across all languages, the LLM-based pipeline (notably GPT-4o) demonstrates robust normalization, especially for Indonesian, Vietnamese, and Thai. In non-Latin scripts, ByT5/ÚFAL approaches (byte-level) degrade, with ERR for ByT5 below 10 on Japanese, Korean, and Thai, but the LLM pipeline lifts Thai to 43.9 ERR and Japanese to 22.5 ERR. The table below summarizes core ERR scores:

Language	MFR	ÚFAL	LLaMA3	Qwen2.5	Gemma3	GPT-4o
IN	58.94	67.94	63.24	64.73	66.45	68.35
JA	5.77	39.24	16.27	20.34	14.66	22.52
KO	6.35	–2.38	–14.82	–6.88	–8.73	–8.73
TH	41.87	6.90	40.39	42.23	43.09	43.95
VI	75.58	77.35	76.41	77.79	77.57	80.04

Automatic error breakdown reveals that ~50% of remaining errors are due to selection of plausible but incorrect candidates, with under-normalization (~30%) and over-normalization (~15%) as lesser contributors. Pipeline detection and lookup failures account for a minority. Manual inspection attributes persistent errors to phonetic substitutions, transliteration, sociolinguistic context, and script ambiguity (Buaphet et al., 23 Jan 2026).

6. Analysis of Segmentation and Script-Specific Challenges

ByT5’s byte-level segmenter induces extreme fragmentation in non-Latin scripts (JA, KO, TH), with average characters per subword ≈ 0.34, versus ≈ 1 for Latn. This results in inefficient and error-prone normalization. Experimentation with Latinized decompositions (e.g., Thai sub-character features) recovers some ERR, suggesting the need for script-aware or sub-character modeling (e.g., stroke/radical encodings or Latinized scripts) (Buaphet et al., 23 Jan 2026).

7. Future Directions and Benchmark Extension

Several avenues are proposed to advance MultiLexNorm++:

• Script-aware and sub-character representations: Deploying encodings such as StrokeNet or script-adapted subword vocabularies to mitigate the tokenization bottleneck in non-Latin scripts.
• Multi-level normalization: Integrating character-, word-, and sentence-level transformation pipelines, guided by downstream task improvement (e.g., parsing, machine translation).
• Tighter extrinsic evaluation: Coupling normalization quality metrics to effects on structured prediction tasks (UAS/LAS, sentiment macro-F₁, etc.).
• Model improvements: Fine-tuning open-access LLMs on language- and script-specific normalization data, and exploiting adaptive prompt construction to handle high morphological and script diversity robustly.
• Benchmark expansion: Further inclusion of datasets in additional scripts, code-switching contexts, and sociolinguistically marked registers, to generalize findings and catalyze research (Buaphet et al., 23 Jan 2026).

A plausible implication is that the LLM pipeline’s detection–lookup–generation modularity, coupled with script- and context-adapted subword modeling, marks a more scalable and robust frontier for multilingual lexical normalization. Released evaluation toolkits, datasets, and error analyses provide a foundation for ongoing research into script-diverse normalization under real-world and cross-lingual constraints.