Norwegian-Extended Tokenizer

• The Norwegian-Extended Tokenizer is a subword tokenization strategy that utilizes SentencePiece to handle Norwegian’s rich morphology and dialectal diversity.
• It constructs specialized vocabularies from curated multilingual corpora, optimizing language model performance in both NLU and NLG tasks.
• Advanced transfer methods like AIM warm-up retrofit preexisting models, ensuring efficient adaptation from English-centric frameworks to Norwegian support.

The Norwegian-Extended Tokenizer refers to a class of tokenization strategies and artifacts developed to provide robust subword representations for Norwegian and related Scandinavian languages, with special attention to the linguistic idiosyncrasies and rich morphology characteristic of these low-resource settings. The central approach is rooted in data-driven subword tokenization using the SentencePiece toolkit, applied to curated multilingual and Norwegian-centric corpora, often with increased vocabulary sizes to account for morphological complexity and dialectal diversity. These tokenizers are adopted in large-scale Scandinavian and multilingual LLMs, are evaluated for downstream performance (e.g., NLU, NLG, MT), and are increasingly complemented by advanced transfer methods for retrofitting preexisting English-centric LLM architectures to support Norwegian with maximal coverage and efficiency.

1. Algorithmic Foundations and Tooling

The Norwegian-Extended Tokenizer is instantiated primarily using the SentencePiece framework. SentencePiece is an unsupervised text tokenizer and detokenizer that implements subword units via BPE (byte pair encoding) or unigram LLM algorithms. Although several studies (e.g., "Training and Evaluation of a Multilingual Tokenizer for GPT-SW3" (Stollenwerk, 2023)) specify BPE, recent large-scale Norwegian LLMs such as NorwAI's NorLLM family merely report that “SentencePiece” is used, without explicitly distinguishing between BPE and unigram-LM variants (Gulla et al., 6 Jan 2026). The underlying pipeline is purely off-line, meaning no neural or hybrid token-classification layers are used during main model training or inference.

Each model family—e.g., NorGPT, NorwAI-Mistral, NorwAI-Mixtral, NorwAI-Magistral—uses a distinct SentencePiece vocabulary file, with sizes chosen empirically (64,000; 68,000; 158,000 subword tokens). Vocabulary learning is conducted jointly over large multilingual corpora, with Norwegian (Bokmål, Nynorsk, and Sámi) data as the primary focus but incorporating Danish, Swedish, German, and secondary English for improved code-switching robustness (Gulla et al., 6 Jan 2026).

2. Vocabulary Construction and Linguistic Adaptation

The construction of the Norwegian-Extended Tokenizer vocabulary is shaped by the selection and preprocessing of training corpora, not by hand-written tokenization grammars or language-specific heuristics. SentencePiece operates at the raw UTF-8 Unicode level: no diacritic normalization is performed, and Norwegian-specific characters such as å, æ, ø are included natively. Compound words in Norwegian—a key typological feature—are implicitly managed through learned subword splits and merges. For example, “langtidsvirkning” is split as “▁lang”, “tids”, “virk”, “ning” (Gulla et al., 6 Jan 2026). Downstream linguistic coverage is achieved by inclusion of monolingual and cross-lingual resources: CulturaX, HPLT, Norwegian Colossal Corpus for Norwegian; expanded in later corpus versions to include Sámi; General Scandinavian material ensures coverage for dialects and related forms.

Empirical studies in related settings (e.g., GPT-SW3 (Stollenwerk, 2023)) demonstrate that roughly 15–19% of induced subwords correspond to high-frequency Norwegian morphemes. Vocabulary overlap between a broad multilingual vocab and a Norwegian-only trained vocab is found to be ≈58% at 64,000 tokens, demonstrating substantial but not complete language-specific optimization.

3. Tokenizer Transfer and Retrofitting Strategies

A key limitation of deploying LLMs with fixed, English-centric tokenizers is inadequate coverage and morphological adaptation for Norwegian. Model-Aware Tokenizer Transfer (MATT) provides a two-stage framework to retrofit such models with a new Norwegian-extended tokenizer (Haltiuk et al., 24 Oct 2025). Stage 1 (“AIM warm-up”) matches segment-level inter-token attention patterns from an original model (teacher/tokenizer T) to the new model (student/tokenizer T′) by freezing original weights and only updating embeddings for new tokens. The core loss minimizes discrepancy between aggregated attention-weighted segment representations:

$$L_{AIM} = \frac{2}{m(m+1)}\sum_{i=1}^{m}\sum_{j=1}^{i} L^*(\mathfrak{s}_{\ell_T(i),j}, \mathfrak{s}'_{\ell_{T'}(i),j})$$

where $L^*$ is typically mean-squared error or cosine distance. Stage 2 resumes LLM fine-tuning over Norwegian data, often requiring only a single epoch to recover upwards of 90% of native-tokenizer accuracy. Best practices include using a moderately large Norwegian-specific vocabulary (50k–100k), preserving all Norwegian-specific characters, and validating coverage/fertility on held-out sets.

4. Tokenization Workflow and Practical Usage

Downstream tokenization utilizes the generated SentencePiece vocabulary files directly. Model inference is mediated via standard frameworks (e.g., AutoTokenizer from HuggingFace Transformers) and expects raw Norwegian UTF-8 text, with no required pre-normalization or OOV handling beyond SentencePiece’s native <unk> token. For resource-constrained deployments, quantized GGUF versions are produced; these retain the SentencePiece vocabulary structure, altering only storage of token IDs and embeddings (Gulla et al., 6 Jan 2026).

Preprocessing for tokenizer training includes: (1) deduplication of documents, (2) language identification to select Norwegian and related languages, (3) normalization (e.g., Unicode NFC), and (4) sentence segmentation to avoid skewing statistics with very long sequences.

Example tokenization (68k-token Norwegian model):

Input	Tokens
Hvorfor heter det osloboer når man bor i Oslo?	▁Hvorfor ▁heter ▁det ▁oslo bof er ▁når ▁man ▁bor ▁i ▁Oslo ▁?

Token IDs (illustrative): 111, 54, 23, 111, 876, 4321, 78, 12, 33, 47, 876, 17.

5. Coverage, Efficiency, and Analytical Metrics

The Norwegian-Extended Tokenizer’s efficiency and granularity are typically measured through fertility (average subwords per word), proportion of continued (split) words, vocabulary overlap, and OOV rates. While the NorwAI report (Gulla et al., 6 Jan 2026) does not publish these statistics, GPT-SW3 analysis (Stollenwerk, 2023) finds:

• Fertility: $f \approx 1.06$ (tokens/word) for Norwegian.
• Continued word ratio: $p \approx 0.07$ (proportion of words split into multiple tokens).
• OOV: Nearly zero, due to byte-fallback (all characters covered).
• Vocabulary overlap: ≈ 58% between multilingual and Norwegian-only vocab at 64k tokens.

These figures indicate high coverage and efficient subword granularity, comparable to Swedish and Danish. Byte-fallback guarantees that rare Norwegian-specific diacritics and character forms are not lost to <unk> tokens.

6. Alternative Formalisms: Flat Automata Tokenizers

In contexts requiring precise control over tokenization (e.g., lexers for programming or symbolic processing), flat automata can be constructed to explicitly accommodate the Norwegian alphabet, including all relevant diacritics (Nivelle et al., 2022). These systems define border functions over the Unicode interval, efficiently handle intervals such as æ, ø, å, and their uppercase variants, and compose regular-language token patterns using operations like concatenation, union, and Kleene star. The process includes determinization and minimization to produce compact DFAs suitable for real-time C++ tokenizers, with Δ-transitions parameterized by code-point borders without hand-written character-class logic.

7. Best Practices, Limitations, and Recommendations

For practical developments and evaluations:

• Ensure training data represent both Bokmål and Nynorsk, as imbalanced corpora can bias the vocabulary to one variant (Haltiuk et al., 24 Oct 2025).
• Vocabulary size should balance coverage of morphology and computational tractability (50k–158k empirically observed) (Gulla et al., 6 Jan 2026, Haltiuk et al., 24 Oct 2025).
• Explicit Norwegian subword quantization is empirically unnecessary—SentencePiece subword learning and careful UTF-8 character coverage suffice for robust downstream accuracy, provided the training corpus is representative (Stollenwerk, 2023).
• Model-aware transfer methods like AIM warm-up (MATT) are recommended for retrofitting pre-existing architectures, outperforming semantic shortcut initializers in both generative and discriminative tasks (Haltiuk et al., 24 Oct 2025).
• Developers are advised to check new-token embedding drift, fertility, and parity, and run ablations on transfer pipelines to verify improved Norwegian performance.

The Norwegian-Extended Tokenizer thus comprises a family of empirically derived, robust, and coverage-maximizing subword tokenizers specialized for the demands of Norwegian and closely related languages. Its evolution reflects both advances in unsupervised segmentation and practical transfer techniques for enabling Nordic language support in modern LLM architectures.