TOBA Tokenizer for Indonesian NLP

• TOBA Tokenizer is a linguistically-informed system that segments Indonesian text using deterministic syllabification and byte-pair encoding.
• It leverages Indonesian morphophonology to construct a high-coverage, compact vocabulary while minimizing computational load.
• Empirical evaluations using Rényi efficiency and benchmark comparisons demonstrate its superior performance over standard subword tokenizers.

The TOBA Tokenizer (“Tokenisasi Optimum Berbasis Aglutinasi”) is a hybrid, linguistically-informed tokenization system designed specifically for Indonesian LLMs. It operationalizes a two-stage pipeline: deterministic, rule-based syllabification exploiting Indonesian morphophonology, followed by byte-pair encoding (BPE) over these syllable and character units. This approach yields a compact, high-coverage vocabulary that aligns with both linguistic and information-theoretic criteria, offering marked improvements in efficiency, compression, and computational tractability for Indonesian natural language processing (Situngkir et al., 14 Jan 2026).

1. Linguistically Informed Syllable Segmentation

Initial token segmentation in the TOBA Tokenizer applies a deterministic, linear-time ($O(n)$) operation $T : X^* \to T^*$, where $X$ is the full Unicode character set in the training corpus. The algorithm is grounded in Indonesian orthography and the Max-Onset Principle, constraining permissible consonant clusters to those attested in the language (e.g., “kr,” “pl,” “tr,” but not “bt”). Digraphs (e.g., “ng,” “ny,” “sy”) are treated as atomic consonantal units. The steps are as follows:

1. Vowel identification: All positions of vowels ($V = \{a, e, i, o, u, ê\}$) are located.
2. Onset determination: Consonants between vowels are assigned to onsets where legal; illegal onsets are attached as codas.
3. Syllable segmentation: If a candidate syllable is not in a precomputed high-frequency set $\Sigma$ (see Section 3), it is decomposed into constituent characters.

Empirical frequency counts over the entire Indonesian Wikipedia and folklore corpora inform the high-frequency syllable set. Example segmentations include:

• “bahasa” → ba-ha-sa
• “menikmati” → me-nik-ma-ti
• “penyanyi” → pe-nyan-yi
• “komputer” → kom-pu-ter

This syllabification process preserves morphophonological boundaries crucial for Indonesian, internalizing character-level dependencies within meaningful units (Situngkir et al., 14 Jan 2026).

2. Byte-Pair Encoding Applied to Syllable Units

After segmentation, BPE is executed over the sequence of tokens drawn from the set $\Sigma$ (high-frequency syllables + all observed single Unicode characters) to construct larger subword units. The vocabulary is iteratively merged until a target size $N=3{,}500$ is achieved, using the following pseudocode:

$T : X^* \to T^*$9

Because $\Sigma$ already encapsulates common syllabic and character patterns, the number of merges required to reach higher-order morphological and compound structures is minimized. Any out-of-vocabulary unit observed at inference time falls back to character-level segmentation, preserving 100% coverage.

3. Information-Theoretic Evaluation: Rényi Efficiency

The TOBA Tokenizer’s segmentation quality is quantitatively evaluated using Rényi entropy of order $\alpha=2.5$, yielding the Rényi efficiency metric: $$H_\alpha(p_\Delta) = \frac{1}{1-\alpha} \log \left( \sum_{\delta \in \Delta} p_\Delta(\delta)^\alpha \right), \ \eta_\alpha = \frac{H_\alpha(p_\Delta)}{\log |\Delta|}.$$ On the Indonesian Wikipedia, the syllable-based BPE (TOBA) achieves $T : X^* \to T^*$0, significantly outperforming both byte-level BPE ($T : X^* \to T^*$1) and the GPT-2 tokenizer ($T : X^* \to T^*$2), despite using a vocabulary an order of magnitude smaller. This implies a more uniform and balanced token distribution.

4. Vocabulary Construction and Fallback Mechanism

The final TOBA vocabulary construction proceeds as follows:

• Syllable set $T : X^* \to T^*$3: Top 1,500 high-frequency syllables, augmented by all Unicode code points observed (to guarantee strict coverage), $T : X^* \to T^*$4.
• BPE merges: Iterated to reach a total vocabulary size of 3,500 tokens.
• Character-level fallback: At tokenization time, any OOV segmented unit is decomposed to single characters; since all corpus characters are in $T : X^* \to T^*$5, coverage is complete.

This compact vocabulary covers affixes, morphs, and syllable units characteristic of Indonesian agglutinative morphology, while the fallback ensures robustness during inference.

5. Empirical Performance and Comparative Results

TOBA’s segmentation and tokenization efficiency has been empirically evaluated against both monolingual and multilingual baselines on the Indonesian Wikipedia (“WikiID”) and PDBI folklore corpora. The following tables present summary metrics (Situngkir et al., 14 Jan 2026):

Segmentation/Unigram Model (Bits per Character):

Segmentation	Vocab	Bits/Segment (WikiID)	Chars/Segment (WikiID)	BPC (WikiID)
Character-level	—	—	—	2.88
Syllable-based	680	5.27	1.93	2.73
BPE	680	5.72	2.38	2.40
Syllable-based	1,166	5.69	2.17	2.62
BPE	1,166	6.17	2.73	2.26

Tokenizer Properties:

Tokenizer	Vocab size	Avg. Chars/Token (WikiID)	$T : X^* \to T^*$6 (WikiID)
Plain BPE	3,500	3.55	0.64
Syllable-based BPE	3,500	3.67	0.74
Pre-trained GPT-2	50,257	2.72	0.50
Pre-trained BERT	30,522	3.48	0.49

Despite the relatively small vocabulary size, TOBA generates longer tokens (average 3.67 characters) and a significantly higher $T : X^* \to T^*$7 compared to standard subword and character tokenizers.

6. Linguistic Alignment and Computational Properties

By constructing token units that map to actual syllable and affix boundaries in Indonesian, TOBA preserves morphological transparency. Common prefixes (me-, pe-, ke-), infixes (-el-, -em-), and suffixes (-an, -i) are often treated as independent tokens or merged with roots early in the BPE sequence. This explicit mapping of linguistic structure reduces model cross-entropy (per character $T : X^* \to T^*$8), accelerates convergence, and enhances data efficiency in transformer architectures.

Shorter token sequences (higher characters/token) achieved through compact vocabulary entail fewer attention operations and a reduced embedding matrix (3,500 entries vs. 30,000+ for baseline tokenizers), resulting in improved computational efficiency, lower memory footprint, and reduced FLOPs. This is particularly beneficial for deployment in resource-constrained environments (Situngkir et al., 14 Jan 2026).

7. Significance for Indonesian NLP and Broader Implications

The TOBA Tokenizer exemplifies an information-theoretically and linguistically grounded strategy for tokenization in morphologically rich and underresourced languages. Internalizing Indonesian’s agglutinative and morphophonological structure with a minimal vocabulary, TOBA bridges human literacy pedagogy (Gasing Literacy System) and statistical modeling. A plausible implication is that analogous frameworks may be effective for other agglutinative or morphologically complex languages with systematically defined syllabification and high-frequency morphs. The methodology demonstrates empirically that linguistically motivated token units can simultaneously optimize model efficiency and maintain or exceed performance of vastly larger subword vocabularies (Situngkir et al., 14 Jan 2026).