HuffmanTranslit: Reversible Low-Resource Transliteration

• HuffmanTranslit is a reversible transliteration framework that maps native scripts to Latin code-words with 100% lossless recovery.
• It employs Huffman coding to compress transliterated text, achieving up to 2.6× file reduction and 3.9× token reduction.
• The framework integrates with LLM tokenizers via optimized strategies, improving minority language classification by 30–50 F1 points.

Searching arXiv for the specified paper and closely related transliteration context. arxiv_search: {"query":"(Zhuang et al., 22 Sep 2025) HuffmanTranslit reversible transliteration Huffman-based low-resource languages","max_results":5,"sort_by":"submittedDate"} HuffmanTranslit is a reversible transliteration framework for low-resource languages that combines character transliteration with Huffman coding in order to improve cross-lingual transfer in LLMs while preserving exact recoverability of the source script (Zhuang et al., 22 Sep 2025). It is presented as a complete framework for integrating transliteration into LLM training and deployment, with four stated properties: compression, accuracy, efficiency, and scalability. In the reported formulation, the method reduces storage requirements for low-resource language content, achieves up to 50% reduction in file size and 50–80% reduction in token count, guarantees 100% lossless conversion from transliterated text back to the source language, eliminates the need for vocabulary expansion for low-resource languages, and is designed to be extendable to other low-resource languages (Zhuang et al., 22 Sep 2025).

1. Problem setting and design rationale

The framework is motivated by a specific limitation of multilingual LLMs: although such models are trained on increasingly diverse and extensive multilingual corpora and demonstrate cross-lingual transfer capabilities, these capabilities often fail to effectively extend to low-resource languages, particularly those utilizing non-Latin scripts (Zhuang et al., 22 Sep 2025). Transliteration into Latin script is treated as a natural solution, but the paper identifies the lack of a comprehensive framework for integrating transliteration into both LLM training and deployment.

HuffmanTranslit takes a pragmatic approach by combining two components that are usually discussed separately: a reversible character-level transliteration scheme and Huffman coding for compression. The transliteration layer addresses script mismatch and tokenizer compatibility, while the coding layer targets storage efficiency. The framework is validated on multiple downstream tasks, specifically text classification, machine reading comprehension, and machine translation, and the reported experiments show improved processing of low-resource languages while maintaining performance on high-resource languages (Zhuang et al., 22 Sep 2025).

A central implication of this construction is that transliteration is not treated merely as a preprocessing heuristic. Instead, it is formulated as an end-to-end representational interface between the native script, the LLM tokenizer, the model’s continued pre-training regime, and the post-processing step that restores native-script outputs. This suggests that the contribution lies as much in systems integration as in the transliteration map itself.

2. Reversible transliteration mapping

The formal core of HuffmanTranslit is a pair of bijective maps between source-language characters and Latin “code-words.” Let $\Sigma_s$ be the set of source-language characters and $\Sigma_l$ the set of Latin code-words used for transliteration. The framework defines

$M_{c2l} : \Sigma_s \to \Sigma_l,$

and

$M_{l2c} = (M_{c2l})^{-1} : \Sigma_l \to \Sigma_s.$

Given an input string

$s = c_1 c_2 \dots c_n, \quad c_i \in \Sigma_s,$

the transliteration function is

$\mathrm{ToLatin}(s) = \ell_1 \ell_2 \dots \ell_n,$

where $\ell_i = M_{c2l}(c_i)$. Restoration is defined as

$$\mathrm{FromLatin}(\ell_1 \ell_2 \dots \ell_n) = c_1 c_2 \dots c_n,$$

where $c_i = M_{l2c}(\ell_i)$ (Zhuang et al., 22 Sep 2025).

To preserve characters outside $\Sigma_s$, including emojis and punctuation, the scheme escapes them by wrapping them in @…@. The stated guarantee is exact round-trip recovery:

$\Sigma_l$0

The paper explicitly characterizes this as 100% lossless round-trip conversion. That property distinguishes the framework from transliteration systems that optimize readability or approximate phonology but do not enforce perfect invertibility. A common misconception is that transliteration necessarily sacrifices script fidelity; in this framework, fidelity is preserved by construction because the mapping is bijective rather than many-to-one (Zhuang et al., 22 Sep 2025).

The encoding design in Stage 1 is described as using Unicode frequencies to determine the core $\Sigma_l$1 and assigning code-words via the Huffman + “First-letter-capitalized + lowercase” scheme. The description does not frame the Latin strings as conventional romanization; rather, they function as code-words engineered for reversibility and tokenizer interaction.

3. Huffman coding and compression objective

In addition to the one-to-one mapping $\Sigma_l$2, HuffmanTranslit applies a variable-length Huffman code over the Latin alphabet to compress transliterated sequences (Zhuang et al., 22 Sep 2025). Let $\Sigma_l$3 with empirical frequencies $\Sigma_l$4 on a training corpus. The normalized weights are defined as

$\Sigma_l$5

The objective is to assign each symbol $\Sigma_l$6 a binary codeword of length $\Sigma_l$7 so as to minimize the expected code length

$\Sigma_l$8

subject to the prefix-free property.

The paper provides the standard Huffman-tree construction adapted to this setting: initialize a min-priority queue with symbol-weight pairs, repeatedly extract the two nodes with minimum weight, merge them into a new internal node whose weight is the sum of the two, and then recursively traverse the final tree to assign bitstrings by appending 0 on left branches and 1 on right branches. By classical Huffman theory, the resulting code satisfies

$\Sigma_l$9

where

$M_{c2l} : \Sigma_s \to \Sigma_l,$0

is the source entropy (Zhuang et al., 22 Sep 2025).

This compression layer is separate from the reversible character mapping. The transliterated sequence remains the semantic carrier for LLM training and inference, whereas the Huffman encoding is used to compress the transliterated stream losslessly for storage. The paper therefore distinguishes two compression notions: file-size compression and token-count compression. If $M_{c2l} : \Sigma_s \to \Sigma_l,$1 and $M_{c2l} : \Sigma_s \to \Sigma_l,$2 denote the byte sizes of the original and transliterated plus Huffman-encoded files, then file compression is defined as

$M_{c2l} : \Sigma_s \to \Sigma_l,$3

If $M_{c2l} : \Sigma_s \to \Sigma_l,$4 and $M_{c2l} : \Sigma_s \to \Sigma_l,$5 are the subword-token counts produced by the LLM tokenizer on the original and transliterated text, then token compression is defined as

$M_{c2l} : \Sigma_s \to \Sigma_l,$6

For Tibetan (bo), the paper reports that the original file corresponds to 1 × file, 6.92× tokens, while HuffmanTranslit (Tokenizer-optimized) yields file 2.61×, tokens 1.80×. In the compression analysis table, selected results are: bo with Token-Opt at 2.61× file compression and 1.80× token compression, mn with Token-Opt at 2.07× and 3.85×, and ug with Token-Opt at 1.27× and 1.39×; the summary statement is that the method achieves up to 2.6× file reduction and 3.9× token reduction simultaneously (Zhuang et al., 22 Sep 2025).

4. Tokenizer and vocabulary integration

A distinctive claim of HuffmanTranslit is that it does not enlarge the model’s embedding matrix beyond a small set of Latin code-words. The paper studies three integration strategies with the Llama2 tokenizer and vocabulary (Zhuang et al., 22 Sep 2025).

The Basic Strategy keeps the original 32 K tokens of Llama2. Under this setup, each source-character code, such as “B” or “Aa”, typically spans multiple tokens, and the paper characterizes this as sub-optimal.

The Tokenizer-Optimized Strategy analyzes Llama2’s tokenizer and chooses 162 single-token Latin code-words such as B–X and Aa–Fk to cover all $M_{c2l} : \Sigma_s \to \Sigma_l,$7. Under this strategy, each source character maps to exactly one existing token, so $M_{c2l} : \Sigma_s \to \Sigma_l,$8 is minimized.

The Hybrid Vocabulary Strategy trains a small 4 K subword vocabulary over transliterated text, using common digrams and trigrams, and merges it with the original 32 K vocabulary to obtain 33 738 total tokens. The paper reports that this strategy achieves the best average compression, specifically 3.04× file, 2.23×–4.98× token reduction across bo,mn,ug (Zhuang et al., 22 Sep 2025).

In all three cases, embedding lookups remain unchanged for original Llama2 tokens; for the hybrid strategy, new code-words use the same 768-dim embeddings, which are randomly initialized and then updated during continued pre-training. The framework’s efficiency claim is therefore not only about shorter files or fewer tokens, but also about avoiding large-scale vocabulary expansion for low-resource languages.

The reported baselines make the tokenizer issue especially visible. In the text classification experiments, the comparison includes Expanded Vocab and UROMAN alongside the three HuffmanTranslit variants. This suggests that the paper positions reversible transliteration not as a generic romanization alternative, but as a tokenizer-aware representational design whose performance depends on how cleanly source characters align to existing model tokens.

5. Training and inference pipeline

The framework is organized into three stages, referenced in the paper as Figure 1 (Zhuang et al., 22 Sep 2025).

Stage 1 – Character Frequency & Encoding Design samples approximately 3 000 lines per low-resource language, computes Unicode frequencies to determine the core $M_{c2l} : \Sigma_s \to \Sigma_l,$9, and assigns code-words via the Huffman + “First-letter-capitalized + lowercase” scheme.

Stage 2 – Transliteration & Model Continued Pre-training converts multilingual raw text to Latin code-words, with Chinese→pinyin and English unchanged. Huffman bit-encoding may optionally be applied for storage. The model is then continued-pretrained using Llama2-7B on the transliterated corpus for 1 epoch with LR 1e-4, batch 1024, and bf16.

Stage 3 – End-to-End Inference begins with a FastText input-language classifier that detects bo/mn/ug/zh/en/other. If the input is low-resource, it is transliterated via $M_{l2c} = (M_{c2l})^{-1} : \Sigma_l \to \Sigma_s.$0\Sigma_s$M_{l2c} = (M_{c2l})^{-1} : \Sigma_l \to \Sigma_s.$1\mathrm{FromLatin}(\cdot)$, and for Chinese it applies the pinyin→Chinese seq2seq converter, returning a native-script answer (Zhuang et al., 22 Sep 2025).

This pipeline clarifies that HuffmanTranslit is not only a corpus transformation method. It is a deployment-time interface that supports reversible entry into and exit from a Latin-coded internal representation. A plausible implication is that the framework is designed to preserve the operational simplicity of a Latin-script LLM interface while maintaining native-script usability at the system boundary.

6. Empirical performance and interpretive significance

The evaluation covers three downstream tasks and follows a consistent transfer setting: fine-tuning on a high-resource language, specifically zh or en, and then zero-shot evaluation on bo, mn, and ug (Zhuang et al., 22 Sep 2025).

For text classification on WCM-v2, the reported metrics are Accuracy and F1 on bo, mn, ug, and zh. The Base Llama2 model yields minority-language aggregate performance of 13.48/9.01 and all-language performance of 48.15/48.78. By comparison, Expanded Vocab obtains 62.58/64.56 on minorities and 75.64/76.34 overall; UROMAN obtains 62.10/63.47 and 75.33/75.84; Basic Trans. obtains 63.40/65.17 and 76.06/76.60; Token-Opt Trans. obtains 60.80/63.54 and 74.81/75.86; and Hybrid Trans. obtains 58.80/62.15 and 73.68/75.02. The paper’s summary statement is that all three HuffmanTranslit variants outperform the base model on minority-language F1 by 30–50 pts without harming Chinese (Zhuang et al., 22 Sep 2025).

For machine reading comprehension on TibetanQA zero-shot, the fine-tuning source is either CMRC (zh) or SQuAD (en). Under CMRC→bo, the Base model reports EM 7.9 and F1 45.8, while Token-Opt Trans. reports EM 16.0 and F1 60.2. Under SQuAD→bo, the scores rise from EM 6.5/F1 50.8 to EM 13.5/F1 66.5 under Token-Opt (Zhuang et al., 22 Sep 2025).

For machine translation on Flores-200 with 3-shot prompts, the metrics are BLEU ↑, chrF ↑, [TER](https://www.emergentmind.com/topics/trusted-emotion-recognition-ter) ↓. The paper reports zh→bo (Base) at BLEU 3.5, chrF 0.28, TER 0.92, and zh→bo (Token-Opt) at BLEU 6.3, chrF 0.39, TER 0.82. For Uyghur, zh→ug (Base) is BLEU 4.2, chrF 0.31, TER 0.89, and zh→ug (Token-Opt) is BLEU 7.0, chrF 0.42, TER 0.79. The paper summarizes this as +2.8 BLEU on Tibetan, +2.8 on Uyghur over Base Llama2 (Zhuang et al., 22 Sep 2025).

Taken together, the reported results support the paper’s concluding summary: HuffmanTranslit, and particularly the Tokenizer-optimized variant, achieves 100 % reversible transliteration via bijective $M_{l2c} = (M_{c2l})^{-1} : \Sigma_l \to \Sigma_s.$2, uses classic Huffman coding for bit-level storage compression, integrates with Llama2’s tokenizer in a single-token mapping without extra embedding expansion, improves cross-lingual transfer on bo/mn/ug by 30–50 F1 points on classification, 10+ F1 on MRC, and +2.8 BLEU on translation, and compresses data by 2–3× in file size and 1.5–4× in token count relative to raw UTF-8 (Zhuang et al., 22 Sep 2025). The data, code, mappings, and pre-trained models are stated to be publicly available at the project repository linked in the paper.