PureTC-1B: Stable Traditional Chinese Adapter

• PureTC-1B is an adapter-only, three-stage fine-tuned model built on Llama-3.2-1B-Instruct to ensure pure Traditional Chinese output and prevent code-switching errors.
• It employs Continual Pre-Training, Supervised Fine-Tuning, and Direct Preference Optimization to adjust language priors and enhance instruction-following for TC stability.
• Evaluations demonstrate significant reductions in non-TC token outputs, with marked improvements in benchmarks like Named Entity Translation.

Searching arXiv for the specified paper and closely related methodological context. PureTC-1B is an adapter-only, three-stage fine-tuned version of Llama-3.2-1B-Instruct designed to generate more stable, cleaner Traditional Chinese (TC). Its defining objective is not generic task accuracy but token-level instability: under Traditional Chinese prompts, small open-weight models may emit non-TC characters, Simplified Chinese, English, Japanese, or otherwise code-switched outputs. The system addresses this with a sequential pipeline of Continual Pre-Training (CPT), Supervised Fine-Tuning (SFT), and Direct Preference Optimization (DPO), while keeping the base model frozen and training only LoRA adapters. On the paper’s main benchmark, PureTC-1B reports a 51.3% relative reduction in non-TC output tokens versus the base model in micro-average OLR, and on Named Entity Translation (NET) it reports a 77.2% relative reduction relative to Llama-3B and 57.2% relative to Qwen-1.5B (Chih et al., 2 Oct 2025).

1. Problem definition and deployment motivation

The paper frames PureTC-1B around a specific failure mode in Traditional Chinese small LLM deployment: token-level linguistic stability. The concern is that a model can answer correctly at the semantic level while still producing outputs contaminated by unintended code-switching, insertion of Simplified Chinese, English, Japanese, or mixed-script segments. In the paper’s formulation, the central production requirement is the ability to maintain pure Traditional Chinese output under Traditional Chinese prompting, rather than merely achieving acceptable downstream task performance (Chih et al., 2 Oct 2025).

This deployment gap is presented as particularly consequential for government, law, education, and other professional settings, where even a small number of wrong-script or wrong-language tokens can reduce trust and usability. The proposed remedy is a practical stabilization recipe that first shifts language priors toward Traditional Chinese, then aligns instruction-following behavior, and finally explicitly prefers purer TC outputs over semantically similar but code-switched alternatives.

An important nuance is terminological. The paper uses the phrase “token-level instability,” but its operational measurement is character-based rather than tokenizer-token-based. Thus, what is diagnosed as token-level instability is evaluated as character-level contamination under a strict TC-only legality policy.

2. Backbone model and adapter-only design

The exact backbone is

$\texttt{meta-llama/Llama-3.2-1B-Instruct}$

which the paper describes as an open-weight, instruction-tuned model released by Meta. The choice is motivated by practical deployment considerations: it is a 1B-scale small LLM suitable for cost-sensitive settings, already instruction-tuned, open-weight, and amenable to adaptation without full-model retraining through LoRA (Chih et al., 2 Oct 2025).

The approach is explicitly adapter-only. The base model remains frozen throughout training, while LoRA adapters are applied to all linear layers in attention blocks and all linear layers in MLP/feed-forward blocks. The paper emphasizes that this substantially reduces trainable parameter count and memory use, making the pipeline feasible on a single consumer GPU and practical for 12 GB GPUs.

Although the paper does not print the LoRA equation explicitly, it invokes the standard parameterization:

$W' = W + \Delta W$

with low-rank update

$\Delta W = BA$

and a common scaling form

$W' = W + \frac{\alpha}{r}BA$

where $W$ is the frozen pretrained weight, $A$ and $B$ are trainable low-rank matrices, $r$ is the LoRA rank, and $\alpha$ is the LoRA scaling coefficient. In PureTC-1B, this design is operationally coupled to staged adapter merging: train CPT adapter, merge; continue with SFT, merge; then train a thin DPO adapter and merge again. This produces a single final checkpoint rather than a stack of live adapters.

3. Three-stage stabilization pipeline

PureTC-1B is trained in three sequential stages: Continual Pre-Training, Supervised Fine-Tuning, and Direct Preference Optimization. Each stage has a distinct role in the stabilization process (Chih et al., 2 Oct 2025).

Continual Pre-Training (CPT). CPT uses a causal language modeling objective to shift the model’s language priors toward Traditional Chinese. It proceeds in two steps. First, a high-rank adapter is trained for one epoch on a broad Traditional Chinese corpus and then merged. Second, a new adapter is trained on the updated model using a specialized legal corpus for multiple epochs. The broad corpus is Taiwan-Text-Excellence-2B, described as a large-scale high-quality TC collection of news/articles with approximately 1.78 million documents and roughly 2 billion tokens. The specialized corpus is republic_of_china_judgements_4_continue_pretrain, approximately 300 MB of Taiwanese legal judgments. The datasets were already preprocessed by their publishers through document-level deduplication, length filtering, and general text cleaning; the authors explicitly state that they did not apply additional script normalization and did not apply code-switch filtering. CPT uses context length 8192, LoRA rank $r=512$, LoRA alpha $W' = W + \Delta W$0, LoRA dropout 0.4, packing enabled, AdamW, cosine learning rate schedule, 3% warmup, per-device batch size 1 with gradient accumulation, bf16, FlashAttention, and native RoPE. The learning rate is $W' = W + \Delta W$1 for the general corpus step and $W' = W + \Delta W$2 for the specialized corpus step. The high-rank LoRA is justified as providing enough capacity for language adaptation without unfreezing the full base model.

Supervised Fine-Tuning (SFT). SFT is intended to align instruction-following behavior, especially single-language output, context faithfulness, and structured long-form generation. The SFT dataset is synthetic and contains 1,500 examples generated using GPT-4-mini from curated prompts, organized as five tasks with 300 examples per task, formatted as (instruction, input, output) under the Llama 3 instruction template. The five task types are Chinese to English Translation; English to Chinese Translation; Faulty-Premise Question Answering; Grounded Self-Ask/Answer; and Hierarchical NER and Outline Extraction. The design principles are explicitly single-language output enforcement, faithfulness to context, and structure control for long-form generation. SFT continues training the high-rank CPT adapter with context length 4096, LoRA rank $W' = W + \Delta W$3, LoRA alpha $W' = W + \Delta W$4, LoRA dropout 0.4, learning rate $W' = W + \Delta W$5, packing disabled, AdamW, cosine schedule, 3% warmup, and per-device batch size 1 with gradient accumulation. Packing is disabled to avoid cross-sample interference during instruction tuning.

Direct Preference Optimization (DPO). DPO is the final alignment stage and is the stage most explicitly targeted at TC adherence as a preference. The DPO dataset is derived from the same five instruction task families used in SFT. For each prompt $W' = W + \Delta W$6, the preferred response $W' = W + \Delta W$7 is generated by GPT-4-mini and curated to be pure Traditional Chinese only and semantically faithful; the rejected response $W' = W + \Delta W$8 is generated by the authors’ own CPT+SFT model and selected because it exhibits undesirable behavior such as code-switching or faithfulness violations. The preference dataset is written as

$W' = W + \Delta W$9

Before DPO, the SFT adapter is merged into the base model. A new low-rank, thin LoRA adapter is then trained with LoRA rank $\Delta W = BA$0, LoRA alpha $\Delta W = BA$1, LoRA dropout 0.4, learning rate $\Delta W = BA$2, context length 4096, AdamW, cosine schedule, 3% warmup, and per-device batch size 1 with gradient accumulation. The paper’s stated rationale is that this preserves merged SFT capabilities while enabling efficient preference correction and avoiding catastrophic forgetting.

4. Evaluation policy, metrics, and benchmark construction

The paper’s evaluation framework is intentionally strict and centers on output purity rather than conventional task accuracy (Chih et al., 2 Oct 2025).

Before scoring, outputs are normalized by Unicode normalization NFKC and whitespace unification, while lowercasing is not applied so that Latin letters remain detectable. A character is considered TC-legal only if it is either a Han ideograph, defined as CJK Unified Ideographs plus Extensions and Compatibility Ideographs, or a decimal digit with Unicode General Category Nd. Everything else, including Latin letters, most punctuation, and special symbols, is treated as non-compliant. The paper notes that this is intentionally strict and does not perfectly correspond to natural writing, but functions as a clean stress-test for contamination.

Two metrics are introduced. The first is Other-Language Rate (OLR), a contamination-density measure defined over characters:

$\Delta W = BA$3

Lower OLR is better, and $\Delta W = BA$4 indicates that every character satisfies the TC-only policy. The second metric is Pass@TC:

$\Delta W = BA$5

This is a binary strict-purity measure indicating whether an output contains no illegal character at all. The distinction is significant: a model may reduce average contamination without frequently producing perfectly clean outputs.

The benchmark compares four models under identical decoding settings: Llama-3.2-1B-Instruct (Base), PureTC-1B (Ours), Llama-3B, and Qwen-1.5B. Decoding uses temperature 0.2, top-p 0.9, repetition penalty 1.05, and a listed but numerically omitted max new tokens field. The prompt pool is described as approximately 605 items in the detailed evaluation section, although an earlier contribution summary mentions a 660-prompt benchmark; the detailed section appears to provide the operative count used in the reported setup.

The prompt set is generated by GPT-4o-mini and grouped into eight task families: EN→TC Translation, Bilingual Purge, SC→TC Conversion, Structured JSON, Content Organization, Long-form Summary, Noise Robustness, and Entity Translation. The paper separately highlights Named Entity Translation (NET), which appears in the tables as the final task family and is the setting where PureTC-1B shows particularly strong gains against larger baselines.

Because the task families are not equally represented, the paper reports both macro-average and micro-average aggregation. Macro-average OLR is

$\Delta W = BA$6

which weights each family equally. Micro-average OLR is described as a weighted mean over all prompts:

$\Delta W = BA$7

The reported relative reduction in non-TC output tokens is effectively

$\Delta W = BA$8

5. Quantitative results and task-wise behavior

The paper’s headline empirical claim is that PureTC-1B reduces contamination substantially relative to the base 1B model, while also improving strict purity frequency (Chih et al., 2 Oct 2025).

On the main benchmark, macro-average OLR is reported as 0.214 for the base model and 0.098 for PureTC-1B. Macro-average Pass@TC rises from 9.5% to 29.9%, a gain of 20.4 percentage points. However, the paper contains an inconsistency in the reported macro-average relative reduction: the abstract states 54.1%, whereas Table 2 states 66.8%.

For micro-average results, the base model’s OLR is reported as 0.231. PureTC-1B’s micro-average OLR is given as 0.113 in the text and 0.104 in Table 1, indicating another inconsistency. The reported headline relative reduction is 51.3%. Micro-average Pass@TC rises from 10.4% for the base model to 30.3% for PureTC-1B, a gain of 19.9 percentage points. The paper summarizes this pattern as over 50% reduction in contamination and roughly a threefold increase in strict purity success.

Against larger baselines, the strongest comparison is on Named Entity Translation. PureTC-1B records OLR 0.057, compared with 0.250 for Llama-3B and 0.133 for Qwen-1.5B. The corresponding relative reductions are 77.2% versus Llama-3B and 57.2% versus Qwen-1.5B. Pass@TC on NET is 28.7% for PureTC-1B, compared with 6.4% for Llama-3B, 12.8% for Qwen-1.5B, and 3.2% for the base 1B model.

The broader comparison tables indicate that, in macro-average terms, PureTC-1B improves over Llama-3B by +69.5% in OLR and +18.9 percentage points in Pass@TC, and over Qwen-1.5B by +44.7% in OLR and +5.3 percentage points in Pass@TC. In micro-average terms, it improves over Llama-3B by +59.6% in OLR and +18.5 percentage points in Pass@TC, and over Qwen-1.5B by +28.4% in OLR while showing a -1.0 percentage point change in Pass@TC. This last comparison indicates a trade-off: lower contamination density does not always coincide with higher strict-pass frequency.

Per-task results show gains in 7 of 8 task families. The strongest areas highlighted are Bilingual Purity, SC→TC Conversion, Named Entity Translation, Structured JSON, Long-form Summary, and Content Organization. Illustrative cases include Bilingual Purity, where PureTC-1B reaches OLR 0.016 versus 0.125 for the base model and Pass@TC 59.6% versus 33.3%; SC Conversion, where it reaches OLR 0.014 versus 0.062 and Pass@TC 46.3% versus 14.8%; and Long-form Summary, where it reaches OLR 0.017 versus 0.130 and Pass@TC 26.7% versus 2.2%.

The sole reported regression is Noise Robustness. On that task, PureTC-1B has OLR 0.467, compared with 0.389 for the base model, 0.329 for Llama-3B, and 0.480 for Qwen-1.5B. The authors interpret this as a tendency to over-translate or transliterate noise such as URLs, code, and emojis rather than ignoring it.

6. Limitations, reproducibility, and broader significance

The paper is explicit that its evaluation regime is severe and that some apparent errors under OLR may not always be practically unacceptable (Chih et al., 2 Oct 2025).

One limitation is metric strictness. Because punctuation, Latin letters, and many symbols are treated as illegal, OLR and Pass@TC are useful as stress metrics for contamination but can over-penalize outputs that might be acceptable in real use. The authors also note that the goal is not to remove English at all costs; some English tokens may be legitimate, including proper nouns and technical acronyms. A plausible implication is that the benchmark is better understood as a controlled purity probe than as a full utility measure for all professional writing contexts.

A second limitation is task imbalance, which is why both macro and micro averaging are needed. A third is baseline scope: the comparisons include Llama-3B and Qwen-1.5B but not larger 7B+ models, which the paper attributes to hardware constraints. The most concrete behavioral failure is the Noise Robustness regression, where the model tends to rewrite noise rather than suppress it.

Reproducibility is substantial but partial. Reported implementation details include the use of LLaMA-Factory, bf16, FlashAttention, native RoPE, the original unchanged tokenizer, LoRA applied to all linear layers in attention and MLP blocks, AdamW, cosine scheduling, 3% warmup, per-device batch size 1, CPT context length 8192, SFT and DPO context length 4096, CPT/SFT LoRA settings of $\Delta W = BA$9, $W' = W + \frac{\alpha}{r}BA$0, dropout 0.4, DPO LoRA settings of $W' = W + \frac{\alpha}{r}BA$1, $W' = W + \frac{\alpha}{r}BA$2, dropout 0.4, and learning rates of $W' = W + \frac{\alpha}{r}BA$3, $W' = W + \frac{\alpha}{r}BA$4, $W' = W + \frac{\alpha}{r}BA$5, and $W' = W + \frac{\alpha}{r}BA$6 across the respective stages. However, the paper does not specify the exact GPU model, exact VRAM usage during training, total training time, number of SFT or DPO epochs, total training steps, checkpoint save strategy, inference max new tokens, or exact gradient accumulation count.

The broader significance claimed for PureTC-1B is methodological rather than language-exclusive. The paper argues that the CPT→SFT→DPO sequence is not inherently TC-specific: continual pre-training strengthens target-language priors, instruction tuning aligns desired behavior, and preference optimization penalizes mixed-language leakage. This suggests a general stabilization blueprint for other non-English languages that suffer from multilingual interference or code-switching instability in open-weight small models.