---
title: 'HSKBenchmark: Chinese SLA Evaluation Suite'
url: https://www.emergentmind.com/topics/hskbenchmark
type: topic
---

# HSKBenchmark: Chinese SLA Evaluation Suite

HSKBenchmark refers to a technical evaluation suite and framework designed for modeling, benchmarking, and advancing the understanding of Chinese second language acquisition (SLA) in large language models (LLMs) through a curriculum-tuning methodology. It targets phase-wise training and assessment of LLM writing abilities on authentic, level-segmented data reflecting the Hanyu Shuiping Kaoshi (HSK) proficiency standards, specifically HSK Levels 3–6. HSKBenchmark includes curated corpora, linguistically grounded evaluation criteria, synthetic writing task datasets, and an automated assessment agent, enabling systematic and reproducible study of LLMs in Chinese SLA [2511.15574].

## 1. Data Composition and Corpus Structure

HSKBenchmark comprises three core components:

- **Authentic Textbook Corpus:** A 6.76-million-token dataset, extracted from 79 widely-used Chinese-as-L2 textbooks (e.g., HSK Standard Course, Boya Chinese) covering HSK Levels 3–6. All Pinyin, English glosses, and images are removed.

| Level  | Tokens   | Sentences | Avg tokens/snt |
|--------|----------|-----------|----------------|
| HSK 3  | 895,037  | 22,743    | 39.35          |
| HSK 4  | 1,473,516| 34,637    | 42.66          |
| HSK 5  | 1,717,178| 41,044    | 41.84          |
| HSK 6  | 2,678,621| 63,650    | 42.08          |
| **Total** | 6,764,352| 162,074   | 41.74          |

- **Synthetic Instruction Samples:** 16,462 prompt-completion pairs, generated for 591 distinct grammar points defined by the Chinese Proficiency Grading Standards, covering types including word, phrase, fixed format, sentence component, sentence type, and emphatic usage. Generation employed GPT-4.1-mini, DeepSeek-Chat-V3, and Gemini-2.5-Flash with in-context 2-shot prompting. Manual verification (Fleiss’ κ = 0.91) ensured high validity (95%).

- **Test Topics:** 30 exam-style writing prompts, sampled from the HSK Dynamic Composition Corpus v2.0, span major genres (narrative, argumentative, descriptive, expository) and pragmatic functions. All test topics were held out from training data.

## 2. Curriculum-Tuning Methodology

Training in HSKBenchmark follows a staged curriculum closely mirroring human SLA. The methodology advances models through HSK Levels 3→4→5→6, simulating the “i + 1” learning paradigm.

At each level ℓ, the process is:

1. **Causal LM Pretraining** on the level-specific textbook set $\mathcal{T}^{(\ell)}$:
   $$
   \mathcal{L}_{PT}^{(\ell)} = -\sum_{x \in \mathcal{T}^{(\ell)}} \sum_{t=1}^{|x|} \log P_{\theta^{(\ell-1)}}(x_t | x_{<t})
   $$

2. **Instruction Tuning** on the associated synthetic dataset $\mathcal{D}^{(\ell)}$:
   $$
   \mathcal{L}_{IT}^{(\ell)} = -\sum_{(p, y) \in \mathcal{D}^{(\ell)}} \sum_{t=1}^{|y|} \log P_{\theta_{PT}^{(\ell)}}(y_t | p, y_{<t})
   $$

Model parameters evolve as:
$$
\theta_{PT}^{(\ell)} = \text{Pretrain}(\theta^{(\ell-1)}, \mathcal{T}^{(\ell)})
$$
$$
\theta_{IT}^{(\ell)} = \text{InstructTune}(\theta_{PT}^{(\ell)}, \mathcal{D}^{(\ell)})
$$

Optionally, aggregate objectives can employ curriculum loss weighting
$$
\mathcal{L}_\text{curric} = \sum_{\ell=3}^6 w_\ell (\mathcal{L}_{PT}^{(\ell)} + \mathcal{L}_{IT}^{(\ell)})
$$
where $w_\ell$ captures stage importance.

Base models include LLaMA2-7B-Chat, Mistral-7B-Instruct, and Chinese-Alpaca-2-7B, fine-tuned via LoRA. No curriculum baselines utilize GPT-4.1-mini, DeepSeek-Chat-V3, and Gemini-2.5-Flash.

## 3. Linguistically-Grounded Evaluation System

Evaluation in HSKBenchmark aims for multi-dimensional linguistic rigor using automated metrics and the HSKAgent:

- **Grammar Coverage ($C_g^{(\ell)}$):** Fraction of distinct grammar points used at each HSK level—computed by binary classification across all 591 points.
- **Writing Errors:** Quantified by the sum of character-level, lexical, syntactic, and discourse-level errors per output.
- **Lexical Complexity:** Includes type-token ratio (TTR), moving-average TTR (MATTR), and measure of textual lexical diversity (MTLD).
- **Syntactic Complexity:** Captures mean sentence length (MSL), mean dependency distance (MDD), subordination index (SI), and parse-tree depth (PTD).
- **Holistic Scoring:** Mimics HSK criteria on a 0–100 scale, aggregating length, task fulfillment, cohesion, structural range, and accuracy. The automated rubric achieves quadratic weighted kappa (QWK) ≈ 0.80 with human gold.

## 4. Automated Assessment: HSKAgent

HSKAgent is a fine-tuned evaluation agent based on Qwen3-8B, extended with LoRA for all core linguistic tasks:

- **Grammar Detection:** F1 = 0.97.
- **Error Detection:** Accuracy = 90%.
- **Holistic Scoring:** F1 = 0.81, QWK = 0.7969, Spearman = 0.8010, Pearson = 0.8023.

Datasets for HSKAgent include 16K positive/negative grammar instances and 10K human-annotated L2 essays with error tags and scores.

## 5. Experimental Results and Modeling Human-Like SLA Trajectories

Experimental results show that HSKBenchmark-trained LLMs not only model but emulate human-like second language acquisition patterns:

| Model / Level        | $C_g(3)$ | Err  | MATTR | MDD   | Score  |
|----------------------|----------|------|-------|-------|--------|
| Native speakers      | 0.341    | 1.40 | 0.806 | 2.977 | 88.33  |
| HSK learners (95*)   | 0.356    | 2.87 | 0.817 | 2.839 | 85.00  |
| GPT-4.1-mini         | 0.398    | 0.00 | 0.829 | 2.603 | 91.50  |
| LLaMA2 (base)        | 0.484    | 0.90 | 0.686 | 2.425 | 70.00  |
| LLaMA2_HSK3          | 0.492↑   |0.63↓ |0.719↑ |2.505↑ |75.83↑  |
| LLaMA2_HSK6 (final)  | 0.425    |0.53↓ |0.764↑ |2.556↑ |81.83↑  |

Key observations include:

- Consistent monotonic improvement in grammar coverage, lexical, and syntactic complexity with curriculum progression.
- Decreased error rates following each instruction-tuning phase.
- Emergent usage and generalization behaviors at advanced levels.
- Ablation with shuffled data yields inferior late-stage (HSK 5–6) performance versus strict curriculum, empirically supporting the curriculum-based “i + 1” paradigm in LLMs.
- Curriculum-tuned LLMs maintain or slightly improve general Chinese and English benchmark scores, with no catastrophic forgetting.

## 6. Impact and Significance

HSKBenchmark establishes a systematic research platform for both dynamic modeling and assessment of Chinese SLA in LLMs. Its modular and phased design enables controlled evaluation of acquisition-phase sensitivity, emergent linguistic behaviors, and cross-task transfer in neural language models. The result is a benchmark suite that delivers both fine-grained, linguistically motivated metrics and reproducible assessment, facilitating future work in computational linguistics, interpretability, and educational technology for Chinese as a second language [2511.15574].

Source: https://www.emergentmind.com/topics/hskbenchmark