Index-1.9B: Open 1.9B Model Series

• Index-1.9B is a series of 1.9 billion parameter small language models featuring four variants for base, control, dialogue, and role-playing tasks.
• The models use a deep 36-layer decoder-only Transformer with Norm-Head, SwiGLU activation, and optimized multilingual tokenization to enhance training stability.
• It integrates controlled pre-training, supervised fine-tuning, direct preference optimization, and retrieval-augmented generation, achieving competitive scores on benchmarks like MMLU and HellaSwag.

Searching arXiv for the Index-1.9B technical report to ground the article in the cited paper. Index-1.9B is a series of open small LLMs developed at Bilibili. The series comprises four models: Index-1.9B-Base, a foundation model with 1.9 billion non-embedding parameters pre-trained on 2.8 trillion predominantly Chinese and English tokens; Index-1.9B-Pure, a control variant trained with an identical recipe but with all instruction-like data strictly filtered from the corpus; Index-1.9B-Chat, aligned from the base model with supervised fine-tuning and direct preference optimization; and Index-1.9B-Character, which augments the chat model with retrieval-augmented generation for few-shot role-playing customization. Pre-training employs a Warmup-Stable-Decay learning-rate schedule together with a Norm-Head output layer, and the reported study includes controlled analyses of depth, learning-rate magnitude and scheduling, data quality during decay, and instruction-data inclusion during pre-training (Zhang et al., 10 Jul 2026).

1. Model family and release structure

The released family is organized around four principal variants and two additional research-oriented releases. The core distinction is between a foundation model, a control model for instruction-data effects, an aligned dialogue model, and a role-playing extension (Zhang et al., 10 Jul 2026).

Model	Construction	Stated role
Index-1.9B-Base	Pre-trained foundation model	Base bilingual SLM
Index-1.9B-Pure	Same recipe, instruction-like data strictly filtered	Control variant
Index-1.9B-Chat	Base + SFT + DPO	Dialogue model
Index-1.9B-Character	Chat + RAG	Few-shot role-playing

Two further releases are identified in the report. Index-1.9B-Constant-LR is an intermediate checkpoint before learning rate decay, intended for research on training dynamics. Index-1.9B-32K is a long-context extension with 32K tokens (Zhang et al., 10 Jul 2026).

The release policy includes all models, together with evaluation code, and the report also lists a project repository and a Hugging Face checkpoint. This suggests that the family is intended not only as a deployment artifact but also as a controlled experimental platform for studying small-model pre-training, alignment, and benchmark sensitivity (Zhang et al., 10 Jul 2026).

2. Architecture and tokenization

Index-1.9B is a decoder-only Transformer with a Llama-like design. The base architectural specification is: 36 layers, hidden size 2048, FFN size 5888, 16 attention heads, and sequence length 4096. The parameter count is 1.9 billion non-embedding parameters (Zhang et al., 10 Jul 2026).

The report emphasizes a deeper-than-typical architecture under a fixed parameter budget. The 36-layer configuration is presented as a deliberate choice and is tied to an explicit depth-versus-width study. The model also uses SwiGLU activation, RMSNorm normalization, rotary embeddings, and a Norm-Head output layer that applies weight normalization to the output projection and is reported to stabilize training under large learning rates (Zhang et al., 10 Jul 2026).

The tokenizer is SentencePiece BPE with 65,029 tokens and is described as optimized for multilingual compression and Chinese handling. Tokenization efficiency is reported through the compression-rate definition

$$\mathrm{Compression~Rate} = \frac{\mathrm{len}(\text{token ids})}{\mathrm{len}(\text{text})}.$$

Lower values are stated to be better, and the tokenizer is reported to achieve strong performance across Chinese, English, Japanese, and Korean (Zhang et al., 10 Jul 2026).

A further implementation detail is that no whitespace is prepended. The report states that this improves Chinese support. Within the overall design, tokenizer construction, architectural depth, and the Norm-Head layer are treated as coupled choices rather than isolated engineering decisions (Zhang et al., 10 Jul 2026).

3. Pre-training corpus and optimization recipe

The pre-training corpus contains 2.8 trillion tokens and is predominantly Chinese and English, with a Chinese:English ratio of approximately 4:5. The corpus also includes 6% code and approximately 10% curated data such as books, encyclopedias, and STEM materials (Zhang et al., 10 Jul 2026).

Data quality control is described in operational terms. The pipeline uses bias-aware filtering through heuristic and classifier-based methods, with annotations bootstrapped from a larger Index-Large model. Deduplication combines global document-level MinHash with exact substring deduplication at scale, extending suffix-array tools. The corpus mix is described as using public sources together with strict filtering and diversity selection (Zhang et al., 10 Jul 2026).

Optimization uses AdamW with $\beta_1=0.9$, $\beta_2=0.95$, $\epsilon=10^{-8}$, gradient clipping of 1.0, and weight decay of 0.1. The batch size is 4 million tokens, precision is bfloat16, and training is reported on 128 Huawei Ascend 910B accelerators for approximately 28 days. The implementation uses full-sequence sample packing with attention masks, position resets at document boundaries, and selective activation checkpointing (Zhang et al., 10 Jul 2026).

The learning-rate policy is the Warmup-Stable-Decay schedule. Warmup lasts 100 steps, the stable phase keeps the learning rate fixed at $5 \times 10^{-4}$, and the decay phase reduces it to $5 \times 10^{-6}$, covering the final 400B tokens. During decay, the concentration of curated data is raised substantially. The report attributes the feasibility of these unusually large learning rates to the Norm-Head output layer (Zhang et al., 10 Jul 2026).

This schedule is not presented merely as a standard annealing heuristic. The report explicitly links the decay phase to data-quality reweighting, making schedule design and corpus composition part of a single pre-training recipe (Zhang et al., 10 Jul 2026).

4. Alignment pipeline and character specialization

The chat model is obtained from the base model through supervised fine-tuning and direct preference optimization. For SFT, approximately 10M instruction-response pairs are filtered to fewer than 100K high-quality examples. Training uses a system-query-response format, a learning rate of $1 \times 10^{-5}$, and optimizer state loaded from pre-training (Zhang et al., 10 Jul 2026).

DPO is then applied to writing, instruction following, and safety. The preference set contains more than 100K pairs, built from reward-model scoring and human or automated labeling. The DPO stage uses a learning rate of $1 \times 10^{-6}$, a cosine schedule, $\beta=0.1$, and one epoch (Zhang et al., 10 Jul 2026).

Index-1.9B-Character extends the chat model with retrieval-augmented generation. The report states that approximately 80K character dialogues are extracted and cleaned, and the retrieval system provides relevant history for persona conditioning. The stated goal is few-shot persona instantiation from user-uploaded corpora (Zhang et al., 10 Jul 2026).

The alignment stack therefore proceeds in three layers: generic pre-training, conversational alignment, and persona-specific augmentation. This suggests a modular interpretation of the family: the base model is a pre-training object, the chat model is an alignment object, and the character model is a retrieval-conditioned specialization (Zhang et al., 10 Jul 2026).

5. Benchmark profile

The evaluation suite spans examination, understanding and reasoning, mathematics, code, role-playing, and an internal prompt set. Named benchmarks include MMLU, C-Eval, CMMLU, HellaSwag, ARC-Challenge, ARC-Easy, GSM8K, HumanEval, CharacterEval, and an internal 300+ prompt set covering instruction, QA, writing, and reasoning (Zhang et al., 10 Jul 2026).

Index-1.9B-Base is reported to attain an average score of 64.92. The same table gives Avg. (en) 69.93, MMLU 52.53, C-Eval 57.01, CMMLU 52.79, HellaSwag 80.69, ARC-C 65.15, and ARC-E 81.35. The report states that this is competitive with or exceeds open models of several times its size (Zhang et al., 10 Jul 2026).

Model	Avg. All	Selected benchmark profile
Index-1.9B	64.92	MMLU 52.53, C-Eval 57.01, HellaSwag 80.69
Index-1.9B-Pure	50.61	MMLU 46.24, C-Eval 46.53, HellaSwag 62.63
Qwen2-1.5B	65.17	MMLU 56.50, C-Eval 70.60
Llama-2-13B	57.51	MMLU 55.78, C-Eval 39.93

The report further states that Index-1.9B-Base outperforms Llama-2-13B and some larger models on aggregate and English-focused benchmarks, while remaining highly competitive with Qwen2-1.5B. On GSM8K and HumanEval, the values 28.2 and 18.9 are reported, and these are described as on par with Llama-2-13B (Zhang et al., 10 Jul 2026).

For aligned models, the in-house evaluation reports an average of 2.295 for Index-1.9B-SFT and 2.318 for Index-1.9B-DPO, with DPO improving writing and open QA. For role-playing, CharacterEval reports 3.109 for Index-1.9B-Character, compared with 3.129 for Baichuan2-13B, 3.034 for InternLM-7B, 2.849 for Qwen-7B, and 2.381 for GPT-3.5. The report describes Index-1.9B-Character as achieving role-playing consistency and attractiveness comparable to much larger models (Zhang et al., 10 Jul 2026).

6. Controlled studies and training dynamics

A central feature of the report is the inclusion of controlled empirical studies rather than only end-point results. The first such study concerns the Norm-Head output layer. Gradient normalization of the output projection is reported as crucial for stability and performance, especially at high learning rates, with consistently higher benchmark scores and faster early training (Zhang et al., 10 Jul 2026).

A second study examines model shape. Deeper and narrower models with 36 layers are reported to consistently outperform equally parametered shallow and wide 9-layer models on downstream benchmarks at fixed size, despite heavier activation memory. This is presented as an argument for depth under a small-model parameter budget (Zhang et al., 10 Jul 2026).

A third study concerns learning-rate magnitude. Higher peak learning rates, specifically $5 \times 10^{-4}$ versus $\beta_1=0.9$0, are reported to yield higher and more stable scores throughout training. A fourth study examines learning-rate schedule and data coupling: the Warmup-Stable-Decay schedule alone does not improve final scores unless paired with concentrated curated data in decay, while the combination of curated data during decay yields the best generalization score (Zhang et al., 10 Jul 2026).

The most methodologically distinctive study concerns instruction data in pre-training. The report states that adding 7% instruction data during the decay phase boosts MMLU by approximately 7 points and similarly raises other benchmark scores. The released ablation values are:

• ablation-pure: MMLU 43.75, CMMLU 42.35, C-Eval 43.61, ARC-C 42.75, ARC-E 61.61, HellaSwag 63.21, GSM8K 12.81, HumanEval 12.20.
• ablation-boost: MMLU 51.21, CMMLU 49.79, C-Eval 52.41, ARC-C 59.57, ARC-E 78.86, HellaSwag 57.80, GSM8K 28.89, HumanEval 18.29 (Zhang et al., 10 Jul 2026).

The report explicitly frames this as benchmark-score inflation caused by instruction data in pre-training and releases both Pure and non-Pure variants for transparent benchmarking. It also documents an unexplained sharp gain in exam benchmark scores during the constant-learning-rate phase, with MMLU and C-Eval rising by about 10 points at roughly 1.0–1.2T tokens even though data and learning rate remain unchanged. This suggests that Index-1.9B is also a study of training dynamics, not only a model release (Zhang et al., 10 Jul 2026).