Index SLM Technical Report

Abstract: We present Index-1.9B, a series of open small LLMs developed at Bilibili. The series comprises four models: Index-1.9B-Base, a foundation model with 1.9 billion non-embedding parameters pre-trained on 2.8 trillion predominantly Chinese and English tokens; Index-1.9B-Pure, a control variant trained with an identical recipe but with all instruction-like data strictly filtered from the corpus; Index-1.9B-Chat, aligned from the base model with supervised fine-tuning and direct preference optimization; and Index-1.9B-Character, which augments the chat model with retrieval-augmented generation for few-shot role-playing customization. Pre-training employs a Warmup-Stable-Decay learning-rate schedule in which the concentration of curated data is raised substantially during the decay phase, together with a Norm-Head output layer that stabilizes training under large learning rates. On a suite of standard benchmarks covering examination, reasoning, mathematics, and code, Index-1.9B-Base attains an average score of 64.92, competitive with or exceeding open models of several times its size. We further report controlled studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training, and we document an unexplained surge in benchmark performance midway through the constant-learning-rate phase. All models, together with evaluation code, are released at https://github.com/bilibili/Index-1.9B.

• The paper introduces Index-1.9B, a 1.9B-parameter SLM leveraging a Warmup–Stable–Decay schedule and Norm-Head for stable, high-rate training.
• It demonstrates strong multilingual performance and benchmark results, rivaling larger models with advanced dialogue and role-play capabilities.
• Ablation studies reveal a 7-point boost from instruction data in the pre-training decay phase, highlighting effective data curation strategies.

Index-1.9B: Design, Training, and Analysis of an Open 1.9B-Parameter SLM

Model Suite and Pre-training Strategy

The Index SLM Technical Report (2607.09885) details the construction, experimental rationale, and evaluation of the Index-1.9B small LLM (SLM) suite. This series encompasses four key variants:

• Index-1.9B-Base: a 1.9B non-embedding parameter foundation model, pre-trained on an unprecedented 2.8T tokens over predominantly Chinese and English text;
• Index-1.9B-Pure: a corpus-control ablation with strict exclusion of instruction-like data during pre-training;
• Index-1.9B-Chat: a dialogue-aligned model via sequential SFT and DPO;
• Index-1.9B-Character: a retrieval-augmented, role-play model for persona instantiation.

Pre-training employs a Warmup–Stable–Decay (WSD) schedule leveraging the model's ability to tolerate large learning rates via the Norm-Head output projection, allowing the decay phase to be both longer and coincide with a substantial increase in curated and instruction data proportion. This yields a model that is highly efficient—competitive with much larger models in multilingual and code/mathematical benchmarks.

The training corpus is characterized by a 4:5 Chinese-English ratio, rigorous multi-granularity deduplication, and targeted data curation. The vocabulary (65,029 tokens) is tightly matched to model size constraints and multilingual requirements, using distinct approaches for Chinese segmentation and avoiding Llama-style whitespace pre-padding.

Figure 1: Composition of the Index-1.9B pre-training corpus.

Model Architecture and Training Dynamics

Architecturally, Index-1.9B adheres to a decoder-only transformer paradigm influenced by Llama, with pivotal modifications:

• Depth: 36 layers, motivated by ablation studies showing consistent downstream metric gains over shallower, wider configurations at equal parameter/FLOP count.
• Norm-Head: A layer-wise normalization applied to the LM output head. This yields more stable gradient distributions, mitigates rare-token oscillation, and supports substantially elevated peak learning rates.

A controlled experiment evidences that the Norm-Head yields consistent uplifts throughout training on benchmark performance.

Figure 2: Average benchmark score of a 1B model trained on 1T tokens, with and without Norm-Head.

In fixed-FLOP, fixed-parameter count studies, deeper configurations outperform wider ones, although the trade-off between residual activation memory and total parameter allocation is noted as a practical limitation.

Figure 3: Deep (36-layer) versus wide-and-shallow (9-layer) models at equal parameter count (1.01B non-embedding).

Increasing the peak learning rate (validated stable up to $5\times10^{-4}$ with Norm-Head) confers a significant and persistent advantage, counter to some conventions from large-model scaling law literature.

Figure 4: Effect of peak learning rate on a 1B model trained on 1T tokens (cosine schedule).

Learning Rate Scheduling and Data Curation

Comparison of LR schedules (cosine, linear, WSD) finds little difference at convergence but a marked interaction when data curation is varied. The greatest benefit is realized only when late-stage curated data is coupled with a decaying learning rate, empirically validating recent SLM strategies.

Figure 5: The three learning-rate schedules compared in Section~\ref{sec:lr-schedule.

Figure 6: Interaction between learning-rate schedule and late-stage data curation. ``+ curated data'' raises the proportion of curated data over the final 10\% of training.

A transparent ablation using matched checkpoints and a single controlled intervention quantifies the effect of instruction data injection during pre-training decay. This practice alone can generate a 7-point average inflation on MMLU and related benchmarks, a magnitude sufficient to reorder leaderboard standings for this class.

Figure 7: MMLU across pre-training. The stable phase (constant learning rate, natural text) is followed by two 50K-step decay branches from the same checkpoint: natural text only, and natural text plus 7\% instruction data. Scores are measured on intermediate checkpoints. The unexplained surge at 1.0--1.2T tokens is discussed in Section~\ref{sec:surge.

Notably, the authors observe an unexplained performance surge mid-stable-phase—an abrupt increase in metrics at approximately 1–1.2T tokens before any explicit decay. The cause remains open, although it suggests new dynamics in high-quality, high-rate, long-horizon SLM training.

Alignment and Few-shot Personalization

Dialogue alignment proceeds via SFT (with targeted clustering, reward-model filtering, and data supplementation) and DPO. DPO is shown to particularly enhance open-ended writing and generative quality, while providing modest improvements in instruction-following and safety. For safety preference data, preference pairing using model-generated refusals is superior to using human-written refusals, which otherwise raise perplexity and yield undesirable side-effects in refusal calibration.

For few-shot role-play, retrieval-augmented generation enables dynamic persona instantiation, with high-character-consistency and attractiveness as evaluated on CharacterEval—Index-1.9B-Character outperforms all comparably-sized open models and approaches larger closed systems.

Figure 8: A conversation with Index-1.9B-Chat (web demo). The model answers an open-ended question about gaming culture with a structured, grounded response.

Figure 9: The few-shot role-playing interface of Index-1.9B-Character. Uploading a dialogue corpus and a role description instantiates a new persona.

Evaluation and Empirical Results

On a comprehensive suite of benchmarks (MMLU, HellaSwag, ARC, C-Eval, CMMLU), Index-1.9B achieves a 64.92 average with especially strong showing on English and Chinese academic tasks, rivaling or surpassing models up to Llama-2-13B in size. The impact of instruction-containing decay on aggregate scores and benchmark methodology is explicitly isolated. For mathematics and code (GSM8K, HumanEval), results are comparable to Llama-2-13B, indicating remaining headroom, particularly via domain-targeted data or specialized architectures.

In role-playing (CharacterEval), Index-1.9B-Character achieves top-tier scores in overall, character-consistency, and attractiveness, validating the effectiveness of its retrieval-augmented fine-tuning in a multi-character, dialogue-rich context.

Implications and Future Directions

The results demonstrate that careful integration of curated and instruction data, schedule–data coupling, explicit architectural adaptations (Norm-Head, increased depth), and systematic control experiments meaningfully increase the practical value of SLMs. The magnitude of benchmark inflation from instruction data inclusion underscores the necessity for transparent reporting and controlled ablation in model comparison.

Future work is indicated in several areas:

• Further resolution of stable-phase performance surges and their relation to data distribution and optimization schedules.
• Domain-specialized variants targeting mathematical reasoning, robust coding, or multi-modal integration, building upon the baseline demonstrated in this report.
• Investigation of efficient retrieval mechanisms or longer-context adaptation to support more data-efficient personalization and hybrid architectures.

Conclusion

Index-1.9B establishes new standards in open 2B-parameter SLMs, achieving strong multilingual, general-purpose, and dialogue capability with compute- and memory-efficient techniques. Through rigorous ablation, transparent methodology reporting, and targeted architectural innovations, the Index series advances the tractable deployment of SLMs and sets a clear empirical foundation for subsequent SLM research. The open release of models, checkpoints, and evaluation pipelines enables further systematic study on training dynamics, data attribution, and model scaling within the SLM regime.