ChatGLM-6B: Open Bilingual Dialogue Model

• ChatGLM-6B is an open-source bilingual dialogue model built on a 6.2B parameter decoder-only Transformer using techniques like RoPE and GLU for efficient context encoding.
• It was pre-trained on 1 trillion tokens across Chinese, English, and 24 other languages with rigorous deduplication and an autoregressive blank-infilling objective.
• Post-training alignment via supervised fine-tuning and early RLHF, along with successful adaptation for multimodal applications, highlights its role as a fast-iteration research platform.

ChatGLM-6B is an open-source bilingual LLM developed within the GLM (General LLM) family, notable as the first open bilingual GLM dialogue model. It is positioned as a 6.2 billion-parameter, decoder-only Transformer optimized for dialog-based tasks and specifically targeted toward both Chinese and English, with a modest representation of additional languages. ChatGLM-6B has served as a core platform for open research on fast-iteration pre-training, efficient alignment methodology, and downstream adaptation, ultimately enabling the rapid advancement and deployment of subsequent, more capable GLM generations (GLM et al., 2024).

1. Model Architecture and Design

ChatGLM-6B utilizes a standard decoder-only Transformer architecture, directly inheriting several design elements from the larger GLM-130B model. It comprises 6.2 billion parameters and supports a context window of 2,048 tokens. The model is trained using the GLM autoregressive blank-infilling objective, in which randomly masked spans must be generated in left-to-right order. Its architecture incorporates:

• Rotary positional embeddings (RoPE) in all attention layers to encode sequence ordering.
• Gated Linear Units (GLU) with GeLU activations in each feed-forward sub-layer for improved expressivity and stabilization.
• DeepNorm normalization strategy replacing standard LayerNorm, yielding deeper practical model depth before divergence.
• Scaled dot-product self-attention:

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left( \frac{Q K^{\top}}{\sqrt{d_k}} \right) V$$

No specific public details on layer count or attention-head number are provided, but ChatGLM-6B scales along the same empirical scaling curve as other GLM models (GLM et al., 2024).

2. Pre-Training Procedure

ChatGLM-6B was pre-trained on a dataset of approximately 1 trillion tokens, with heavy emphasis on Chinese and English, alongside limited representation for 24 other languages. Pre-training data sources included web pages, Wikipedia, books, scientific papers, code, and other textual corpora. The data processing pipeline featured:

• Exact and fuzzy deduplication to minimize redundancy (Broder 1997).
• Rigorous quality filtering to remove low-quality samples, placeholder text, code fragments, or offensive content.
• Multi-stage tokenization: Separate byte-level BPE on Chinese and a multilingual set, with post-hoc vocabulary merging into a final 150,000-token set using OpenAI's cl100k_base vocabulary (tiktoken).
• Source re-weighting to amplify the impact of high-quality text (notably books and Wikipedia).
• The pre-training objective was the GLM autoregressive blank-infilling loss:

$$L_{\mathrm{AR}} = -\sum_{t \in \mathrm{masked\ tokens}} \log P( x_t \mid x_{<t}, x_{\backslash \mathrm{masked}} )$$

This training paradigm enforces context dependence and aligns representation with span-level, rather than token-level, generation (GLM et al., 2024).

3. Alignment and Post-Training

ChatGLM-6B alignment is achieved through a two-stage post-training procedure:

• Supervised Fine-Tuning (SFT) using human-authored prompt–response pairs focused on instruction following and safe outputs. Human annotators rate responses for safety, factuality, relevance, and overall helpfulness, and these ratings guide data selection in SFT.
• Early-stage Reinforcement Learning from Human Feedback (RLHF), using approaches in the PPO and DPO regime (referencing Hou et al. 2024), though neither RLHF weights nor comprehensive recipes were disclosed for the initial public release. The RLHF loss used:

$$L_{\mathrm{RL}} = -\mathbb{E}_{\pi_\theta}[ r(y) ] + \beta \cdot KL[ \pi_\theta(\cdot \mid x) \Vert \pi_{\mathrm{ref}}(\cdot \mid x) ]$$

No hyperparameter details for RLHF are reported for the initial ChatGLM-6B; later generations increased RLHF scale and fine-tuned $\beta$ between 0.01–0.1 (GLM et al., 2024).

4. Evaluation and Benchmark Performance

ChatGLM-6B was evaluated on an array of standard English and Chinese NLP benchmarks, with all results reported in Table 1 of the source. The following tables summarize its performance:

English Benchmarks (zero-shot or Chain-of-Thought):

Dataset	Score (%)
GSM8K (CoT)	1.5
MATH (CoT)	3.1
BBH (CoT)	0.0
MMLU	25.2
HumanEval	0.0
BoolQ	51.8
CommonSenseQA	20.5
HellaSwag	30.4
PIQA	65.7
DROP	3.9

Chinese Benchmarks:

Dataset	Score (%)
C-Eval	23.7
CMMLU	25.3
GAOKAO-Bench	26.8
C³	35.1

Comparative results indicate substantial improvements in successor models ChatGLM2-6B and ChatGLM3-6B (e.g., GSM8K from 1.5% in ChatGLM-6B to 72.3% in ChatGLM3-6B), highlighting ChatGLM-6B's primary role as a research platform rather than a state-of-the-art solution. For reference, external baselines (GPT-3.5, GPT-4) reported much higher scores on comparable metrics as of 2023 (GLM et al., 2024).

5. Applications and Downstream Adaptation

ChatGLM-6B has been adopted as a generic bilingual language backbone for multimodal and domain-specialized applications. Notably, in the context of medical report generation, ChatGLM-6B was employed as the decoder in an encoder–decoder framework for medical caption (report) prediction (Yang et al., 2023). The approach connected a vision transformer (EVA-ViT-g), a lightweight Query Transformer (Q-Former), and ChatGLM-6B. Key points include:

• ChatGLM-6B receives visual embeddings as prefix tokens, enabling visual-textual conditional generation.
• Parameter-efficient adaptation is achieved via P-tuning v2: inject four learned continuous soft tokens per self-attention layer, resulting in only 0.9M additional parameters while freezing all original LLM weights.
• On the ImageCLEF 2023 Caption Prediction Task, the best configuration (with fully trainable vision encoder and P-tuning ChatGLM-6B) achieved 0.61484 BERTScore (rank 4/13) and 0.25328 ROUGE-1 (rank 2/13).
• Ablation studies confirm that both Q-Former adaptation and P-tuning of ChatGLM-6B are critical for transfer to biomedical writing conventions, with P-tuning alone yielding +0.9pp ROUGE-1 improvement.

Remaining challenges observed include occasional hallucination, catastrophic forgetting of zero-shot QA abilities, and code-switching between languages (Yang et al., 2023).

6. Scalability, Limitations, and Open Source Impact

ChatGLM-6B was explicitly positioned as a "fast-iteration" research vehicle:

• At 6.2B parameters, ChatGLM-6B can be fully quantized to INT4 and run on a single consumer-grade GPU, facilitating accessible experimentation and deployment.
• The model lacks native support for function calling or tool use, and its context window is limited to 2,048 tokens—restrictions that were addressed in subsequent GLM generations.
• Academic benchmark performance is modest: math/reasoning skill limited (e.g., GSM8K 1.5%, MMLU 25.2%), and instruction following is below the level of contemporary closed-source LLMs.
• Model weights and inference code are available under open-source license with support for quantized (INT4, INT8) inference, extensive downloads via Hugging Face (>10 million in 2023), and API access via BigModel.cn (GLM et al., 2024).

7. Legacy and Role within the GLM Model Family

ChatGLM-6B marks the first open bilingual GLM dialogue model and provided the empirical basis for innovations later incorporated in the GLM-4 series, including FlashAttention, long-context extension, advanced RLHF, and tool-calling capabilities. It served as the primary testbed for pre-training and alignment recipes key to the performance scaling observed in subsequent GLM models. A plausible implication is that the open, accessible nature of ChatGLM-6B enabled accelerated iterations in pre-training methodology, alignment, and efficient deployment protocols across the GLM ecosystem (GLM et al., 2024).