GEB-1.3B: Lightweight Bilingual LLM

• GEB-1.3B is a lightweight bilingual large language model with 1.3B parameters, optimized for efficient CPU inference and multilingual tasks.
• It integrates rotary positional embedding, group-query-attention, and FlashAttention-2 to balance computational efficiency with advanced transformer capabilities.
• Its fine-tuning via supervised instruction tuning and direct preference optimization results in robust performance, low toxicity, and competitive benchmark scores.

GEB-1.3B is an open-source, lightweight bilingual LLM comprising 1.3 billion parameters, designed to maximize inference and training efficiency, particularly on CPU platforms, while maintaining competitive performance on multilingual academic and general-purpose tasks. Its architecture integrates rotary positional embedding (RoPE), group-query-attention (GQA), and FlashAttention-2, and is trained on a large-scale, deduplicated bilingual corpus. The alignment and fine-tuning pipeline includes both supervised instruction tuning and direct preference optimization (DPO), tailoring the model for low toxicity and robust instruction-following. GEB-1.3B outperforms previous lightweight open LLMs in multiple benchmark settings and achieves efficient inference on consumer-grade hardware (Wu et al., 2024).

1. Model Architecture and Transformer Design

GEB-1.3B is a decoder-only transformer architecture with 24 layers, a hidden size of 2048, a feedforward network (FFN) inner dimension of 5632 (approximately 8/3 × dmodel), and 16 attention heads per layer. The model accepts context windows of up to 4096 tokens and utilizes a vocabulary of 64,896 tokens. A distinguishing feature is the use of group-query-attention with 4 key-value (KV) groups, enabling computational and memory savings compared to standard multi-head attention by sharing KV projections among multiple query heads.

Layers	Hidden Size	FFN Size	Heads	KV Groups	Context
24	2048	5632	16	4	4096

Rotary positional embedding (RoPE) is employed to inject continuous, relative position information directly into the query (Q) and key (K) matrices, improving extrapolation and long-range dependency modeling. Formally, RoPE operates by partitioning the per-head dimension into pairs and rotating them using a set of angular frequencies, enabling sequence position-dependent transformations prior to the attention operation.

2. Training Data and Pre-processing

The model is pre-trained on a 550 billion token corpus, comprising roughly 2 PB of raw text prior to filtering. The final working dataset after extensive rule-based cleansing, perplexity-based filtering, keyword-density control, and deduplication totals approximately 1.3 TB of text. The language distribution is 33.1% English and 52.2% Chinese, with the remainder comprising code and other content. English sources include C4, GitHub, and StackExchange, while Chinese content is sourced from curated Common Crawl, WuDaoCorpus, and SkyPile.

Preprocessing emphasizes domain and syntactic balance to facilitate bilingual modeling. Aggressive deduplication and PPL-based filtering increase data quality and reduce memorization risk. This diversified and massive pre-training corpus enables cross-lingual generalization and robust downstream performance.

3. Efficiency-Enhancing Attention Mechanisms

GEB-1.3B incorporates two key architectural optimizations for efficiency:

• Group-Query-Attention (GQA): Instead of computing separate K and V projections for every attention head (H), GQA divides H heads into G groups, reusing the same K and V for all heads within each group while maintaining separate query projections. This reduces the number of K and V computations by a factor of H/G. For GEB-1.3B, H = 16 and G = 4, so each KV group is shared by 4 heads, substantially reducing both compute and memory requirements while incurring minimal accuracy loss relative to MHA.
• FlashAttention-2: Attention computation is fused in a single GPU kernel that processes Q, K, and V blocks (~B×B) in on-chip memory, never materializing the full attention matrix in GPU RAM. This design reduces memory footprint to O(N·dmodel + B²) and minimizes DRAM traffic, facilitating training with long context windows and large batch sizes.

These optimizations balance architectural expressiveness with pragmatic resource constraints, enabling deployment on both GPU and CPU hardware without extensive infrastructure.

4. Fine-Tuning and Alignment

The fine-tuning pipeline for GEB-1.3B comprises two stages:

• Supervised Fine-Tuning (SFT): The model is further tuned on approximately 16 million instruction–answer pairs covering general knowledge, programming, mathematics, and safety. The SFT objective is standard token-wise cross-entropy to encourage alignment with curated human demonstrations.
• Direct Preference Optimization (DPO): Around 10,000 human-rated response pairs (good vs. bad) are used to optimize with a DPO loss function:

$$\mathcal{L}_\text{DPO}(\theta) = -\mathbb{E}_{(x, y_+, y_-)} [\log \sigma( r_\theta(x, y_+) - r_\theta(x, y_-) )]$$

where $r_\theta$ denotes the model log-probability. This explicitly aligns the model’s generations with human preferences and induces marked improvements in safety and instruction-following.

Stabilization techniques include batch replacement and gradient shrinkage on shallow layers, as well as learning rate adjustment matched to batch sizes, optimizing for both convergence and stability during large-batch training.

5. Performance and Benchmarking

GEB-1.3B demonstrates state-of-the-art performance within its parameter class across multilingual and general benchmarks. Zero-shot evaluation scores are as follows:

Model	MMLU (%)	C-Eval (%)	CMMLU (%)	Avg (%)
GEB-1.3B	31.20	33.30	32.20	32.23
MindLLM-1.3B	26.20	26.10	25.33	25.88
TinyLLaMA-1.1B	25.34	25.02	24.03	24.80

GEB-1.3B improves on MindLLM-1.3B by +5.0 (MMLU), +7.2 (C-Eval), +6.9 (CMMLU), and even surpasses LLaMA-7B in Chinese evaluation. On ToxiGen toxicity, GEB-1.3B scores 3.00, substantially below Falcon-7B (7.89), MPT-instruct (16.33), and Llama2-7B (21.25), indicating markedly reduced generation of toxic utterances relative to model size (Wu et al., 2024).

6. Inference, Quantization, and Deployment

GEB-1.3B achieves 12 tokens/second throughput on a single 12-core x86 CPU (2.3 GHz, FP32) and latency of ~80 ms/1k tokens, lowering barriers to deployment on commodity and edge hardware. Ongoing quantization efforts (to int8, int4, and quantization-aware training) are anticipated to yield 4×–8× model size reduction and 2×–4× throughput improvement, with the goal of real-time inference on laptops and mobile devices. The architecture is thus positioned for ubiquitous deployment scenarios, including IoT and local privacy-preserving applications.

7. Open Source Availability and Research Directions

GEB-1.3B is publicly released under an open-research license at https://huggingface.co/GEB-AGI/geb-1.3b. Recommended research avenues include:

• Fine-grained quantization and pruning for <1 GB model footprints.
• Domain adaptation using adapters or LoRA for specialized applications (e.g., legal, medical).
• Knowledge distillation methodologies for further model compression.
• Architectural ablation studies on alternative activations (such as SwiGELU) and sparse attention mechanisms.

These directions underscore its utility as a research substrate for both efficient LLM deployment and continued innovation in lightweight multilingual modeling (Wu et al., 2024).