H2O-Danube-1.8B: Open Sub-2B Transformer

• The model achieves state-of-the-art performance among sub-2B LLMs by leveraging innovations like grouped-query attention and FP8 precision.
• H2O-Danube-1.8B is a 24-layer, decoder-only Transformer that integrates key architectural elements from Llama 2 and Mistral-inspired methods.
• Its training on up to 3 trillion tokens and open Apache 2.0 licensing support scalable research and practical applications across language tasks.

H2O-Danube-1.8B is an open, small-scale LLM comprising approximately 1.8 billion parameters, designed using core principles from Llama 2 and Mistral architectures. Trained initially on 1 trillion tokens of high-quality, code-free web and textual data, H2O-Danube-1.8B and its subsequent variant H2O-Danube2-1.8B, which received an additional 2 trillion tokens, demonstrate leading performance among sub-2B parameter LLMs across major benchmarks. All model checkpoints, including supervised fine-tuned (SFT) and chat-tuned (DPO) variants, are released under the Apache 2.0 license, supporting economically accessible research and deployment (Singer et al., 2024).

1. Model Architecture

H2O-Danube-1.8B adopts a 24-layer decoder-only Transformer, largely implementing Llama 2’s architectural blueprint with significant augmentation by several Mistral-inspired techniques.

Key architectural features:

• Hidden size ($d_\text{model}$): 2,560
• Number of layers: 24
• MLP intermediate size ($d_{ff}$): 6,912
• Attention heads: 32 (with Grouped-Query Attention (GQA): 8 key/value heads to optimize memory bandwidth)
• Maximum context window: 16,384 tokens at inference; sliding-window (chunked) attention of 4,096 tokens during pre-training
• Positional encoding: Rotary positional embeddings (RoPE) applied in every attention layer
• Normalization: RMSNorm for both pre- and post-normalization
• Special architectural decisions: no linear-layer biases; word embeddings and $\text{lm}_\text{head}$ are untied
• Precision: 8-bit floating-point (FP8) for Transformer linear layers (GQA & MLP), with $\text{lm}_\text{head}$ in bfloat16

Component	Parameter Count	Description
Embedding/Output	$\sim$163.8M total	$32\mathrm{K}\times2,560$ input, $2,560\times32\mathrm{K}$ output
Transformer	$\sim$1,488M total	24$\times$(QKV/O, MLP, Norm)
Other	$\sim$150M	Norms and ancillary parameters
Total	$d_{ff}$01.8B

Architectural adaptations include sliding-window (chunked) self-attention with FlashAttention 2 and grouped-query attention (GQA), reducing key/value memory bandwidth while maintaining inference efficiency. Rotary position encodings, as well as the use of FP8 arithmetic for memory and throughput optimizations, are present throughout, except for the language modeling head.

2. Training Data and Tokenization

H2O-Danube-1.8B was initially pre-trained on 1T tokens, curated exclusively from text sources including filtered large-scale web crawls, encyclopedias, public-domain books, and news, with explicit exclusion of code data. Extensive preprocessing involved standard de-duplication, language filtering, HTML stripping, and advanced quality filtering with heuristics and compact classifier models (e.g., GBM/BERT).

Tokenization for H2O-Danube-1.8B: Original Llama 2 tokenizer (SentencePiece/BPE) with a 32,000-token vocabulary.

For H2O-Danube2-1.8B, an additional 2T tokens were used, split into three data-mix stages of escalating high-quality (Wikipedia, academic, instructional) content proportions as follows:

Stage	Tokens	Web Proportion	High-Quality Proportion
1	1T	84.5%	15.5%
2	0.95T	72.8%	27.2%
3	0.05T	55.5%	44.5%

The tokenizer for Danube2 was updated to Mistral’s (remapping shared BPE tokens, randomizing new tokens), maintaining a similar vocabulary magnitude.

3. Training Procedure

Pre-training utilized a single node with 8 NVIDIA H100 GPUs, leveraging model replication via PyTorch Distributed Data Parallel (DDP). The curriculum for sequence length was:

1. 700B tokens, max sequence length 2,048
2. 100B tokens, max sequence length 4,096
3. 100B tokens, max sequence length 8,192
4. 100B tokens, max sequence length 16,384

Batch size was approximately 1.18M tokens (summed across all GPUs). Optimization used AdamW with $d_{ff}$1, $d_{ff}$2, weight decay $d_{ff}$3, and gradient clipping at 1.0. The learning rate followed a cosine decay, linearly warming up over approximately 2.36B tokens to a peak of $d_{ff}$4 and decaying to $d_{ff}$5. FP8 was the main precision, with bfloat16 restricted to the output head. Estimated throughput was 292,700 tokens/s, translating to ~950 hours (39.5 days, ≈7,600 GPU-hours) per 1T tokens processed.

The pre-training objective was standard next-token cross-entropy: $d_{ff}$6 Final validation loss reached ≈2.9 ($d_{ff}$7 perplexity), with consistent loss curve declines across training and validation.

4. Evaluation and Benchmarks

H2O-Danube-1.8B exhibits competitive metrics for models below 2B parameters on a broad range of commonsense reasoning and QA tasks. Selected 0-shot benchmarks are as follows:

Task	H2O-Danube-1.8B (%)	Qwen 1.8B (%)
ARC-Easy	62.29	58.25
ARC-Challenge	35.84	34.98
BoolQ	65.81	67.13
HellaSwag	68.20	58.82
OpenBookQA	37.60	33.40
PIQA	76.93	72.85
TriviaQA (5s)	38.99 EM	23.92
Winogrande	61.96	58.96

Aggregated on the Open LLM Leaderboard for sub-2B models (across ARC, HellaSwag, MMLU, TriviaQA, Winogrande, GSM8k), H2O-Danube-1.8B attained an average (including GSM8k) of 39.21% (excluding GSM8k: 46.64%), securing the highest overall ranking at release for models in this size class.

5. Instruction Tuning, Preference Optimization, and Chat Models

5.1 Supervised Fine-Tuning (SFT)

SFT used 157,000 instruction/response pairs, spanning datasets including OpenOrca, MetaMathQA, UltraChat200k, and Oasst2. SFT fine-tuning was performed for a single epoch (batch size 8, learning rate $d_{ff}$8, cosine decay with short warm-up, full 16,384-token context, prompt tokens excluded from the loss).

5.2 Direct Preference Optimization (DPO)

Chat variants subsequently underwent DPO using UltraFeedback (binarized), Orca DPO pairs, Distilabel Math preferences, and a final stage with $d_{ff}$9K Oasst2-derived multi-turn English preference pairs. The DPO phase employed LoRA adapters (rank $\text{lm}_\text{head}$0, $\text{lm}_\text{head}$1), training for one epoch per DPO phase at batch sizes of 2 and learning rates $\text{lm}_\text{head}$2 (then $\text{lm}_\text{head}$3 for the Oasst2 stage). The DPO loss follows the formulation of Rafailov et al. (2023): $\text{lm}_\text{head}$4 with $\text{lm}_\text{head}$5 being the log-likelihood for candidate $\text{lm}_\text{head}$6 given prompt $\text{lm}_\text{head}$7, and $\text{lm}_\text{head}$8.

5.3 Chat Performance

Evaluation of the chat model (excluding coding tasks) using MT-Bench (GPT-4 judged) yielded a turn 1 average score of 6.41 (on par with StableLM-2; Qwen-Chat: 5.66; TinyLlama-Chat: 5.11). Commonsense/QA results for the chat-based variant were: ARC-Easy, 67.51%; ARC-C, 39.25%; BoolQ, 77.89%; HellaSwag, 67.60%; OpenBookQA, 39.20%; PIQA, 76.71%; TriviaQA, 36.29% EM; Winogrande, 65.35%. On the Open LLM Leaderboard (chat), averages reached 43.96% across six tasks.

6. Licensing and Model Availability

All versions—base, SFT-intermediate, and final DPO (chat)—are released under the permissive Apache 2.0 license. Model checkpoints are hosted on Hugging Face:

• Base: h2o-danube-1.8b-base
• SFT: h2o-danube-1.8b-sft
• Chat: h2o-danube-1.8b-chat

The permissiveness of the Apache 2.0 license facilitates broad downstream use and further research, consistent with the model’s stated goal to democratize LLM capabilities at small scale (Singer et al., 2024).

7. Context, Significance, and Related Work

H2O-Danube-1.8B is positioned at the intersection of efficiency-focused sub-2B LLM research and the pursuit of open science. By drawing on Llama 2 and Mistral advancements, including GQA, FP8 arithmetic, and scalable attention via sliding windows, the platform achieves state-of-the-art evaluation metrics for its parameter class. The progressive high-quality content blending scheme and the inclusion of supervised and preference-based chat tuning align the model closely with contemporary research priorities in scalable, instruction-following LLMs. Its open availability under a permissive license underscores a trend towards accessible, reproducible research in language modeling (Singer et al., 2024).