H2O-Danube-1.8B-Chat: Open-Source Conversational Model

• H2O-Danube-1.8B-Chat is an open-source conversational language model with 1.8B parameters, using a transformer decoder architecture and fine-tuning techniques.
• The model employs efficient pretraining with local sliding-window self-attention and rotary positional embeddings to enhance long-context performance.
• It integrates supervised fine-tuning and direct preference optimization for alignment, achieving competitive results across diverse benchmarks.

H2O-Danube-1.8B-Chat is an open-source conversational LLM developed as part of the H2O-Danube series, distinguished by a 1.8 billion parameter decoder architecture and a training curriculum tailored for high performance in both general and chat-based settings. The model is licensed under Apache 2.0, enabling commercial use and downstream fine-tuning. Its design follows the core architectural and training principles of the Llama 2 and Mistral models, with refinements targeting efficiency and performance at the sub-2B parameter scale (Singer et al., 2024).

1. Architectural Design and Pretraining Protocol

H2O-Danube-1.8B's architecture consists of a transformer decoder stack with 24 layers, each featuring a hidden dimension of 2,560 and a feed-forward (MLP) intermediate dimension of 6,912. The attention mechanism comprises 32 heads (8 key/value grouped-query heads) and employs rotary positional embeddings (RoPE). Input and output embeddings are untied, and all normalization uses RMSNorm, with separate pre- and post-layer normalization. Notably, the model eschews linear-layer biases and supports a vocabulary of 32,000 tokens (Llama 2 tokenizer).

Pretraining employed local (sliding-window) self-attention with a 4,096-token context and allowed up to 16,384 tokens per sequence. The overall pretraining proceeded in two main phases: an initial 1T-token run over heterogeneous public knowledge corpora (Phase I), followed by an incremental 2T-token extension (Phase II) emphasizing higher-quality data and systematic reduction of noisy web content from 84.5% to 55.5%. A token-length curriculum advanced from 2,048 to 16,384 tokens per sequence over several hundred billion tokens, a design intended to match model capacity growth to longer context skills.

The autoregressive pretraining objective minimizes the standard negative log-likelihood of next-token prediction:

$$L_{\mathrm{LM}}(\theta) = - \sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})$$

where $x_{<t} = (x_1, \ldots, x_{t-1})$ and $T$ is the sequence length. Optimization uses AdamW, with mixed precision (FP8 for most layers, bfloat16 for the final head), a cosine decay learning rate (initial $2 \times 10^{-4}$ to final $1 \times 10^{-5}$), weight decay 0.1, and gradient clipping at norm 1.0. Training throughput on 8 × H100 GPUs reaches approximately 292.7 K tokens/sec.

2. Supervised Fine-Tuning via Instruction Datasets

Supervised fine-tuning (SFT) is conducted on approximately 157,000 instruction–response pairs drawn from OpenOrca, MetaMathQA, UltraChat200k, and OASST2 datasets. The procedure trains all model parameters for one epoch with a batch size of 8, a fixed learning rate of $1 \times 10^{-5}$, full 16,384-token context windows, and a cosine decay scheduler. The fine-tuning loss remains the standard autoregressive cross-entropy, but only response tokens are included in the loss—the prompt portion is masked to focus gradient updates on output generation conditioned on instructions.

3. Direct Preference Optimization for Alignment

Direct Preference Optimization (DPO) is employed to enhance alignment to human preferences, following the formulation of Rafailov et al. (2023) but without any explicit reward model or RL policy gradient stage. DPO frames preference learning as a single-stage optimization over ranked response pairs $(\mathbf{x}, y^+, y^-)$:

$$L_{\mathrm{DPO}}(\theta) = -\mathbb{E}_{(\mathbf{x},y^+,y^-)} \left[\, \log \sigma\left(s_\theta(y^+|\mathbf{x}) - s_\theta(y^-|\mathbf{x}) -\beta\right) \right]$$

where $$s_\theta(y|\mathbf{x}) = \log P_\theta(y|\mathbf{x})$$, $\sigma$ is the sigmoid function, and the margin hyperparameter is $x_{<t} = (x_1, \ldots, x_{t-1})$0.

The DPO procedure occurs in two LoRA-adapted stages ($x_{<t} = (x_1, \ldots, x_{t-1})$1): the first over UltraFeedback Binarized, Orca DPO Pairs, and Distilabel Math Preference DPO (batch size 2, learning rate $x_{<t} = (x_1, \ldots, x_{t-1})$2), and the second over 5,000 high–low ranked OASST2-derived English pairs (batch size 2, learning rate $x_{<t} = (x_1, \ldots, x_{t-1})$3).

4. Evaluation, Benchmarks, and Comparative Results

H2O-Danube-1.8B and its chat variant achieve highly competitive performance on a broad suite of benchmarks, consistently surpassing other open models in its parameter class. On 0-shot commonsense and world-knowledge tests (ARC-E, ARC-C, BoolQ, HellaSwag, OBQA, PIQA, TriviaQA, Winograd), the base model reports:

Task	ARC-E	ARC-C	BoolQ	Hella	OBQA	PIQA	Trivia	Wino
Score (%)	62.29	35.84	65.81	68.20	37.60	76.93	38.99	61.96

On the Open LLM Leaderboard (0/5/10-shot, six tasks):

Task	ARC	HS	MMLU	TQA	Wino	GSM8K	Avg.	Avg. ($x_{<t} = (x_1, \ldots, x_{t-1})$4GSM)
Score	39.68	69.75	25.97	33.63	64.17	2.05	39.21	46.64

For chat evaluation, the model attains an MT-Bench (GPT-4–graded multi-turn) Turn 1 average of 6.41 and Turn 2 average of 4.63, with best-in-category performance on 5/7 turn types. Chat-oriented 0-shot metrics (ARC-E 67.51%, ARC-C 39.25%, BoolQ 77.89%, Wino 65.35%) indicate parity or superiority to StableLM-2-Zephyr-Chat, and consistent outperformance relative to Qwen-Chat and TinyLlama-Chat. Open LLM Leaderboard chat scores (average ≈ 43.96%, excl. GSM8K) align with these findings.

H2O-Danube-1.8B outperforms all other sub-2B open models on the aggregate leaderboard, exceeding Qwen-1.8B despite 2.2× less pretraining data and trailing StableLM 2 on some tasks, though the latter utilized four times more pretraining tokens and does not offer Apache 2.0 licensing.

5. Inference Efficiency and Deployment

The architectural choices—local sliding-window self-attention (4,096-token blocks), grouped-query attention, and parameter-efficient LoRA adapters for DPO—contribute to low FLOPs and reduced memory bandwidth demands. The model supports inference contexts of up to 16,384 tokens, can execute interactively on a single A100 or H100 GPU, and implements both beam and greedy decoding. Practical implications include deployment flexibility for applications constrained by memory- or bandwidth-limited environments.

6. Access, Licensing, and Community Impact

H2O-Danube-1.8B-Chat and related variants are released under the open Apache 2.0 license, allowing unrestricted commercial use and downstream model development. All versions—base, SFT, and DPO/chat—are available with model cards at https://huggingface.co/h2oai/h2o-danube-1.8b-base and https://huggingface.co/h2oai/h2o-danube-1.8b-chat. The open licensing and efficient architecture are positioned to democratize access to high-performing conversational LLMs for both research and production contexts (Singer et al., 2024).