SmolTulu-1.7b-Instruct: Optimized Small LLM

• SmolTulu-1.7b-Instruct is an instruction-tuned small language model built from SmolLM2-1.7B using specialized post-training alignment techniques.
• It employs a two-stage pipeline with Supervised Finetuning and Direct Preference Optimization, optimizing the learning-rate-to-batch-size ratio for reasoning and pattern recognition.
• Empirical results show state-of-the-art gains on benchmarks like GSM8K and IFEval, highlighting its balanced performance across diverse tasks.

SmolTulu-1.7b-Instruct, also referenced as SmolTulu-DPO-1130, is an instruction-tuned LLM that adapts AllenAI’s Tulu 3 post-training pipeline to Huggingface’s SmolLM2-1.7B base model. Its design leverages empirical findings on the critical role of the learning-rate-to-batch-size ratio in optimizing reasoning and pattern recognition performance in small LLMs (SLMs). SmolTulu-1.7b-Instruct achieves state-of-the-art results among sub-2B parameter models on instruction following and mathematical reasoning tasks by meticulous adaptation of post-training procedures and optimization dynamics, without modifying the underlying model architecture (Alrashed, 2024).

1. Model Foundation and Architectural Characteristics

SmolTulu-1.7b-Instruct is built upon the SmolLM2-1.7B decoder-only Transformer architecture described by Allal et al. (2024). The key configuration parameters are:

• Number of Transformer layers: 32
• Model (hidden) dimensionality: 2,048
• Feed-forward inner dimensionality: 8,192
• Number of self-attention heads: 16
• Total parameter count: approximately 1.7 billion

No architectural modifications are introduced by SmolTulu; all improvements are attributable to post-training and alignment strategies. The model’s enhancements derive exclusively from procedure-level adaptations, encompassing instruction tuning and preference modeling, rather than network structural changes.

2. Instruction-Tuning and Alignment Pipeline

SmolTulu-1.7b-Instruct employs a post-training pipeline closely following the Tulu 3 recipe but calibrated for 1.7B parameters, comprising two principal stages:

A. Supervised Finetuning (SFT):

• Data: allenai/tulu-3-sft-mixture (includes sources such as alpaca, BBH, GSM8K, IFEval).
• Max sequence length: 2,048 tokens.
• Training: Single epoch over the dataset.
• Learning rate schedule: Linear decay with 10% warm-up.

B. Direct Preference Optimization (DPO):

• Data: allenai/llama-3.1-tulu-3-8b-preference-mixture (UltraFeedback + Tulu3 synthetic preferences).
• Loss function: Length-normalized DPO loss with KL-penalty $\beta=5$,

$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$

where $\pi_{\rm ref}$ refers to the SFT model.

• Max sequence length: 2,048 tokens.
• Training: One epoch, linear learning rate decay with 10% warm-up.

These procedures allow transfer of alignment strategies designed for larger LLMs to computationally efficient SLM baselines, preserving key alignment properties post-tuning.

3. Optimization Dynamics and Task-Dependent Hyperparameter Effects

A central finding is the strong impact of the learning-rate-to-batch-size ratio $R = \eta / B$ (where $\eta$ is peak learning rate and $B$ is batch size) on downstream model performance. Empirical exploration of several $(\eta, B)$ configurations elucidated that optimal $R$ is task dependent:

SFT Hyperparameters:

Variant	$\eta$	$B$	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$0
SmolTulu SFT-1130	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$1	8	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$2
SmolTulu SFT-1207	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$3	32	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$4
Tulu 3 (8B)	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$5	128	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$6
Tulu 3 (70B)	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$7	128	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$8

DPO Hyperparameters:

Variant	$$\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}} \log \sigma\biggl( \beta \Bigl[\log \frac{\pi_\theta(y_c|x)}{\pi_{\rm ref}(y_c|x)}\Bigr] -\beta \Bigl[\log \frac{\pi_\theta(y_r|x)}{\pi_{\rm ref}(y_r|x)}\Bigr] \biggr)$$9	$\pi_{\rm ref}$0	$\pi_{\rm ref}$1
SmolTulu DPO-1130	$\pi_{\rm ref}$2	12	$\pi_{\rm ref}$3
SmolTulu DPO-1207	$\pi_{\rm ref}$4	32	$\pi_{\rm ref}$5
Tulu 3 DPO 8B	$\pi_{\rm ref}$6	128	$\pi_{\rm ref}$7
Tulu 3 DPO 70B	$\pi_{\rm ref}$8	128	$\pi_{\rm ref}$9

In all cases, reasoning-centric benchmarks (e.g., ARC, GSM8K) benefit from higher $R = \eta / B$0, while pattern-recognition benchmarks (e.g., HellaSwag, IFEval) favor lower $R = \eta / B$1. The model capacity at 1.7B parameters generates a regime in which a single $R = \eta / B$2 cannot optimize all metrics uniformly, necessitating task-specific tuning.

4. Empirical Performance Evaluation

Post SFT and DPO, SmolTulu-1.7b-Instruct (DPO-1130) demonstrates substantial improvements on key tasks:

Task	SmolLM2-1.7B	SmolTulu DPO-1130	$R = \eta / B$3
GSM8K (5-shot)	48.2%	51.6%	+3.4%
ARC (avg)	51.7%	51.5%*	–0.2%*
ARC (alt config)	—	57.1%	+5.4%
HellaSwag	66.1%	61.1%*	–5.0%*
IFEval (avg)	56.7%	67.7%	+11.0%

*DPO-1130 yields strong reasoning gains on GSM8K and IFEval, with alternate (lower $R = \eta / B$4) DPO runs restoring ARC accuracy substantially. The improvements on GSM8K and IFEval are substantially above typical benchmark noise (±1–2%). No formal statistical confidence intervals were reported.

5. Ablation Analyses and Theoretical Context

Comprehensive ablation studies on a 135M-parameter SmolLM2 model reveal near-monotonic improvement in ARC and GSM8K accuracy as $R = \eta / B$5 increases, whereas HellaSwag and IFEval show non-monotonic behavior, peaking at intermediate $R = \eta / B$6 values ($R = \eta / B$7–$R = \eta / B$8). At 1.7B parameters, the model reveals capacity-limited divergence: high $R = \eta / B$9 benefits reasoning, while low $\eta$0 benefits pattern tasks, limiting universal gains from a single configuration.

Varying training epochs (from 1 to 2) shifts absolute scores by approximately 1–2% without altering $\eta$1–performance curve shapes. These trends substantiate the theoretical perspective articulated by Keskar et al. (2017) and Masters & Luschi (2018), suggesting that high-$\eta$2 (small batch, higher LR) training encourages flatter minima conducive to reasoning, whereas low-$\eta$3 (large batch) training supports pattern matching.

A plausible implication is that in small model regimes, optimization hyperparameters deserve particular scrutiny, specifically with respect to the end-task distribution of interest.

6. Training Recipe, Best Practice, and Generalization Guidelines

The canonical SmolTulu-1.7b-Instruct (DPO-1130) was produced with:

• SFT: LR $\eta$4, BS $\eta$5 ($\eta$6), 1 epoch, max seq. length $\eta$7, 10% warm-up.
• DPO: LR $\eta$8, BS $\eta$9 ($B$0), 1 epoch, max seq. length $B$1, 10% warm-up, KL penalty $B$2.

Best practice guidelines for sub-2B parameter LMs are documented as:

1. For SFT, use $B$3 if $B$4.
2. For DPO, employ $B$5 values 5–10× larger than those for 8B+ models ($B$6–$B$7).
3. Favor higher $B$8 for math and logic reasoning tasks.
4. Favor lower $B$9 for pattern recognition/commonsense tasks.
5. Only adjust $(\eta, B)$0 and $(\eta, B)$1—keep all other recipe elements (sequence length, warmup, epochs) consistent with Tulu 3.

Careful calibration of $(\eta, B)$2 enables SLMs to bridge much of the capability gap with multi-billion-parameter models on both instruction-following and reasoning benchmarks (Alrashed, 2024).