--- title: Open-Source Instruction-Tuned Models url: https://www.emergentmind.com/topics/open-source-instruction-tuned-models type: topic --- # Open-Source Instruction-Tuned Models Open-source instruction-tuned models are large language models (LLMs) whose parameters or adapters are freely available and which have been further trained via supervised or reinforcement learning on datasets of natural language “instructions” with corresponding outputs. Originating as a response to the dominance of closed, API-restricted systems, open-source instruction-tuned LLMs facilitate scientific progress, reproducibility, and wide accessibility across general, code, mathematical, social, multilingual, and specialized domains. These models, exemplified by major community and institutional efforts, provide both competitive performance on challenging benchmarks and serve as a platform for methodological innovation in data curation, fine-tuning, alignment, and evaluation. ## 1. Data Construction and Corpus Engineering Instruction-tuning relies fundamentally on the quality, diversity, and compositional structure of the instruction–response corpus. Open-source initiatives have produced both broad-coverage and specialist datasets: - **General-Domain Mixtures:** Projects such as TÜLU and Infinity-Instruct aggregate instructions from human-curated (e.g., SuperNI, FLAN, Dolly, OpenAssistant) and synthesized sources (e.g., Alpaca, Self-Instruct, ShareGPT, WizardLM evolutions), targeting coverage of tasks ranging from factual Q&A, reasoning, and translation, to dialogue and safety [2306.04751][2506.11116]. - **Domain-Specific Datasets:** OpenMathInstruct-1 contributes 1.8M math problem–solution pairs focusing on GSM8K and MATH via careful Mixtral-based synthesis, whereas OpenCodeInstruct offers 5M Python code instruction–solution pairs generated and filtered using open LLMs and multi-stage quality control [2402.01980][2504.04030]. - **Language/Locale Adaptation:** Efforts like Aya (101 languages), Okapi (26 languages, RLHF), Panda LLM (Chinese), and Camoscio (Italian) rely on translation pipelines—human or LLM-driven—paired with targeted filtering, deduplication and, in some cases, round-trip quality assurance [2402.07827][2307.16039][2305.03025][2307.16456]. - **Selection and Filtering:** Infinity-Instruct and OpenBezoar employ hybrid sampling, importance resampling (e.g., DSIR), and LLM-based (or GPT-4-based) post-generation filtering. This multi-stage curation significantly elevates mixture quality and empirical utility [2506.11116][2404.12195]. Typical datasets range from tens of thousands to over one hundred million instructions, with the best-performing systems placing particular emphasis on the proportional up-sampling of high-quality, diverse, and human-annotated domains (e.g., COIG for Panda-LLM; InfInstruct seed replay) [2305.03025][2506.11116]. ## 2. Model Architectures and Adaptation Recipes Instruction-tuned models build on a range of backbone LLM architectures, with adaptation mechanisms determined by compute and target deployment scenario: - **Transformer Variants:** The field is dominated by decoder-only architectures (LLaMA [Meta], Mistral, Falcon, OPT, Qwen, Pythia), though encoder–decoder (T5, mT5/UL2) variants are prominent for multilingual and discriminative tasks [2306.04757][2402.07827]. - **Parameter-Efficient Fine-Tuning:** LoRA and QLoRA are widely adopted, enabling low-rank adapter insertion and reducing the memory/computational cost for updating large language models (e.g., Socialite-Llama, Camoscio, OpenBezoar) [2402.01980][2307.16456][2404.12195]. Typical ranks are r=8–64 and adapters are inserted into attention projections. QLoRA further quantizes backbone weights to 4 bits (NF4) while training LoRA adapters. - **Alignment Methods:** While supervised fine-tuning (SFT) is the backbone of most instruction-tuning, recent models leverage reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO) for improved helpfulness, harmlessness, and honesty, as seen in Okapi, OpenBezoar, and large multilingual systems [2307.16039][2404.12195]. For model alignment through reward modeling or DPO, auxiliary datasets of ranked or paired human/LLM preferences are required. ## 3. Training Objectives, Optimization, and Schedules The dominant training objective for instruction tuning is cross-entropy over the target response, optionally masked to score only “assistant” roles in dialogue-style corpora: \[ \mathcal{L}(\theta)=-\sum_{i=1}^N\sum_{t=1}^{|Y_i|}\log P_\theta(y_{i,t}\mid y_{i,