SmolTalk: Open Dataset for Small LLMs

• SmolTalk is an open-source post-training dataset designed to enhance small language models' conversational fluency and instruction-following capabilities.
• It comprises 1.1 million high-quality instruction-response pairs, curated from synthetic generation and public datasets with rigorous filtering and deduplication.
• Empirical results show that integrating SmolTalk into the training pipeline boosts benchmarks like IFEval, MT-Bench, and GSM8K, especially in multi-turn dialogue tasks.

Within the cited literature, SmolTalk primarily denotes an open-source post-training and instruction-following dataset created during the development of SmolLM2, a 1.7 billion parameter LLM. It was introduced because fine-tuning the base model on existing public instruction datasets such as MagPie-Pro, OpenHermes2.5, UltraChat did not yield competitive post-trained model results compared to other small models, which motivated the creation of a new dataset specifically designed to improve alignment, conversational ability, and instruction-following performance in small LLMs (Allal et al., 4 Feb 2025). In subsequent comparative analysis, SmolTalk was examined side by side with Tulu-3-SFT-Mix as one of two prominent open post-training datasets, with particular attention to turn structure, task composition, and sample quality (Djuhera et al., 6 Jun 2025).

1. Origin, motivation, and design goal

SmolTalk emerged from a data-centric effort to strengthen post-training for small models. In the SmolLM2 development process, the authors state that existing datasets were at points problematically small or low-quality, leading them to introduce new specialized datasets, including SmolTalk, for stages where stronger instruction and conversational data were needed (Allal et al., 4 Feb 2025). The dataset is described as a curated mixture of newly generated synthetic datasets and carefully selected public datasets, with a strong emphasis on conversational, reasoning, and instruction-following coverage (Allal et al., 4 Feb 2025).

A later comparative study characterizes SmolTalk as targeting conversational depth and pragmatic rewriting for “smol” (small) models, with the aim of providing resource-efficient, rich multi-turn dialog without requiring large compute (Djuhera et al., 6 Jun 2025). The same study describes the dataset’s motivation as covering practical tasks and enhancing conversational fluency, context-handling, and non-trivial dialog for small LLMs (Djuhera et al., 6 Jun 2025). Taken together, these descriptions place SmolTalk at the intersection of alignment data engineering, multi-turn dialogue modeling, and efficient post-training for sub-2B models.

2. Construction and internal composition

SmolTalk has a reported total size of 1.1 million instruction-response pairs (Allal et al., 4 Feb 2025). Its construction combines newly generated synthetic conversational datasets, task-specific resources, and selected existing instruction data.

Dataset source	# Samples in SmolTalk
MagPie-Ultra	431k
Smol-Rewrite	56.2k
Smol-Constraints	36.2k
Smol-Summarization	101k
NuminaMath-CoT	112k
MetaMathQA	50k
Self-OSS-Starcoder2-Instruct	50.7k
APIGen-Function-Calling	87.5k
SystemChats2.0	35.9k
LongAlign	3.73k
Everyday-Conversations	2.38k
Explore-Instruct-Rewriting	32k
OpenHermes2.5	100k

The newly generated components include MagPie-Ultra, Smol-Constraint, Smol-Rewrite, and Smol-Summarization (Allal et al., 4 Feb 2025). MagPie-Ultra is described as 1 million three-turn conversation samples, generated using Llama-3.1-405B-Instruct with system prompts and then filtered and deduplicated (Allal et al., 4 Feb 2025). Smol-Constraint contains 36k+ instructions with strict and detailed constraints, targeting hard instruction-following (IFEval-like) behavior (Allal et al., 4 Feb 2025). Smol-Rewrite and Smol-Summarization focus on high-quality text rewriting and summarization, using synthetic source texts created with PersonaHub and FinePersonas, then rewritten with Qwen2.5-72B-Instruct (Allal et al., 4 Feb 2025).

The task-specific and domain-specific portions include NuminaMath-CoT, MetaMathQA, Self-OSS-Starcoder2-Instruct, APIGen-Function-Calling, SystemChats2.0, LongAlign, Everyday-Conversations, and Explore-Instruct-Rewriting (Allal et al., 4 Feb 2025). The dataset also includes 100k random OpenHermes2.5 samples for knowledge, and broad coverage (Allal et al., 4 Feb 2025).

Quality control is described in concrete terms. Synthetic generation used leading models with system prompts engineered for diversity, careful topic balance, and depth. Filtering was performed using small Llama models and custom reward models (ArmoRM), alongside safety filtering (Llama-Guard-3-8B). Deduplication used advanced embedding-based and semantic similarity (gte-large-en-v1.5) (Allal et al., 4 Feb 2025). A later study additionally states that all subsets underwent deduplication, quality filtering, n-gram decontamination (as in Tulu) (Djuhera et al., 6 Jun 2025).

3. Structural profile and annotated quality

Using the Magpie framework, a comparative study annotated SmolTalk at the sample level with metrics for turn structure, task category, input quality, and response quality (Djuhera et al., 6 Jun 2025). After annotation, the dataset contained 1,024,791 valid samples (Djuhera et al., 6 Jun 2025).

The most distinctive structural property is turn organization. 70% of SmolTalk is Multi-turn (MT), corresponding to 718,164 MT samples, with the remaining 306,627 being single-turn (Djuhera et al., 6 Jun 2025). The dataset is reported as being dominated by 6-turn convos (39.8% of data), then 3-turn convos (28%) (Djuhera et al., 6 Jun 2025). By task category, brainstorming, role playing, creative writing are reported as >90% multi-turn, whereas Math ST: 68% and Coding ST: 53.6% of respective categories (Djuhera et al., 6 Jun 2025).

Task category distribution was reported as General 57.6%, Knowledge Recall 9.4%, Math 4.6%, Reasoning 9.9%, Coding 7.1%, Safety 3.3%, and Precise IF tasks 8.2% (Djuhera et al., 6 Jun 2025). The same analysis states that Conversational/Creative Tasks—including brainstorming, editing, advice seeking, role playing, creative writing—make up ~30%, which is much higher than Tulu's ~10% (Djuhera et al., 6 Jun 2025). By contrast, Math & Coding are 4.6% and 7.1% respectively, both lower than Tulu’s 21.5% and 15.5% (Djuhera et al., 6 Jun 2025).

Input quality was summarized as 61.2% “excellent,” 23.8% “good”, meaning 85%+ are good/excellent (Djuhera et al., 6 Jun 2025). The same study reports that SmolTalk has a lower incidence of “poor”/“very poor” (8.5% combined) than Tulu (Djuhera et al., 6 Jun 2025). By turn type, ST: 95% excellent/good and MT: 80% excellent/good (Djuhera et al., 6 Jun 2025). For response quality, single-turn data exhibited a bell-shaped reward distribution with peak at +1, range -9 to +7, while multi-turn data showed that nearly 90% scored 5/5, ~8% scored 4/5, and ~2% score 0–3/5 (Djuhera et al., 6 Jun 2025).

Additional annotation axes further specify the corpus. The dataset is 99.3% English, with negligible multilingual content, and 99.1% samples flagged “safe” (Djuhera et al., 6 Jun 2025). The authors also note that multi-turn often recovers from poor initial queries via clarification and iterative improvement, which is used to explain why input quality–Reward correlation is weaker on MT than on ST (Djuhera et al., 6 Jun 2025). This suggests that SmolTalk’s multi-turn structure is not merely a formatting property; it is bound up with the dataset’s intended conversational dynamics.

4. Role in SmolLM2 post-training

In SmolLM2, SmolTalk was used for the Supervised Fine-Tuning (SFT) stage for 2 epochs, with sequence length increased to 8192 to maintain long-context capabilities (Allal et al., 4 Feb 2025). The reported SFT hyperparameters were batch size of 128 and learning rate of $3.0 \times 10^{-4}$ (Allal et al., 4 Feb 2025). The resulting SFT model then underwent Direct Preference Optimization (DPO) for 2 epochs, using UltraFeedback for preference data; the authors explicitly note that this did not affect context ability (Allal et al., 4 Feb 2025).

The SFT phase used the full SmolTalk mixture, maintaining a broad domain balance, with empirically tuned proportions per dataset (Allal et al., 4 Feb 2025). For the smaller 135M and 360M models, a filtered version of SmolTalk was used, removing most complex and hard instruction samples, in order to better match model capacity (Allal et al., 4 Feb 2025).

Ablation studies compared SmolLM2 fine-tuned on OpenHermes2.5, UltraChat, MagPie-Pro, and SmolTalk-related mixtures (Allal et al., 4 Feb 2025). The authors report that fine-tuning on only MagPie-Ultra and then incrementally adding Smol-constraint/rewrite/summarization produced stepwise improvements, particularly on instruction-following (IFEval), text rewriting, and math (Allal et al., 4 Feb 2025). They also report that mixing MagPie-Ultra+ with either NuminaMath-CoT or MetaMathQA improved different math benchmarks, and therefore both were included in SmolTalk (Allal et al., 4 Feb 2025).

The benchmark values given for SmolTalk were IFEval 46.67, MT-Bench 5.49, GSM8K 43.75, MATH 18.60, ARC 40.02, and MMLU-Pro 18.19 (Allal et al., 4 Feb 2025). The final SmolLM2-SFT†, trained on the full SmolTalk mixture, reached IFEval 57.09, MT-Bench 6.11, GSM8K 47.54, MATH 19.64, ARC 42.49, and MMLU-Pro 19.06 (Allal et al., 4 Feb 2025). The paper states that the final SFT checkpoint using the full SmolTalk dataset achieved the strongest results across major benchmarks (Allal et al., 4 Feb 2025).

5. Comparison with Tulu-3-SFT-Mix and the creation of TuluTalk

The comparative study in (Djuhera et al., 6 Jun 2025) frames SmolTalk and Tulu-3-SFT-Mix as complementary open post-training datasets. SmolTalk is summarized as 70% multi-turn, whereas Tulu is 95% single-turn (Djuhera et al., 6 Jun 2025). In qualitative terms, SmolTalk is described as strong for fluency in conversation, dialog adaptation, natural turn-taking, while Tulu is described as strong for factual accuracy, structured reasoning, and coding/math protocols (Djuhera et al., 6 Jun 2025).

The quantitative contrast is also explicit. SmolTalk has 4.6% math and 7.1% coding, compared with Tulu’s 21.5% and 15.5% (Djuhera et al., 6 Jun 2025). The same study states that SmolTalk SFT outperforms Tulu on OpenLLM Leaderboards; Tulu does better at code benchmarks. Both > Orca baseline (Djuhera et al., 6 Jun 2025). This comparison is central to the design of TuluTalk, a curated mixture built from both sources.

The TuluTalk construction pipeline is reported in three steps: Step 1 filters both datasets for high input and response quality using Magpie; Step 2 identifies underrepresented task categories in the filtered result; Step 3 re-adds samples from underrepresented tasks—often drawn from SmolTalk—to restore task diversity (Djuhera et al., 6 Jun 2025). The resulting dataset, TuluTalk, is reported as 808k samples, 14% smaller than Tulu, and 27% smaller than SmolTalk, while matching or exceeding the source datasets on benchmarks (Djuhera et al., 6 Jun 2025). The study attributes a specific role to SmolTalk here: it supplied the bulk of multi-turn, dialogically rich, and creative instruction samples in the final blend (Djuhera et al., 6 Jun 2025).

6. Extensions, derivatives, and terminological scope

SmolTalk has also served as a template for derivative corpora. Smoltalk-chinese, introduced as part of the OpenCSG Chinese Corpus, is described as a high-quality, open-source synthetic dataset of Chinese multi-turn, chat-format conversations for LLM training and instruction fine-tuning (Yu et al., 14 Jan 2025). It is explicitly said to be inspired by SmolTalk and Magpie-ultra-1M, while extending them from 11 task categories to 18 categories by adding format constraints, summarization, rewriting, document-QA, safe-QA, translation, everyday-talk (Yu et al., 14 Jan 2025). Construction used Deepseek-V2.5 and Qwen2.5-72B-Instruct; the first user query in each dialog was scored using Qwen2.5-7b-instruct, and only conversations with score > 3 out of 5 were retained (Yu et al., 14 Jan 2025). Deduplication used gte-zh-large, with cosine similarity thresholds of 0.8 for multi-turn dialogs and 0.7 for single-turn samples (Yu et al., 14 Jan 2025). The final size is reported as ~70,000 high-quality samples, and fine-tuning on this dataset achieved the strongest overall performance on AlignBench among the datasets compared (Yu et al., 14 Jan 2025).

The name also appears in adjacent literatures with different meanings. In one human-robot interaction study, robot-initiated small talk is termed “SmolTalk” and tested in a collaborative assembly task using a Franka Emika Panda robotic arm; the study reports significantly higher levels of rapport in the social condition, while participants in that condition also recorded longer task durations (Pineda et al., 22 Jan 2025). In multimodal language modeling, the similarly named SmolTolk denotes a family of Text-Speech LLMs built on SmolLM; that paper explicitly states that SmolTolk is not synonymous with, or related to, other existing “SmolTalk” models or agents (Cuervo et al., 8 Mar 2025). These distinctions matter because the dominant technical usage in language-model post-training refers to the dataset lineage described above, whereas the label is reused in robotics and multimodal speech-text work for different objects of study.

SmolTalk’s significance in small-model research therefore lies in a specific combination of properties documented across these papers: it is dialog-heavy, quality-filtered, task-diverse, and directly tied to the empirical post-training gains of SmolLM2 and to later curation work such as TuluTalk (Allal et al., 4 Feb 2025). A plausible implication is that SmolTalk’s main research value is not only its scale, but the way it operationalizes multi-turn dialogue, pragmatic rewriting, and hard instruction-following within an open post-training corpus for resource-constrained LLMs.