---
title: Nemotron-Synth Data Pipeline
url: https://www.emergentmind.com/topics/nemotron-synth
type: topic
---

# Nemotron-Synth Data Pipeline

Searching arXiv for the cited Nemotron and DPO papers to ground the article.
Use the arXiv search tool now.
Nemotron-Synth is the synthetic data generation pipeline described in the "Nemotron-4 340B Technical Report" [2406.11704]. It is presented as the data engine underlying the alignment of the Nemotron-4-340B model family, with over 98% of the data used in the model alignment process being synthetically generated. In the report, Nemotron-Synth is organized as a staged pipeline linking prompt preparation, synthetic dialogue generation, synthetic preference data generation, and an iterative weak-to-strong alignment loop. Its stated purpose is to support the training of instruct and reward models, and more broadly to facilitate the generation of synthetic data for smaller language models and related open research [2406.11704].

## 1. Definition and system-level structure

Nemotron-Synth consists of four major stages, each feeding into the next [2406.11704]. The first stage is **Prompt Preparation**, which includes synthetic single-turn prompts, synthetic instruction-following prompts, synthetic two-turn prompts for preference data, and real-world prompts drawn from LMSYS-Chat-1M and ShareGPT. The second stage is **Synthetic Dialogue Generation**, which uses role-playing over three-turn conversations, greedy decoding for demonstration data, and reward-model filtering with Nemotron-4-340B-Reward. The third stage is **Synthetic Preference Data Generation**, which uses multi-model response sampling together with ground-truth or “verifier” judging for tasks with known answers, followed by LLM-as-Judge and ultimately Reward-Model-as-Judge using Nemotron-4-340B-Reward. The fourth stage is **Iterative Weak-to-Strong Alignment**, defined as a cycle in which the current “teacher” instruct model generates prompts and responses, the base model is supervised fine-tuned, the instruct model is preference-tuned, and the resulting stronger model is used to repeat the process [2406.11704].

The report explicitly schematizes this process as a linear pipeline from prompt preparation to dialogue generation to preference generation to an alignment loop defined as “SFT + DPO/RPO.” This framing is significant because it places synthetic data generation and alignment within a single closed system rather than treating data creation as a separate preprocessing task. A plausible implication is that Nemotron-Synth is designed not merely to populate datasets, but to recursively improve the quality of the models that generate those datasets.

A common misconception in alignment discourse is that high-performing instruction-tuned models necessarily require predominantly human-annotated alignment corpora. Nemotron-Synth is presented as a counterexample to that assumption: the report states that over 98% of the data used in the model alignment process is synthetic [2406.11704].

## 2. Prompt construction workflows

The report provides pseudocode for three core prompt-generation workflows [2406.11704]. For **synthetic single-turn prompts**, the process begins with model-generated macro-topics, expands them into sub-topics, and then derives task classes including open-QA prompts, writing prompts, closed-QA prompts, and math and coding problems. Open-QA prompts are generated by requesting questions related to macro-topics and sub-topics and then revising those questions “to add context and detail.” Writing prompts are generated in a templated form such as creating a specified document type about a specified topic. Closed-QA prompts are derived from sampled documents from the C4 corpus by asking for questions or tasks conditioned on a provided text. Math and coding prompts are generated from keyword lists spanning math and Python [2406.11704].

For **synthetic instruction-following prompts**, the workflow samples base prompts from synthetic prompts, draws instruction templates from a resource named `"verifiable_instruction_templates"`, and appends those instructions to the sampled prompts. The report also defines a multi-turn variant using a “global instruction” format of the form “All future responses must …” followed by a user prompt. This separates topical content generation from controllability constraints and creates prompt instances suitable for instruction-following evaluation and training [2406.11704].

For **synthetic two-turn prompts** used in preference data generation, the workflow begins from a real user prompt sourced from ShareGPT or LMSYS. An instruct model generates the assistant reply, and then the instruct model is prompted again to generate a follow-up question conditioned on that assistant reply. The resulting tuple contains the first-turn user prompt, the assistant reply, and the follow-up question [2406.11704]. This design makes preference instances inherently conversational rather than purely single-turn.

The report does not disclose exact numeric values for several generation settings. It states that no explicit temperature or top-$k$ values were reported. Where prompt-source mixing is concerned, the report notes that synthetic and real-world prompts were “combined in a balanced ratio,” but exact weights were not disclosed. It provides only the generic weighting form
$$
P(\text{pick source }i) \;=\; \frac{w_i}{\sum_j w_j}\,.
$$
This is important for interpretation: many operational details of prompt mixture calibration are intentionally left unspecified in the report [2406.11704].

## 3. Dialogue synthesis, response sampling, and judging

In the dialogue-generation stage, Nemotron-Synth uses **role-playing over three-turn conversations** and **greedy decoding for demonstration data** [2406.11704]. The reward model used for filtering is **Nemotron-4-340B-Reward**. The report’s description indicates that synthetic dialogue generation is not based on stochastic exploration for demonstrations; rather, demonstrations are generated greedily and then filtered by the reward model. This suggests an emphasis on producing high-confidence demonstrations before later introducing diversity through preference-response sampling.

The preference-data stage uses a different strategy. For **preference responses**, the report specifies **sampling with different random seeds / models to ensure response diversity** [2406.11704]. Selection among these responses is performed using multiple judging mechanisms depending on task type. For tasks with known answers, **ground-truth / “verifier” judging** is used. More generally, the report describes a progression from **LLM-as-Judge** to **Reward-Model-as-Judge**, culminating again in the use of Nemotron-4-340B-Reward [2406.11704].

This separation between demonstration generation and preference generation is methodologically consequential. Demonstration data are obtained with greedy decoding and reward filtering, whereas preference data depend on diversity-inducing sampling and adjudication. A plausible implication is that Nemotron-Synth distinguishes between two training objectives: imitation of strong outputs in supervised fine-tuning, and comparative optimization over candidate outputs in preference tuning.

## 4. Training configuration and alignment loop

The report states that the **generation model for prompts** is **Mixtral-8x7B-Instruct-v0.1 in bfloat16**, while the **reward model** is **Nemotron-4-340B-Reward** [2406.11704]. Alignment training runs on **8×H100 per DGX node**, with **FP8 inference** and **bfloat16 training**. The broader Nemotron-4 340B family was sized to fit on a single DGX H100 with 8 GPUs when deployed in FP8 precision [2406.11704].

The supervised fine-tuning schedule is split into **Code SFT** and **General SFT**. Code SFT uses a dataset of **800 K synthetic code instructions (Genetic Instruct)** for **1 epoch**, with **LR=3e-7** and **batch size=128**. General SFT uses **200 K blended supervised samples (+2% code from stage 1)** for **3 epochs**, with **LR∈[1e-7,5e-7]**, **batch size=128**, and **loss only on assistant tokens** [2406.11704].

Preference fine-tuning is described in two stages: **DPO** and **RPO**. The DPO stage uses **160 K triplets**, **1 epoch**, **LR∈[3e-8,3e-7]**, and **batch size=256**. Its hyperparameters include **KL coeff $\beta\in[3e-4,3e-3]$** and **SFT-loss weight $\in[1e-5,1e-3]$**. The RPO stage consists of **3 iterations**, each with **300 K triplets**, with **initialization & reference = previous checkpoint**, **fixed LR=3e-7**, **SFT-loss weight=1e-5**, **$\eta=1$**, and **tune $\beta\in[1e-3, 1]$** [2406.11704].

The system-level alignment cycle is explicitly described as follows: use the current “teacher” instruct model to generate prompts and responses; supervised fine-tune the base model; preference-tune the instruct model; produce a new stronger model; and repeat [2406.11704]. This iterative weak-to-strong scheme is central to the report’s framing of Nemotron-Synth. It is not merely a static synthetic dataset pipeline, but a recursive alignment process in which successive generators become stronger synthetic teachers.

## 5. Preference optimization objectives

The report defines Nemotron-Synth’s preference optimization using **Direct Preference Optimization (DPO)** and **Reward-aware Preference Optimization (RPO)** [2406.11704]. For DPO, it states that the standard objective from Rafailov et al., 2024 optimizes the gap in log-ratios:
$$
\mathcal{L}_{\mathrm{DPO} \;=\;
-\beta\bigl[\log\tfrac{\pi(y_c\mid x)}{\pi_{\mathrm{ref}(y_c\mid x)}
-\log\tfrac{\pi(y_r\mid x)}{\pi_{\mathrm{ref}(y_r\mid x)}\bigr]\,.
$$
The report further adds a small SFT term on chosen responses:
$$
\mathcal{L} = \mathcal{L}_\mathrm{DPO}
+ \lambda_\mathrm{SFT}\,\bigl(-\log\pi(y_c\mid x)\bigr)\,.
$$
It states that this additional term helps prevent over-shifting [2406.11704].

For RPO, the report states that the objective fits the magnitude of the policy’s implicit gap to the model-predicted reward gap. Letting $r^\star(x,y)$ denote the reward model’s score, it gives
$$
\mathcal{L}_{\mathrm{RPO}(x,y_c,y_r)
= \mathbb{D}\!\Bigl[
\beta\log\tfrac{\pi(y_c\mid x)}{\pi_{\mathrm{ref}(y_c\mid x)}
- \beta\log\tfrac{\pi(y_r\mid x)}{\pi_{\mathrm{ref}(y_r\mid x)}
\;\Big\|\;\eta\bigl(r^\star(x,y_c)-r^\star(x,y_r)\bigr)\Bigr],
$$
where $\mathbb{D}[a\|b]$ is a divergence, described in the report as a KL based on $\sigma(a)$, and $\eta=1$ [2406.11704].

The report interprets this construction as anchoring the policy’s internal gap to the “true” reward gap and as preventing over-penalization of near-optimal rejected responses. Within the Nemotron-Synth pipeline, this is consequential because preference data are themselves synthetically generated and judged. A plausible implication is that RPO is intended to regularize preference learning when comparative labels are abundant but potentially heterogeneous in margin quality.

## 6. Reported empirical effects

The report attributes most of the final instruct model’s performance to Nemotron-Synth on the grounds that over 98% of the alignment data is synthetic [2406.11704]. It presents an ablation showing stepwise gains as synthetic data and alignment algorithms are applied.

| Stage | MT-Bench | MMLU |
|---|---:|---:|
| Base | — | 81.1 |
| Code SFT | 6.79 | 72.2 |
| +Gen SFT | 7.99 | 78.3 |
| +DPO | 7.90 | 78.4 |
| +3×RPO | 8.22 | 78.7 |

| Stage | GSM8K | HumanEval | IFEval(PSA) |
|---|---:|---:|---:|
| Base | — | 57.3 | 46.4 |
| Code SFT | 77.6 | 70.7 | 46.4 |
| +Gen SFT | 87.9 | 66.5 | 61.4 |
| +DPO | 88.5 | 67.1 | 61.7 |
| +3×RPO | 92.3 | 73.2 | 79.9 |

The report specifically highlights that **HumanEval rises from 57.3 to 73.2 (pass@1)**, **GSM8K 0-shot jumps to 92.3%**, and **IFEval prompt-strict accuracy leaps from 46.4% to 79.9%** [2406.11704]. It characterizes these gains as spanning coding, reasoning, instruction-following, and multi-turn dialogue.

The ablation also shows that the pipeline is not monotonic on every intermediate benchmark. For example, MMLU is lower after Code SFT than at Base, while later stages recover part of that reduction, and HumanEval decreases from Code SFT to +Gen SFT before rising again after preference optimization [2406.11704]. This matters because it complicates any simplistic interpretation that every added synthetic stage uniformly improves all capabilities. The report’s own evidence instead indicates stage-specific trade-offs followed by later recovery or improvement.

## 7. Position within open-model and synthetic-data research

The Nemotron-4 340B Technical Report states that the model family is released as open access under the **NVIDIA Open Model License Agreement**, described as a permissive model license allowing distribution, modification, and use of the models and their outputs [2406.11704]. It also states that the synthetic data generation pipeline used in the model alignment process is being open-sourced to support open research and facilitate model development. Within that framing, Nemotron-Synth occupies a dual role: it is both an internal alignment mechanism for the Nemotron-4-340B model family and a reusable research asset intended for broader synthetic-data workflows.

A recurring point of confusion in discussion of such systems is whether “synthetic data” here refers only to model-generated prompts or only to model-generated answers. The report makes clear that Nemotron-Synth covers both. It includes synthetic prompt generation, synthetic dialogue generation, synthetic preference generation, and synthetic judging pathways, while still incorporating real-world prompts from LMSYS-Chat-1M and ShareGPT in a balanced but undisclosed ratio [2406.11704]. The resulting pipeline is therefore hybrid in input provenance but predominantly synthetic in aligned supervision.

Another possible misconception is that the report discloses a fully reproducible sampling recipe. It does not. The report explicitly withholds exact weights for prompt-source balancing and does not report explicit temperature or top-$k$ values [2406.11704]. For technical readers, this means that Nemotron-Synth is precisely specified at the architectural and training-loop levels, but only partially specified at the level of low-level generation controls.

Taken together, the report presents Nemotron-Synth as a staged synthetic-data and alignment framework in which prompt generation, dialogue generation, comparative judging, and iterative preference optimization are tightly coupled. The central empirical claim is that such a pipeline can support a high-performing instruct model even when the alignment corpus is overwhelmingly synthetic [2406.11704].

Source: https://www.emergentmind.com/topics/nemotron-synth