Nemotron-Synth Data Pipeline

• Nemotron-Synth is a four-stage synthetic data generation framework that creates prompts, dialogues, and preference data to align language models.
• It employs sequential processes—from prompt preparation to synthetic dialogue generation and iterative alignment—to recursively improve model performance.
• Empirical results show significant gains across benchmarks such as GSM8K, HumanEval, and IFEval, underscoring its efficiency over traditional human annotations.

Searching arXiv for the cited Nemotron and DPO papers to ground the article. Use the arXiv search tool now. Nemotron-Synth is the synthetic data generation pipeline described in the "Nemotron-4 340B Technical Report" (NVIDIA et al., 2024). It is presented as the data engine underlying the alignment of the Nemotron-4-340B model family, with over 98% of the data used in the model alignment process being synthetically generated. In the report, Nemotron-Synth is organized as a staged pipeline linking prompt preparation, synthetic dialogue generation, synthetic preference data generation, and an iterative weak-to-strong alignment loop. Its stated purpose is to support the training of instruct and reward models, and more broadly to facilitate the generation of synthetic data for smaller LLMs and related open research (NVIDIA et al., 2024).

1. Definition and system-level structure

Nemotron-Synth consists of four major stages, each feeding into the next (NVIDIA et al., 2024). The first stage is Prompt Preparation, which includes synthetic single-turn prompts, synthetic instruction-following prompts, synthetic two-turn prompts for preference data, and real-world prompts drawn from LMSYS-Chat-1M and ShareGPT. The second stage is Synthetic Dialogue Generation, which uses role-playing over three-turn conversations, greedy decoding for demonstration data, and reward-model filtering with Nemotron-4-340B-Reward. The third stage is Synthetic Preference Data Generation, which uses multi-model response sampling together with ground-truth or “verifier” judging for tasks with known answers, followed by LLM-as-Judge and ultimately Reward-Model-as-Judge using Nemotron-4-340B-Reward. The fourth stage is Iterative Weak-to-Strong Alignment, defined as a cycle in which the current “teacher” instruct model generates prompts and responses, the base model is supervised fine-tuned, the instruct model is preference-tuned, and the resulting stronger model is used to repeat the process (NVIDIA et al., 2024).

The report explicitly schematizes this process as a linear pipeline from prompt preparation to dialogue generation to preference generation to an alignment loop defined as “SFT + DPO/RPO.” This framing is significant because it places synthetic data generation and alignment within a single closed system rather than treating data creation as a separate preprocessing task. A plausible implication is that Nemotron-Synth is designed not merely to populate datasets, but to recursively improve the quality of the models that generate those datasets.

A common misconception in alignment discourse is that high-performing instruction-tuned models necessarily require predominantly human-annotated alignment corpora. Nemotron-Synth is presented as a counterexample to that assumption: the report states that over 98% of the data used in the model alignment process is synthetic (NVIDIA et al., 2024).

2. Prompt construction workflows

The report provides pseudocode for three core prompt-generation workflows (NVIDIA et al., 2024). For synthetic single-turn prompts, the process begins with model-generated macro-topics, expands them into sub-topics, and then derives task classes including open-QA prompts, writing prompts, closed-QA prompts, and math and coding problems. Open-QA prompts are generated by requesting questions related to macro-topics and sub-topics and then revising those questions “to add context and detail.” Writing prompts are generated in a templated form such as creating a specified document type about a specified topic. Closed-QA prompts are derived from sampled documents from the C4 corpus by asking for questions or tasks conditioned on a provided text. Math and coding prompts are generated from keyword lists spanning math and Python (NVIDIA et al., 2024).

For synthetic instruction-following prompts, the workflow samples base prompts from synthetic prompts, draws instruction templates from a resource named "verifiable_instruction_templates", and appends those instructions to the sampled prompts. The report also defines a multi-turn variant using a “global instruction” format of the form “All future responses must …” followed by a user prompt. This separates topical content generation from controllability constraints and creates prompt instances suitable for instruction-following evaluation and training (NVIDIA et al., 2024).

For synthetic two-turn prompts used in preference data generation, the workflow begins from a real user prompt sourced from ShareGPT or LMSYS. An instruct model generates the assistant reply, and then the instruct model is prompted again to generate a follow-up question conditioned on that assistant reply. The resulting tuple contains the first-turn user prompt, the assistant reply, and the follow-up question (NVIDIA et al., 2024). This design makes preference instances inherently conversational rather than purely single-turn.

The report does not disclose exact numeric values for several generation settings. It states that no explicit temperature or top-$k$ values were reported. Where prompt-source mixing is concerned, the report notes that synthetic and real-world prompts were “combined in a balanced ratio,” but exact weights were not disclosed. It provides only the generic weighting form

$$P(\text{pick source }i) \;=\; \frac{w_i}{\sum_j w_j}\,.$$

This is important for interpretation: many operational details of prompt mixture calibration are intentionally left unspecified in the report (NVIDIA et al., 2024).

3. Dialogue synthesis, response sampling, and judging

In the dialogue-generation stage, Nemotron-Synth uses role-playing over three-turn conversations and greedy decoding for demonstration data (NVIDIA et al., 2024). The reward model used for filtering is Nemotron-4-340B-Reward. The report’s description indicates that synthetic dialogue generation is not based on stochastic exploration for demonstrations; rather, demonstrations are generated greedily and then filtered by the reward model. This suggests an emphasis on producing high-confidence demonstrations before later introducing diversity through preference-response sampling.

The preference-data stage uses a different strategy. For preference responses, the report specifies sampling with different random seeds / models to ensure response diversity (NVIDIA et al., 2024). Selection among these responses is performed using multiple judging mechanisms depending on task type. For tasks with known answers, ground-truth / “verifier” judging is used. More generally, the report describes a progression from LLM-as-Judge to Reward-Model-as-Judge, culminating again in the use of Nemotron-4-340B-Reward (NVIDIA et al., 2024).

This separation between demonstration generation and preference generation is methodologically consequential. Demonstration data are obtained with greedy decoding and reward filtering, whereas preference data depend on diversity-inducing sampling and adjudication. A plausible implication is that Nemotron-Synth distinguishes between two training objectives: imitation of strong outputs in supervised fine-tuning, and comparative optimization over candidate outputs in preference tuning.

4. Training configuration and alignment loop

The report states that the generation model for prompts is Mixtral-8x7B-Instruct-v0.1 in bfloat16, while the reward model is Nemotron-4-340B-Reward (NVIDIA et al., 2024). Alignment training runs on 8×H100 per DGX node, with FP8 inference and bfloat16 training. The broader Nemotron-4 340B family was sized to fit on a single DGX H100 with 8 GPUs when deployed in FP8 precision (NVIDIA et al., 2024).

The supervised fine-tuning schedule is split into Code SFT and General SFT. Code SFT uses a dataset of 800 K synthetic code instructions (Genetic Instruct) for 1 epoch, with LR=3e-7 and batch size=128. General SFT uses 200 K blended supervised samples (+2% code from stage 1) for 3 epochs, with LR∈[1e-7,5e-7], batch size=128, and loss only on assistant tokens (NVIDIA et al., 2024).

Preference fine-tuning is described in two stages: DPO and RPO. The DPO stage uses 160 K triplets, 1 epoch, LR∈[3e-8,3e-7], and batch size=256. Its hyperparameters include KL coeff $\beta\in[3e-4,3e-3]$ and SFT-loss weight $\in[1e-5,1e-3]$. The RPO stage consists of 3 iterations, each with 300 K triplets, with initialization & reference = previous checkpoint, fixed LR=3e-7, SFT-loss weight=1e-5, $\eta=1$, and tune $\beta\in[1e-3, 1]$ (NVIDIA et al., 2024).

The system-level alignment cycle is explicitly described as follows: use the current “teacher” instruct model to generate prompts and responses; supervised fine-tune the base model; preference-tune the instruct model; produce a new stronger model; and repeat (NVIDIA et al., 2024). This iterative weak-to-strong scheme is central to the report’s framing of Nemotron-Synth. It is not merely a static synthetic dataset pipeline, but a recursive alignment process in which successive generators become stronger synthetic teachers.

5. Preference optimization objectives

The report defines Nemotron-Synth’s preference optimization using Direct Preference Optimization (DPO) and Reward-aware Preference Optimization (RPO) (NVIDIA et al., 2024). For DPO, it states that the standard objective from Rafailov et al., 2024 optimizes the gap in log-ratios:

$\mathcal{L}_{\mathrm{DPO} \;=\; -\beta\bigl[\log\tfrac{\pi(y_c\mid x)}{\pi_{\mathrm{ref}(y_c\mid x)} -\log\tfrac{\pi(y_r\mid x)}{\pi_{\mathrm{ref}(y_r\mid x)}\bigr]\,.$

The report further adds a small SFT term on chosen responses:

$$\mathcal{L} = \mathcal{L}_\mathrm{DPO} + \lambda_\mathrm{SFT}\,\bigl(-\log\pi(y_c\mid x)\bigr)\,.$$

It states that this additional term helps prevent over-shifting (NVIDIA et al., 2024).

For RPO, the report states that the objective fits the magnitude of the policy’s implicit gap to the model-predicted reward gap. Letting $r^\star(x,y)$ denote the reward model’s score, it gives

$\mathcal{L}_{\mathrm{RPO}(x,y_c,y_r) = \mathbb{D}\!\Bigl[ \beta\log\tfrac{\pi(y_c\mid x)}{\pi_{\mathrm{ref}(y_c\mid x)} - \beta\log\tfrac{\pi(y_r\mid x)}{\pi_{\mathrm{ref}(y_r\mid x)} \;\Big\|\;\eta\bigl(r^\star(x,y_c)-r^\star(x,y_r)\bigr)\Bigr],$

where $$P(\text{pick source }i) \;=\; \frac{w_i}{\sum_j w_j}\,.$$0 is a divergence, described in the report as a KL based on $$P(\text{pick source }i) \;=\; \frac{w_i}{\sum_j w_j}\,.$$1, and $$P(\text{pick source }i) \;=\; \frac{w_i}{\sum_j w_j}\,.$$2 (NVIDIA et al., 2024).

The report interprets this construction as anchoring the policy’s internal gap to the “true” reward gap and as preventing over-penalization of near-optimal rejected responses. Within the Nemotron-Synth pipeline, this is consequential because preference data are themselves synthetically generated and judged. A plausible implication is that RPO is intended to regularize preference learning when comparative labels are abundant but potentially heterogeneous in margin quality.

6. Reported empirical effects

The report attributes most of the final instruct model’s performance to Nemotron-Synth on the grounds that over 98% of the alignment data is synthetic (NVIDIA et al., 2024). It presents an ablation showing stepwise gains as synthetic data and alignment algorithms are applied.

Stage	MT-Bench	MMLU
Base	—	81.1
Code SFT	6.79	72.2
+Gen SFT	7.99	78.3
+DPO	7.90	78.4
+3×RPO	8.22	78.7

Stage	GSM8K	HumanEval	IFEval(PSA)
Base	—	57.3	46.4
Code SFT	77.6	70.7	46.4
+Gen SFT	87.9	66.5	61.4
+DPO	88.5	67.1	61.7
+3×RPO	92.3	73.2	79.9

The report specifically highlights that HumanEval rises from 57.3 to 73.2 (pass@1), GSM8K 0-shot jumps to 92.3%, and IFEval prompt-strict accuracy leaps from 46.4% to 79.9% (NVIDIA et al., 2024). It characterizes these gains as spanning coding, reasoning, instruction-following, and multi-turn dialogue.

The ablation also shows that the pipeline is not monotonic on every intermediate benchmark. For example, MMLU is lower after Code SFT than at Base, while later stages recover part of that reduction, and HumanEval decreases from Code SFT to +Gen SFT before rising again after preference optimization (NVIDIA et al., 2024). This matters because it complicates any simplistic interpretation that every added synthetic stage uniformly improves all capabilities. The report’s own evidence instead indicates stage-specific trade-offs followed by later recovery or improvement.

7. Position within open-model and synthetic-data research

The Nemotron-4 340B Technical Report states that the model family is released as open access under the NVIDIA Open Model License Agreement, described as a permissive model license allowing distribution, modification, and use of the models and their outputs (NVIDIA et al., 2024). It also states that the synthetic data generation pipeline used in the model alignment process is being open-sourced to support open research and facilitate model development. Within that framing, Nemotron-Synth occupies a dual role: it is both an internal alignment mechanism for the Nemotron-4-340B model family and a reusable research asset intended for broader synthetic-data workflows.

A recurring point of confusion in discussion of such systems is whether “synthetic data” here refers only to model-generated prompts or only to model-generated answers. The report makes clear that Nemotron-Synth covers both. It includes synthetic prompt generation, synthetic dialogue generation, synthetic preference generation, and synthetic judging pathways, while still incorporating real-world prompts from LMSYS-Chat-1M and ShareGPT in a balanced but undisclosed ratio (NVIDIA et al., 2024). The resulting pipeline is therefore hybrid in input provenance but predominantly synthetic in aligned supervision.

Another possible misconception is that the report discloses a fully reproducible sampling recipe. It does not. The report explicitly withholds exact weights for prompt-source balancing and does not report explicit temperature or top-$$P(\text{pick source }i) \;=\; \frac{w_i}{\sum_j w_j}\,.$$3 values (NVIDIA et al., 2024). For technical readers, this means that Nemotron-Synth is precisely specified at the architectural and training-loop levels, but only partially specified at the level of low-level generation controls.

Taken together, the report presents Nemotron-Synth as a staged synthetic-data and alignment framework in which prompt generation, dialogue generation, comparative judging, and iterative preference optimization are tightly coupled. The central empirical claim is that such a pipeline can support a high-performing instruct model even when the alignment corpus is overwhelmingly synthetic (NVIDIA et al., 2024).