PiKa-SFT: Data-Efficient Synthetic Alignment

• The paper demonstrates that 30k expertly curated instruction–response pairs can outperform larger datasets in aligning LLMs across various benchmarks.
• The dataset is generated via a three-step synthesis pipeline, including persona-driven instruction creation, candidate response scoring with reward models, and GPT-4o-based filtering.
• Experimental results show that PiKa-SFT fine-tuning yields higher win rates on alignment benchmarks compared to larger public SFT datasets, highlighting its sample efficiency.

PiKa-SFT is the supervised fine-tuning component of PiKa, a family of expert-level synthetic alignment datasets designed for data-efficient post-training alignment of LLMs. In the reported formulation, PiKa-SFT contains 30k high-difficulty, expert-quality instruction–response pairs, and it is paired with an additional 30k instruction–(preferred, dispreferred) triples for preference optimization after SFT initialization. Its defining premise is that alignment quality can be improved by emphasizing difficulty, feasibility, and response quality rather than corpus scale alone; the paper evaluates this premise by fine-tuning Llama-3-8B-Base and Qwen2.5 models and reports that PiKa-SFT can outperform much larger public SFT datasets, and on some benchmarks can exceed the official Llama-3-8B-Instruct model trained on over 10 million proprietary examples (Yin et al., 8 Oct 2025).

1. Conceptual position within post-training alignment

PiKa-SFT is presented against the background of RLHF and RLAIF, where instruction data quality is treated as a bottleneck for downstream alignment. The paper argues that many existing alignment datasets are either private or require costly human annotation, which limits reproducibility and scalability, and that even public SFT pipelines often rely on more than 300k examples while still underperforming proprietary instruct models. PiKa-SFT is therefore positioned as a data-efficient alternative: a synthetic SFT dataset intended to align a base model with substantially fewer examples by concentrating on expert-level tasks and rigorous response selection (Yin et al., 8 Oct 2025).

Within the PiKa suite, the distinction between the SFT and preference stages is explicit. PiKa-SFT provides the initial 30k instruction–response pairs for supervised fine-tuning, while PiKa preference data provides 30k instruction–(preferred, dispreferred) triples for preference optimization such as DPO after SFT initialization. The resulting workflow is not an RLHF pipeline in the canonical sense; the reported PiKa experiments use SFT and then optional DPO, and the paper states that the RLHF/RLAIF objective is not used in those experiments.

A common misunderstanding is to treat PiKa-SFT as merely a smaller synthetic instruction corpus. The paper instead frames it as an “expert-level” dataset: persona-driven, knowledge-intensive, and explicitly filtered for difficulty, safety, feasibility, and quality. This emphasis is central to the claim of sample efficiency.

2. Dataset composition and statistical profile

PiKa-SFT comprises 30k SFT pairs drawn from persona-driven generation across biology, engineering, medicine, law, and other specialized areas. The current version emphasizes expert-level general tasks and explicitly notes that it lacks math/code reasoning coverage, which is reserved for a planned future iteration. The synthesis pipeline yields single-turn instruction-to-response pairs; multi-turn distribution is not reported. Language coverage is not explicitly reported, although examples suggest English. The dataset does not claim explicit chain-of-thought annotations, and the exact JSON schema, role tags, and metadata fields are also not reported (Yin et al., 8 Oct 2025).

The reported quantitative profile highlights difficulty, feasibility, response quality, and length. PiKa-SFT has mean instruction difficulty 7.39, compared with 2.65 for Magpie-Pro; feasibility 8.98 versus 8.77; and quality 9.57 versus 9.01, using GPT-4o-based evaluation on a 1–10 scale. Average instruction length is approximately 424 tokens, while responses average more than 5,305 tokens. These statistics are used to support the claim that the dataset emphasizes deep, knowledge-intensive supervision rather than short, high-volume interactions.

Aspect	PiKa-SFT	Reported comparison
SFT size	30k pairs	Magpie about 300k SFT examples
Preference data	30k triples	Used post-SFT for DPO
Mean difficulty	7.39	Magpie-Pro 2.65
Feasibility	8.98	Magpie-Pro 8.77
Quality	9.57	Magpie-Pro 9.01
Avg. instruction length	~424 tokens	Longer, more knowledge-intensive
Avg. response length	>5,305 tokens	Longer, more detailed

Diversity is measured by minimum neighbor distance in embedding space using all-mpnet-base-v2 embeddings and FAISS:

$$\mathrm{MND}(I_i)=\min_{j\neq i}\|e(I_i)-e(I_j)\|_2.$$

The paper reports mean MND $=0.497$ for PiKa and $=0.598$ for Magpie-Pro, while noting that larger MND implies lower redundancy and characterizing PiKa as attaining comparable diversity while avoiding repetition issues highlighted in prior datasets. This suggests that the authors regard redundancy control as a quality objective alongside difficulty and response depth.

3. Synthesis pipeline and quality control

PiKa-SFT is generated through a three-step pipeline. First, the system samples complex personas $\{\pi_i\}$ from PersonaHub across diverse expert domains. GPT-4o then generates a knowledge-intensive instruction for each persona,

$I_i=\mathrm{LLM}(\pi_i),$

and a quality-control gate $Q(I_i)\in\{0,1\}$ retains only instructions judged challenging, safe, and informative. Harmful or policy-violating instructions are removed at this stage, and the paper states that the content is synthesized and filtered to avoid unsafe or sensitive pairs (Yin et al., 8 Oct 2025).

Second, for each validated instruction, the pipeline generates $k$ candidate responses under mild stochasticity with $T<1$:

$R_i=\{r_{i1},\ldots,r_{ik}\}=\mathrm{LLM}(I_i;T).$

This multi-path generation stage is used to create a candidate set rather than committing to a single model output. The dataset’s final responses are therefore selected, not merely sampled.

Third, the candidate responses are scored by the reward model Skywork-Reward-V2-Llama-3.1-8B:

$s_{ij}=\mathcal{R}(I_i,r_{ij}).$

For SFT, the highest-scoring response is retained,

$=0.497$0

For preference optimization, the pipeline keeps the best and worst candidates,

$=0.497$1

Additional GPT-4o assessment functions score difficulty, feasibility, and pair quality:

$=0.497$2

These scores are used for analysis and can guide later selection or curation.

An important boundary condition is that the paper does not report explicit refusal policies or model-specific safety filters such as Llama Guard. Safety is instead described as arising from synthesis-time filtering through the $=0.497$3 gate and from the removal of unsafe or policy-violating instructions.

4. Training recipe and optimization objectives

The evaluation recipe uses Llama-3-8B-Base and the Qwen2.5 series at 0.5B, 1.5B, 3B, and 7B. SFT is described as standard supervised fine-tuning; whether the implementation is full-tuning or LoRA is not explicitly reported. The reported SFT learning rate is $=0.497$4 with a cosine schedule, the maximum sequence length is 8,192 tokens, and instruction templates are required to adhere to the official instruction templates of each respective model architecture. For DPO, the reported learning rate is $=0.497$5 with a cosine schedule (Yin et al., 8 Oct 2025).

The SFT objective is the standard token-level cross-entropy:

$=0.497$6

For preference learning, the paper reports the general DPO form

$=0.497$7

However, the DPO hyperparameters $=0.497$8 and $=0.497$9 are not reported. The paper also includes the canonical RLHF/RLAIF objective for completeness,

$=0.598$0

but explicitly notes that this objective is not used in the PiKa experiments.

Several implementation details remain unspecified. The paper does not report batch size, optimizer, weight decay, dropout, numeric precision, gradient checkpointing, tokenization specifics, sequence packing strategy, number of steps or tokens, compute resources, or training time. For technical readers, these omissions matter because PiKa-SFT responses are unusually long, and the use of an 8,192-token context window indicates that memory and packing strategy are likely operationally significant, even though the exact training stack is not documented.

5. Benchmark results and sample-efficiency claims

The central empirical result is that Llama-3-8B-Base fine-tuned on PiKa-SFT outperforms models trained on much larger public datasets in AlpacaEval 2.0 and Arena-Hard. On AlpacaEval 2.0 judged against GPT-4-Turbo (1106), PiKa-SFT achieves length-controlled win rate $=0.598$1, win rate $=0.598$2, and standard deviation $=0.598$3. The corresponding Magpie-Pro-300k model records LC $=0.598$4 and WR $=0.598$5, and Magpie-Air-300k records LC $=0.598$6 and WR $=0.598$7. Against Llama-3-8B-Instruct as the judge baseline, PiKa-SFT reaches LC $=0.598$8 and WR $=0.598$9, which the paper interprets as a preference over the official model. On Arena-Hard, PiKa-SFT reports WR $\{\pi_i\}$0, compared with $\{\pi_i\}$1 for Magpie-Pro-300k, $\{\pi_i\}$2 for Magpie-Air-300k, and $\{\pi_i\}$3 for Llama-3-8B-Instruct (Yin et al., 8 Oct 2025).

Setting	PiKa result	Reported comparison
AlpacaEval 2.0 vs GPT-4-Turbo	LC 32.82%, WR 30.56%	Magpie-Pro-300k: LC 24.06%, WR 28.60%
AlpacaEval 2.0 vs Llama-3-8B-Instruct	LC 52.42%, WR 50.30%	50% defines baseline
Arena-Hard	WR 33.5%	Llama-3-8B-Instruct: WR 24.5%

The same pattern continues after preference optimization. Using 30k SFT examples and 30k DPO triples, PiKa reports AlpacaEval 2.0 WR $\{\pi_i\}$4 and Arena-Hard WR $\{\pi_i\}$5. Magpie-Pro, with 300k SFT examples and 60k DPO examples, reports $\{\pi_i\}$6 and $\{\pi_i\}$7, while UltraFeedback, with 208k SFT examples and 64k DPO examples, reports $\{\pi_i\}$8 and $\{\pi_i\}$9. The paper therefore presents PiKa as more data-efficient than these alternatives under the reported judge-based evaluations.

Transfer experiments on Qwen2.5 from 0.5B to 7B show consistent gains over the official Instruct references. For example, at 1.5B, the official model scores LC $I_i=\mathrm{LLM}(\pi_i),$0 and WR $I_i=\mathrm{LLM}(\pi_i),$1 against GPT-4-Turbo, whereas PiKa-SFT reaches LC $I_i=\mathrm{LLM}(\pi_i),$2 and WR $I_i=\mathrm{LLM}(\pi_i),$3; against the official reference, PiKa records WR $I_i=\mathrm{LLM}(\pi_i),$4. At 7B, the official model scores LC $I_i=\mathrm{LLM}(\pi_i),$5 and WR $I_i=\mathrm{LLM}(\pi_i),$6, while PiKa-SFT reports LC $I_i=\mathrm{LLM}(\pi_i),$7 and WR $I_i=\mathrm{LLM}(\pi_i),$8, with WR $I_i=\mathrm{LLM}(\pi_i),$9 against the official reference. The paper notes that the largest relative gains occur on smaller models.

On additional HuggingFace Open LLM Leaderboard tasks, averaged over MMLU(5), ARC(25), HellaSwag(10), TruthfulQA(0), WinoGrande(5), and GSM8K(5), PiKa reaches $Q(I_i)\in\{0,1\}$0 on Llama-3-8B-Base with SFT, compared with $Q(I_i)\in\{0,1\}$1 for Llama-3-8B-Instruct. The paper specifically notes slightly lower math performance, for example on GSM8K, due to limited explicit math coverage in PiKa-SFT. This is important because the strongest claims for PiKa concern alignment benchmarks rather than uniform dominance across all knowledge and reasoning tasks.

The data-size ablation further sharpens the sample-efficiency claim. PiKa outperforms Magpie-Pro at every tested scale from 10k to 30k, and PiKa with only 10k examples already surpasses Magpie-Pro at 30k. On Arena-Hard, PiKa peaks at 30k. The paper interprets this as evidence that difficulty and verification matter more than raw size.

6. Limitations, safety, and relation to adjacent alignment methods

The paper identifies several limitations directly. The current version lacks math and code reasoning coverage. Schema details, role metadata, language coverage, multi-turn composition, versioning and dataset splits, and license type are not reported. The examples suggest English, but the paper does not define language scope explicitly. These omissions constrain reproducibility and make it difficult to determine how PiKa-SFT should be integrated into multilingual, multi-turn, or tool-augmented post-training pipelines (Yin et al., 8 Oct 2025).

Safety treatment is synthesis-centric rather than policy-centric. Harmful or policy-violating instructions are removed during filtering, and the paper states that the resulting dataset avoids unsafe, harmful, or sensitive content. At the same time, no explicit refusal policy or separate safety model is reported. A plausible implication is that PiKa-SFT should be understood as a filtered instruction corpus rather than a comprehensive safety alignment framework.

Another potential misconception is that PiKa-SFT provides explicit chain-of-thought supervision because its responses are long. The paper does not report explicit chain-of-thought annotations; it only reports that responses are long and detailed. Similarly, PiKa-SFT should not be described as a multi-turn chat dataset, because the synthesis pipeline yields single-turn instruction-to-response pairs and the multi-turn distribution is not reported.

PiKa-SFT also sits near a separate line of work in which reward learning is incorporated directly into the SFT stage. “Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment” proposes IRL-based SFT methods, including RFT and IRFT, that learn from demonstrations using KL regularization to a reference policy and self-generated negatives, and proves convergence to stationary solutions of the ML-IRL objective under stated assumptions (Li et al., 2024). PiKa does not use those methods: its reported recipe is standard SFT followed optionally by DPO, with reward models used for data selection rather than for an IRFT- or RFT-style training objective. This suggests a possible extension point rather than a current feature of PiKa-SFT.

In that sense, PiKa-SFT occupies a specific methodological niche. Its contribution is not a new optimization rule for SFT, but a synthetic data construction strategy in which persona-driven instruction generation, reward-model-guided response selection, GPT-4o-based scoring, and long-form expert-level supervision are combined to improve alignment with only 30k SFT pairs. The paper’s future-work directions—adding math/code domains, optimizing selection for further size reduction, and studying mixture strategies—follow directly from the observed strengths and current coverage gaps.