Sample Engineering Methods

• Sample engineering is the process of deliberately designing data samples to enhance representativeness, fairness, and performance across multiple empirical domains.
• It employs controlled input, output, and reasoning strategies to optimize learning efficiency, improve generalization, and reduce errors in models like LLMs and RL agents.
• Empirical studies and mathematical formulations in diverse fields validate its role in maximizing data efficiency and achieving robust performance under limited data regimes.

Sample engineering is the discipline and methodological practice of deliberately designing, selecting, and configuring data samples to optimize the empirical, statistical, or algorithmic efficiency of learning, evaluation, or inspection processes. In state-of-the-art contexts—from LLMs to statistical quality control, seismic prediction, reinforcement learning, and software engineering—sample engineering governs not only representativeness and fairness, but also performance, generalization, and data efficiency. This article systematically presents contemporary approaches, mathematical underpinnings, empirical findings, and salient guidelines defining sample engineering across domains.

1. Foundational Principles and Scope

Sample engineering encompasses a spectrum of activities: controlled construction of fine-tuning datasets for LLMs (sample design engineering), crafting single maximally informative samples for one-shot RL, curating or partitioning datasets for statistical hypothesis testing, engineering inspection plans for industrial quality control, and optimizing sample selection for both data-driven and search-based algorithms.

A canonical example is Sample Design Engineering (SDE) for downstream LLM fine-tuning, defined as the systematic process of constructing and selecting fine-tuning examples for task adaptation by explicitly controlling orthogonal dimensions of input, output, and reasoning to maximize post-tuning performance under limited data regimes (Guo et al., 2024).

2. Engineering Dimensions and Strategies

Sample engineering operates by explicit manipulation of key design dimensions, each substantiated by controlled ablation and empirical benchmarking.

2.1 Input Design

SDE for LLMs empirically demonstrates that instruction placement (Inst-first vs Inst-last vs No-inst) and input modeling (inclusion vs exclusion of input tokens in the loss) are critical (Guo et al., 2024):

• No-inst significantly degrades both in-domain (ID) and out-of-domain (OOD) performance (Δκ ≈ –0.04 to –0.05).
• Inst-first is uniformly superior to Inst-last (Δκ = +0.01 to +0.02).
• Excluding the input tokens from loss (No-MI) is beneficial, whereas including them hurts κ by up to –0.1~–0.2.

2.2 Output Design

Sample formatting, handling of unmentioned targets, and label coding strongly impact parsing, adherence, and accuracy:

• Hierarchy for format adherence: JSON < Lines < Natural (error rates: <1%, 1–2%, up to 10% in OOD).
• Uniformly present outputs (placeholders, PU) outperform omissions (OU), with up to –0.23 penalty for rare aspects if omitted.
• Textual labels (TxtLabel) outclass numeric labels (NumLabel) with a Δκ of –0.03 to –0.05 in favor of TxtLabel.

2.3 Reasoning Design

Chain-of-Thought (CoT) and Reverse-CoT (R-CoT) can be included or omitted:

• CoT yields negligible improvement in ID (Δκ ≈ 0) but can raise OOD κ up to +0.04; however, it increases parsing errors by 2–4%.

2.4 Sample Selection and Statistical Power

For hypothesis-driven software engineering studies, sample engineering via platforms like Prolific integrates prescreening, pilot testing, power analysis, and dynamic post-hoc filtering to ensure representativeness and statistical validity (Russo, 2022).

3. Mathematical Formulations

Sample engineering is underpinned by a set of canonical statistical and information-theoretic quantities.

3.1 Performance Metrics in LLM SDE

• Weighted Cohen’s Kappa for MASA:

$\kappa = \frac{P_o - P_e}{1 - P_e},$

where $P_o = \sum_{i,j=1}^R w_{ij}p_{ij}$ and $P_e = \sum_{i,j=1}^R w_{ij}p_{i\cdot}p_{\cdot j}$.

• Perplexity (PPL) as $\mathrm{PPL} = \exp(\mathcal{L})$, with $\mathcal{L}$ the mean cross-entropy loss over tokens.

3.2 Power and Sample Size in Empirical Software Engineering

• A priori sample calculation for two-sided mean tests:

$$n = \left(Z_{1-\alpha/2} + Z_{1-\beta}\right)^2 \frac{\sigma^2}{\Delta^2}$$

3.3 Sampling in Inspection and Control Plans

• Acceptance-sampling statistics based on stagewise $t$-type statistics and OC curve approximations, including adjustments for panel (paired) or spatial-batch sampling (Steland, 2014).

Domain	Core Metric	Governs Trade-off
LLM Fine-tuning	Weighted Kappa, PPL	Generalization, parsing errors
ML Seismic Prediction	Acc, F1, Entropy	Sample size, class balance, OSS
Software Eng. Cohorts	Statistical Power (n, σ, Δ, α)	Representativeness, Type I/II
RL Reward Shaping	Regret Bounds, Effective State #	Sample efficiency, optimism
Control Inspection	OC Curve, Quantile Estimation	Producer/consumer risk, σ, ρ

4. Empirical Findings Across Domains

4.1 LLM Sample Design Engineering

An integrated SDE (ES-SDE) recipe—Inst-first, No-MI, Lines, PU, TxtLabel, No-CoT—outperforms both weak SDE and heuristic baselines in multi-aspect sentiment, event extraction, and nested NER with margins equivalent to doubling or tripling data, and is robust to randomization and instruction rephrasing (Guo et al., 2024).

Training size	Strategy	GENIA F1	Review11 κ	acc
500	Heuristic	0.5747	0.588	.7586
	EW-SDE	0.5432	0.7235	.8327
	ES-SDE	0.6141	0.7691	.8626
1000	Heuristic	0.6228	0.7058	.8262
	EW-SDE	0.5517	0.7565	.8502
	ES-SDE	0.6895	0.7892	.8716

4.2 Extreme Data Efficiency in RL: One-shot Learning

Polymath learning demonstrates that a single engineered “Synthetic Prime” sample with maximal Skill_Count can elicit broad improvements across math, physics, chemistry, and biology, matching or exceeding results from 8K-shot RL or 1K-shot LIMR RL (Li et al., 6 Jan 2026).

• Synthetic Prime outperforms best natural example by ~3 points average.
• Cross-domain average pass rates: Math 38.3%, Physics 20.6%, Chemistry 15.7%, Biology 54.2%.

4.3 ML Sampling Strategy Engineering

In seismic liquefaction prediction, ordered systematic sampling (OSS) consistently yields the highest test Acc and F1 across seven ML models (Hu et al., 11 Dec 2025). Optimal test performance is achieved with a training set size of 200 (out of 250), 80:20 train/test split, and class ratio 1:1–1.5:1.

Influence of key factors (interaction study):

• Split ratio: 44.1% (largest)
• Class distribution: 28.7%
• Sample size: 27.2%

5. Evolutionary, Algorithmic, and Statistical Sampling

5.1 Sample-efficient Generation and Search

In test-time code generation for software engineering, evolutionary test-time scaling (EvoScale) aligns generation with few-sample high-reward regions by iteratively mutating and selecting outputs; sample efficiency is further increased when the policy is RL-trained for self-evolution (Zeng et al., 29 May 2025). This enables a 32B parameter LM to match/exceed 100B+ baselines with only Best@10–50 sampling (16.6s runtime vs 92.8s for unit tests).

5.2 Sampling for Multi-objective Search and Inspection

The SWAY algorithm (Chen et al., 2016) illustrates sample-based search for multi-objective optimization: a large population is recursively clustered, representatives evaluated, and only dominant clusters retained, leading to Pareto-optimal tradeoffs with drastically fewer evaluations compared to evolutionary algorithms (1–8% time and 0–3% model runs).

In acceptance sampling, multi-stage (control + inspection) plans with independent, dependent (panel), or spatial batch sampling provide statistically-controlled risk and sensitivity to quality, with all plan parameters derived explicitly via asymptotic normal expansions and historic distribution quantile estimation (Steland, 2014).

6. Theoretical Insights and Provable Sample Efficiency

Reward shaping in RL is shown to act as sample engineering by pruning non-optimal state regions via shaped bonuses and value clipping (Gupta et al., 2022). Main theorem:

$$\mathrm{Regret}(T) = O\left( H\beta\sqrt{|S\setminus PathPseudoSub_\Delta||A|T\ln\frac{|S||A|T}{\delta}} + \frac{\beta^2\max V^2|BoundaryPseudoSub_\Delta|\ln\frac{|S||A|T}{\delta}}{\Delta^2} \right)$$

Effective sample complexity is governed by pruned state/action counts rather than raw $|S|,H$, with empirical improvement in structured navigation (e.g., “double corridor” tasks).

A plausible implication is that sample engineering—through shaping, selection, or formatting—can dramatically accelerate convergence and generalization by concentrating empirical or exploratory budget on high-value or generalizable regions of the domain.

7. Practical Guidelines

• Prioritize input clarity and explicit task decomposition (Inst-first, No-MI, Lines, PU, TxtLabel, No-CoT for LLMs).
• Engineer polymath-like examples embedding abstraction, compositionality, and cross-domain symbolism to maximize multi-domain transfer.
• Use ordered systematic sampling for small- to medium-sized ML datasets to best balance feature and label distributions.
• Leverage staged statistical plans, dynamic prescreening, and demographic validation for empirical studies.
• In search- or optimization-driven spaces, apply sample-based recursive selection (e.g., SWAY) for baseline benchmarking and rapid front approximation.
• In RL, employ value and bonus shaping, with theoretical calibration, to concentrate learning on effective subspaces.
• Always validate sample impact empirically under full evaluation, as preferences in zero-shot, prompt, or synthetic perplexity may not correlate with downstream gains (Guo et al., 2024).

This paradigm shift toward sample-centric methodology is substantiated across LLMs (Guo et al., 2024, Li et al., 6 Jan 2026), ML (Hu et al., 11 Dec 2025), RL (Gupta et al., 2022), empirical SE (Russo, 2022), and industrial quality control (Steland, 2014), establishing sample engineering as a central practice for empirical performance maximization under data budget constraints.