Choose-Then-Sample: Decision-Coupled Sampling

• Choose-then-sample is a unified framework where a pre-sampling decision conditions the sampling process, enabling immediate, local, and irreversible selection.
• It underpins methodologies from unequal-probability streaming to threshold rules and data curation, ensuring efficient real-time decisions under varied constraints.
• The approach leverages state compression and localized decision rules to match target designs without exhaustive retrospective comparisons, improving practical scalability.

Choose-then-sample denotes a family of procedures in which a selection decision is made before, during, or as part of sampling, rather than after a full batch has been retained and reconsidered. Across the literature, the phrase itself or a closely related interpretation appears in several non-identical forms: immediate accept/reject decisions for streaming unequal-probability sampling, pre-training selection of augmented or candidate data before learning, randomized score-based winner selection under feasibility constraints, and pre-randomization enrollment design in experiments (Panahbehagh et al., 2021, Lin et al., 2023, Dughmi et al., 2018, Hu et al., 2024). Taken together, these works suggest that choose-then-sample is best understood as an umbrella pattern for decision-coupled sampling rather than as a single canonical algorithm.

1. Conceptual scope and recurring formal structure

A common structure in this literature is that the sampling rule is conditioned on a prior or intermediate choice object. In streaming survey sampling, the current unit is observed and its fate is decided immediately from the current cumulative state, with no access to future items. In text augmentation and data curation, candidate examples are filtered or compared before model training. In winner-selection problems, random scores are generated first and the feasible winner set is then chosen by maximizing total realized score. In experimental design, the experimenter first chooses whom to enroll from each subgroup and only then randomizes treatment within the enrolled sample (Panahbehagh et al., 2021, Lin et al., 2023, Dughmi et al., 2018, Hu et al., 2024).

These formulations differ in what is being “chosen.” The choice may be an online inclusion decision, a subset of training data, a score distribution over types, or a vector of subgroup sample sizes. The “sample” stage likewise varies: it may mean the realized survey sample, the selected augmented corpus used for self-training, the winner set induced by dice rolls, or the enrolled trial cohort. This suggests that the unifying feature is not a single probability law, but the placement of a selection operator before or inside the sampling mechanism.

A second recurring feature is state compression. Immediate-decision streaming methods maintain a small number of running quantities; threshold rules depend only on the number already selected; consistent sampling orders items by pseudorandom tickets; greedy LLM data curation uses local windows rather than full traversal; and minimax-regret enrollment compresses downstream welfare into a design objective. In each case, the method avoids exhaustive retrospective comparison over a full stored population.

2. Immediate-decision streaming and unequal-probability sampling

The clearest technical realization of choose-then-sample appears in unequal-probability stream sampling. In "Stream Sampling with Immediate Decision," the population is $U=\{1,2,\dots,N\}$ with prescribed first-order inclusion probabilities

$$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$

and the objective is to construct a sample $S$ such that

$\Pr(k\in S)=\pi_k.$

The method is a stream-sampling implementation of Deville’s systematic or ordered pivotal sampling design, but reformulated so that each arriving unit is selected or rejected immediately after observation. The construction uses cumulative sums $F_j=\sum_{k=1}^j \pi_k$, cross-border units where the cumulative mass crosses an integer boundary, and a one-window logic that allows one unit to be decided as soon as the current window reaches total mass $1$. Result 3 states that IDS and Deville’s method generate the same design,

$P_r(S=s)=P_D(S=s),$

so the method is not an approximation but an equivalent online implementation of that unequal-probability design (Panahbehagh et al., 2021).

The related OSOD method—One-Step One-Decision sampling—pushes the same principle into a sequential update rule for stream populations. At step $t$, the current unit is selected with probability equal to its current updated inclusion probability,

$$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$

Later inclusion probabilities are then adjusted by a proportional push or pull so that the total expected sample size is preserved. The stream version uses the smallest possible sliding window and can add a phantom unit when a local integer-sum condition fails. The paper explicitly links smaller windows to stronger spread along the ordered stream and presents the window length $m$ as a direct control on dispersion (Panahbehagh et al., 2021).

Chromy’s sequential unequal-probability procedure provides a third one-pass formulation. It scans the frame once and conditions the decision for unit $$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$0 on the number already selected. A key structural property is

$$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$1

so the realized cumulative sample size tracks the target cumulative inclusion mass up to rounding. The paper further proves that ordered pivotal sampling and Chromy sampling with the same $$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$2 induce the same sampling design, transferring to Chromy sampling properties such as asymptotic normality of the Horvitz–Thompson estimator and explicit second-order inclusion probabilities. For the randomized circular-shift version used by SAS SURVEYSELECT, positive second-order probabilities allow unbiased Sen–Yates–Grundy variance estimation (Chauvet, 2019).

In this stream-sampling lineage, choose-then-sample has a precise meaning: selection is irrevocable, local, and online, yet the target first-order design is preserved exactly.

3. Threshold, ticket, and cost-based online selection

A second line of work studies choose-then-sample as online acceptance from a stream ordered by quality, threshold, or pseudorandom key. "Threshold rules for online sample selection" formalizes the problem as irrevocable acceptance or rejection of each arriving item so as to keep the eventual retained sample close to an offline optimum. A threshold rule is specified by a nested family of acceptance sets, and in the one-dimensional case the paper studies

$$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$3

Under the paper’s assumptions, such oblivious rules achieve constant-factor or asymptotic $$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$4 guarantees, depending on the model class. Here the “choice” is the immediate threshold test, and the “sample” is the cumulative retained subsequence (Bach et al., 2010).

"Consistent Sampling with Replacement" replaces thresholds with a global pseudorandom order. Each item $$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$5 receives a ticket number $$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$6, and the current sample step chooses the item with the minimum ticket. With replacement, that chosen item is returned to the pool with a new larger ticket

$$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$7

distributed as if uniform on $$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$8. The result is a simple random sample with replacement that is consistent with respect to both sample size and population. This is a particularly literal choose-then-sample mechanism: choose the minimum-key item, then sample again by reinserting it with an updated key (Rivest, 2018).

"Sampling with Costs" studies the meta-decision of how many items to sample when each sample incurs cost. If $$\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,$$9 is the expected worth of the best among $S$0 samples and $S$1 is total sampling cost, the optimal sample size is

$S$2

For constant marginal cost $S$3, the stopping rule is

$S$4

The paper also analyzes measurement error, showing that the expected value of the measured-best item is reduced by a signal-to-noise factor $S$5. In this setting, choose-then-sample operates at the policy level: the decision-maker first chooses a search depth, then samples accordingly (Skufca et al., 2015).

4. Data curation, robust learning, and subset-aware training

In machine learning, choose-then-sample typically means that candidate data are not consumed indiscriminately. "How to choose 'Good' Samples for Text Data Augmentation" proposes a self-training framework with four modules: self-training, encoder, sample selection, and sample recall. A teacher RoBERTa is trained on the original labeled corpus $S$6, pseudo-labels are generated for the augmented corpus $S$7, and a two-stage selector is applied before student training on $S$8. The first selector keeps augmented samples whose predicted label matches the attached ground-truth label and then ranks them by entropy under MC dropout,

$S$9

The second selector recalls discarded samples using Jaccard overlap and perplexity-derived semantic fluency. On SST-5, the reported accuracies are $\Pr(k\in S)=\pi_k.$0 for RoBERTa-base, $\Pr(k\in S)=\pi_k.$1 for the method with EDA, and $\Pr(k\in S)=\pi_k.$2 for the method with nlpaug; the ablation reports $\Pr(k\in S)=\pi_k.$3, $\Pr(k\in S)=\pi_k.$4, $\Pr(k\in S)=\pi_k.$5, and $\Pr(k\in S)=\pi_k.$6 across baseline, augmentation, entropy-based selection, and recall stages, respectively (Lin et al., 2023).

"Add-One-In: Incremental Sample Selection for LLMs via a Choice-Based Greedy Paradigm" shifts from pointwise scoring to subset-aware marginal gain. With current selected set $\Pr(k\in S)=\pi_k.$7 and remaining pool $\Pr(k\in S)=\pi_k.$8, the greedy choice is

$\Pr(k\in S)=\pi_k.$9

The implementation samples local windows $F_j=\sum_{k=1}^j \pi_k$0 and $F_j=\sum_{k=1}^j \pi_k$1, prompts an LLM to choose the best addition, and repeats until $F_j=\sum_{k=1}^j \pi_k$2. The paper reports that with $F_j=\sum_{k=1}^j \pi_k$3K samples, the selected subset can significantly outperform training on all $F_j=\sum_{k=1}^j \pi_k$4K Alpaca samples; with $F_j=\sum_{k=1}^j \pi_k$5 of the data, the method beats full-data training on $F_j=\sum_{k=1}^j \pi_k$6 out of $F_j=\sum_{k=1}^j \pi_k$7 test sets; with $F_j=\sum_{k=1}^j \pi_k$8, it achieves an average win score of $F_j=\sum_{k=1}^j \pi_k$9. For top-$1$0 selection on Alpaca, the reported practical cost is $1$1 minutes versus $1$2 for AlpaGasus and $1$3 for IFD (Li et al., 4 Mar 2025).

A more algorithmic optimization-time version appears in "Choosing the Sample with Lowest Loss makes SGD Robust." At iteration $1$4, the method first chooses a set $1$5 of $1$6 samples, then selects

$1$7

and updates

$1$8

Vanilla SGD is the case $1$9. The paper shows that this induces a surrogate gradient field that weights lower-loss samples more heavily and derives robustness guarantees in quadratic outlier models. Empirically, MKL-SGD is reported to outperform vanilla SGD under outliers in linear regression and under label corruption in small neural-network experiments (Shah et al., 2020).

A related set-aware formulation appears in the "optimal choice" problem, where each training instance is a finite set of candidates—a lot—and exactly one candidate is the prime. The scoring function $P_r(S=s)=P_D(S=s),$0 succeeds on a lot if the prime is top-ranked. On the reported signal-processing dataset of $P_r(S=s)=P_D(S=s),$1 choices grouped into $P_r(S=s)=P_D(S=s),$2 lots, feature augmentation and direct optimization of lot-wise success rate yield reported success rates up to $P_r(S=s)=P_D(S=s),$3, underscoring that in choose-then-sample settings the relevant loss may be set-wise rather than point-wise (Sapir, 2017).

5. Winner selection, subset choice, and experimental enrollment

In mechanism design, choose-then-sample often takes the form “score-then-select.” "Alea Iacta Est: Auctions, Persuasion, Interim Rules, and Dice" introduces winner-selecting dice. Each type $P_r(S=s)=P_D(S=s),$4 is assigned a die $P_r(S=s)=P_D(S=s),$5; after types are realized, scores $P_r(S=s)=P_D(S=s),$6 are drawn independently; and the mechanism chooses a feasible winner set maximizing total realized score,

$P_r(S=s)=P_D(S=s),$7

For matroid feasibility constraints, every feasible first-order interim rule admits a winner-selecting dice implementation. For the $P_r(S=s)=P_D(S=s),$8-uniform matroid, the paper gives an efficient construction, and in i.i.d. symmetric single-winner environments it gives symmetric efficient constructions (Dughmi et al., 2018).

"Random Collection" studies stochastic choice correspondences over subsets rather than single winners. The primitive object is $P_r(S=s)=P_D(S=s),$9, the probability of choosing a non-empty subset $t$0. Several characterized models are explicitly two-stage. In Random Categorization,

$t$1

so a latent category is chosen and then intersected with the menu. Static EBA, RRM, and NSC have analogous choose-latent-structure-then-intersect forms, each identified by distinct axioms such as Relative Additivity, IIS, PIIS, and model-specific positivity conditions. This literature is the closest formal analogue of choose-then-sample for random subset choice (Vu, 23 Nov 2025).

In experimental design, the same logic governs enrollment rather than within-menu winners. "Minimax-Regret Sample Selection in Randomized Experiments" considers a heterogeneous population partitioned into $t$2 groups and requires the experimenter to choose

$t$3

before observing data. Welfare under decision vector $t$4 is

$t$5

Under Gaussian assumptions and separate group decisions, the worst-case regret satisfies

$t$6

with $t$7, yielding the relaxed optimum

$t$8

If only a single pooled decision is allowed, the minimax-regret design becomes proportional, $t$9; under egalitarian welfare, the relaxed optimum is variance-proportional. In the vaccine-trial example with $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$0, $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$1, and $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$2, the reported minimax allocation is around $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$3, compared with proportional $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$4 and egalitarian $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$5 (Hu et al., 2024).

6. Related variants, boundaries, and common misconceptions

A recurrent misconception is that all procedures involving sampling and selection are choose-then-sample in the same sense. Several adjacent literatures show that the boundary is more nuanced. "SAFER: Risk-Constrained Sample-then-Filter in LLMs" is explicitly framed as a two-stage risk control framework comprising abstention-aware sampling and conformalized filtering. It first calibrates the smallest feasible sampling budget

$$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$6

abstains if $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$7, and then calibrates a filtering threshold $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$8 using conformal risk control at level $$\pi_t^t= \begin{cases} 1 & \text{with probability } \pi_t^{t-1},\ 0 & \text{with probability } 1-\pi_t^{t-1}. \end{cases}$$9. The combined guarantee bounds final miscoverage by

$m$0

The paper itself characterizes the method as a risk-constrained sample-then-filter pipeline or a choose-budget-then-sample-then-filter variant, not as a pure choose-then-sample method over answers (Wang et al., 11 Oct 2025).

Conversely, some works are best described as sample-then-choose rather than choose-then-sample. "Lookahead and Hybrid Sample Allocation Procedures for Multiple Attribute Selection Decisions" studies a fixed budget $m$1 of attribute measurements, with posterior updating by Bayes’ rule and final choice deferred until after sampling. The decision-maker may maximize expected utility or the probability of selecting the truly best alternative, and hybrid policies uniformly allocate the first $m$2 samples before switching to one-step lookahead. The paper explicitly states that the decision-maker must allocate samples, collect them, update beliefs, and then select the alternative, so the final choice follows sampling rather than preceding it (Herrmann et al., 2020).

Other adjacent uses of the phrase emphasize latent elimination rather than direct online acceptance. "Consideration Set Sampling to Analyze Undecided Respondents" adopts a choose-then-sample view in which respondents first eliminate alternatives and retain a smaller consideration set, which the survey then samples as a set-valued response $m$3. The paper models these responses as categorical random sets and transfers multinomial regression, regularization, interpretable machine learning, and clustering to the set-valued outcome space. On the German pre-election poll example, the study uses $m$4 observations, of which about $m$5 are undecided and provide non-singleton sets (Kreiss et al., 2023).

Taken together, these boundary cases suggest two clarifications. First, choose-then-sample is not synonymous with any sequential or selective pipeline. Second, the decisive criterion is where the selection operator sits relative to uncertainty resolution: before sampling, inside streaming acceptance, over latent structures that generate subsets, or after a sampled candidate pool has already been produced.