---
title: 'Choose-Then-Sample: Decision-Coupled Sampling'
url: https://www.emergentmind.com/topics/choose-then-sample
type: topic
---

# Choose-Then-Sample: Decision-Coupled Sampling

Choose-then-sample denotes a family of procedures in which a selection decision is made before, during, or as part of sampling, rather than after a full batch has been retained and reconsidered. Across the literature, the phrase itself or a closely related interpretation appears in several non-identical forms: immediate accept/reject decisions for streaming unequal-probability sampling, pre-training selection of augmented or candidate data before learning, randomized score-based winner selection under feasibility constraints, and pre-randomization enrollment design in experiments [2111.09309] [2302.00894] [1811.11417] [2403.01386]. Taken together, these works suggest that choose-then-sample is best understood as an umbrella pattern for decision-coupled sampling rather than as a single canonical algorithm.

## 1. Conceptual scope and recurring formal structure

A common structure in this literature is that the sampling rule is conditioned on a prior or intermediate choice object. In streaming survey sampling, the current unit is observed and its fate is decided immediately from the current cumulative state, with no access to future items. In text augmentation and data curation, candidate examples are filtered or compared before model training. In winner-selection problems, random scores are generated first and the feasible winner set is then chosen by maximizing total realized score. In experimental design, the experimenter first chooses whom to enroll from each subgroup and only then randomizes treatment within the enrolled sample [2111.09309] [2302.00894] [1811.11417] [2403.01386].

These formulations differ in what is being “chosen.” The choice may be an online inclusion decision, a subset of training data, a score distribution over types, or a vector of subgroup sample sizes. The “sample” stage likewise varies: it may mean the realized survey sample, the selected augmented corpus used for self-training, the winner set induced by dice rolls, or the enrolled trial cohort. This suggests that the unifying feature is not a single probability law, but the placement of a selection operator before or inside the sampling mechanism.

A second recurring feature is state compression. Immediate-decision streaming methods maintain a small number of running quantities; threshold rules depend only on the number already selected; consistent sampling orders items by pseudorandom tickets; greedy LLM data curation uses local windows rather than full traversal; and minimax-regret enrollment compresses downstream welfare into a design objective. In each case, the method avoids exhaustive retrospective comparison over a full stored population.

## 2. Immediate-decision streaming and unequal-probability sampling

The clearest technical realization of choose-then-sample appears in unequal-probability stream sampling. In "Stream Sampling with Immediate Decision," the population is \(U=\{1,2,\dots,N\}\) with prescribed first-order inclusion probabilities
\[
\pi = (\pi_1,\pi_2,\dots,\pi_N), \qquad 0<\pi_k\le 1,
\]
and the objective is to construct a sample \(S\) such that
\[
\Pr(k\in S)=\pi_k.
\]
The method is a stream-sampling implementation of Deville’s systematic or ordered pivotal sampling design, but reformulated so that each arriving unit is selected or rejected immediately after observation. The construction uses cumulative sums \(F_j=\sum_{k=1}^j \pi_k\), cross-border units where the cumulative mass crosses an integer boundary, and a one-window logic that allows one unit to be decided as soon as the current window reaches total mass \(1\). Result 3 states that IDS and Deville’s method generate the same design,
\[
P_r(S=s)=P_D(S=s),
\]
so the method is not an approximation but an equivalent online implementation of that unequal-probability design [2111.09309].

The related OSOD method—One-Step One-Decision sampling—pushes the same principle into a sequential update rule for stream populations. At step \(t\), the current unit is selected with probability equal to its current updated inclusion probability,
\[
\pi_t^t=
\begin{cases}
1 & \text{with probability } \pi_t^{t-1},\\
0 & \text{with probability } 1-\pi_t^{t-1}.
\end{cases}
\]
Later inclusion probabilities are then adjusted by a proportional push or pull so that the total expected sample size is preserved. The stream version uses the smallest possible sliding window and can add a phantom unit when a local integer-sum condition fails. The paper explicitly links smaller windows to stronger spread along the ordered stream and presents the window length \(m\) as a direct control on dispersion [2111.08433].

Chromy’s sequential unequal-probability procedure provides a third one-pass formulation. It scans the frame once and conditions the decision for unit \(k\) on the number already selected. A key structural property is
\[
V_k^I\le \sum_{l=1}^k I_l\le V_k^I+1,
\]
so the realized cumulative sample size tracks the target cumulative inclusion mass up to rounding. The paper further proves that ordered pivotal sampling and Chromy sampling with the same \(\pi\) induce the same sampling design, transferring to Chromy sampling properties such as asymptotic normality of the Horvitz–Thompson estimator and explicit second-order inclusion probabilities. For the randomized circular-shift version used by SAS `SURVEYSELECT`, positive second-order probabilities allow unbiased Sen–Yates–Grundy variance estimation [1912.10896].

In this stream-sampling lineage, choose-then-sample has a precise meaning: selection is irrevocable, local, and online, yet the target first-order design is preserved exactly.

## 3. Threshold, ticket, and cost-based online selection

A second line of work studies choose-then-sample as online acceptance from a stream ordered by quality, threshold, or pseudorandom key. "Threshold rules for online sample selection" formalizes the problem as irrevocable acceptance or rejection of each arriving item so as to keep the eventual retained sample close to an offline optimum. A threshold rule is specified by a nested family of acceptance sets, and in the one-dimensional case the paper studies
\[
\_i = \{x : x \ge 1 - c_i\}, \qquad c_i = \frac{1}{i^\alpha}, \quad 0 \le \alpha < 1.
\]
Under the paper’s assumptions, such oblivious rules achieve constant-factor or asymptotic \(1+o(1)\) guarantees, depending on the model class. Here the “choice” is the immediate threshold test, and the “sample” is the cumulative retained subsequence [1002.5034].

"Consistent Sampling with Replacement" replaces thresholds with a global pseudorandom order. Each item \(i\) receives a ticket number \(\tau_{i,1}=f(i,u)\), and the current sample step chooses the item with the minimum ticket. With replacement, that chosen item is returned to the pool with a new larger ticket
\[
\tau_{i,j+1}=g(\tau_{i,j}),
\]
distributed as if uniform on \((\tau_{i,j},1)\). The result is a simple random sample with replacement that is consistent with respect to both sample size and population. This is a particularly literal choose-then-sample mechanism: choose the minimum-key item, then sample again by reinserting it with an updated key [1808.10016].

"Sampling with Costs" studies the meta-decision of how many items to sample when each sample incurs cost. If \(K_n=\mathbb{E}[A_{\max}(n)]\) is the expected worth of the best among \(n\) samples and \(C_n\) is total sampling cost, the optimal sample size is
\[
n^*=\arg\max_n (K_n-C_n).
\]
For constant marginal cost \(c\), the stopping rule is
\[
k_{n^*}>c\ge k_{n^*+1},\qquad k_n:=K_n-K_{n-1}.
\]
The paper also analyzes measurement error, showing that the expected value of the measured-best item is reduced by a signal-to-noise factor \(\eta=\frac{a}{\sqrt{a^2+b^2}}\). In this setting, choose-then-sample operates at the policy level: the decision-maker first chooses a search depth, then samples accordingly [1506.04599].

## 4. Data curation, robust learning, and subset-aware training

In machine learning, choose-then-sample typically means that candidate data are not consumed indiscriminately. "How to choose 'Good' Samples for Text Data Augmentation" proposes a self-training framework with four modules: self-training, encoder, sample selection, and sample recall. A teacher RoBERTa is trained on the original labeled corpus \(D_i\), pseudo-labels are generated for the augmented corpus \(D_g\), and a two-stage selector is applied before student training on \(D_i\cup S_g\). The first selector keeps augmented samples whose predicted label matches the attached ground-truth label and then ranks them by entropy under MC dropout,
\[
H(p_i) = - \sum_{c=1}^{C} p_i^c \log p_i^c.
\]
The second selector recalls discarded samples using Jaccard overlap and perplexity-derived semantic fluency. On SST-5, the reported accuracies are \(54.2\) for RoBERTa-base, \(56.5\) for the method with EDA, and \(56.0\) for the method with nlpaug; the ablation reports \(54.2\), \(54.4\), \(55.5\), and \(56.6\) across baseline, augmentation, entropy-based selection, and recall stages, respectively [2302.00894].

"Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm" shifts from pointwise scoring to subset-aware marginal gain. With current selected set \(\mathcal{A}\) and remaining pool \(\mathcal{B}\), the greedy choice is
\[
x^*=\arg\max_{x_i\in \mathcal{B}} \left[F(\mathcal{A}\cup\{x_i\})-F(\mathcal{A})\right].
\]
The implementation samples local windows \(\mathcal{A}'\subseteq\mathcal{A}\) and \(\mathcal{B}'\subseteq\mathcal{B}\), prompts an LLM to choose the best addition, and repeats until \(|\mathcal{A}|=K\). The paper reports that with \(9\)K samples, the selected subset can significantly outperform training on all \(52\)K Alpaca samples; with \(5\%\) of the data, the method beats full-data training on \(3\) out of \(4\) test sets; with \(15\%\), it achieves an average win score of \(1.257\). For top-\(10\%\) selection on Alpaca, the reported practical cost is \(118.83\) minutes versus \(2466.80\) for AlpaGasus and \(176.80\) for IFD [2503.02359].

A more algorithmic optimization-time version appears in "Choosing the Sample with Lowest Loss makes SGD Robust." At iteration \(t\), the method first chooses a set \(S_t\) of \(k\) samples, then selects
\[
i_t = \arg\min_{i\in S_t} f_i(w_t),
\]
and updates
\[
w_{t+1}=w_t-\eta_t \nabla f_{i_t}(w_t).
\]
Vanilla SGD is the case \(k=1\). The paper shows that this induces a surrogate gradient field that weights lower-loss samples more heavily and derives robustness guarantees in quadratic outlier models. Empirically, MKL-SGD is reported to outperform vanilla SGD under outliers in linear regression and under label corruption in small neural-network experiments [2001.03316].

A related set-aware formulation appears in the "optimal choice" problem, where each training instance is a finite set of candidates—a lot—and exactly one candidate is the prime. The scoring function \(g(x,X)\) succeeds on a lot if the prime is top-ranked. On the reported signal-processing dataset of \(2453\) choices grouped into \(114\) lots, feature augmentation and direct optimization of lot-wise success rate yield reported success rates up to \(0.88\), underscoring that in choose-then-sample settings the relevant loss may be set-wise rather than point-wise [1706.08439].

## 5. Winner selection, subset choice, and experimental enrollment

In mechanism design, choose-then-sample often takes the form “score-then-select.” "Alea Iacta Est: Auctions, Persuasion, Interim Rules, and Dice" introduces winner-selecting dice. Each type \(t\) is assigned a die \(D_t\); after types are realized, scores \(v_i\sim D_{t_i}\) are drawn independently; and the mechanism chooses a feasible winner set maximizing total realized score,
\[
W(\mathbf{t},\mathbf{v}) \in \arg\max_{S \in I} \sum_{i \in S} v_i.
\]
For matroid feasibility constraints, every feasible first-order interim rule admits a winner-selecting dice implementation. For the \(1\)-uniform matroid, the paper gives an efficient construction, and in i.i.d. symmetric single-winner environments it gives symmetric efficient constructions [1811.11417].

"Random Collection" studies stochastic choice correspondences over subsets rather than single winners. The primitive object is \(\mu(T,S)\), the probability of choosing a non-empty subset \(T\subseteq S\). Several characterized models are explicitly two-stage. In Random Categorization,
\[
\mu_{RCG}(T,S) = \frac{\sum_C m(C)\,\mathbf{1}(T=C\cap S)} {\sum_{C:\,S\cap C\ne\emptyset} m(C)},
\]
so a latent category is chosen and then intersected with the menu. Static EBA, RRM, and NSC have analogous choose-latent-structure-then-intersect forms, each identified by distinct axioms such as Relative Additivity, IIS, PIIS, and model-specific positivity conditions. This literature is the closest formal analogue of choose-then-sample for random subset choice [2511.18476].

In experimental design, the same logic governs enrollment rather than within-menu winners. "Minimax-Regret Sample Selection in Randomized Experiments" considers a heterogeneous population partitioned into \(G\) groups and requires the experimenter to choose
\[
\bfn=(n_1,\dots,n_G)^\top,\qquad \sum_{g=1}^G n_g \le N,\qquad n_g\in 2\mathbb Z_{\ge 0},
\]
before observing data. Welfare under decision vector \(\delta\) is
\[
U(\alpha,\tau,\delta)=\sum_{g=1}^G \alpha_g\,\tau_g\,\delta_g.
\]
Under Gaussian assumptions and separate group decisions, the worst-case regret satisfies
\[
H(\bfn)=C_0 \sum_{g=1}^G \alpha_g \sqrt{\frac{2(s_{0,g}^2+s_{1,g}^2)}{n_g}},
\]
with \(C_0\approx 0.17\), yielding the relaxed optimum
\[
\widetilde n_g = \frac{(s_{0,g}^2+s_{1,g}^2)^{1/3}\alpha_g^{2/3}N}
{\sum_{g'=1}^G (s_{0,g'}^2+s_{1,g'}^2)^{1/3}\alpha_{g'}^{2/3}}.
\]
If only a single pooled decision is allowed, the minimax-regret design becomes proportional, \(n_g^\dag=\alpha_g N\); under egalitarian welfare, the relaxed optimum is variance-proportional. In the vaccine-trial example with \(\alpha_1=0.83\), \(\alpha_2=0.17\), and \(N\approx 9320\), the reported minimax allocation is around \((6100,3218)\), compared with proportional \((7734,1584)\) and egalitarian \((2068,7250)\) [2403.01386].

## 6. Related variants, boundaries, and common misconceptions

A recurrent misconception is that all procedures involving sampling and selection are choose-then-sample in the same sense. Several adjacent literatures show that the boundary is more nuanced. "SAFER: Risk-Constrained Sample-then-Filter in Large Language Models" is explicitly framed as a two-stage risk control framework comprising abstention-aware sampling and conformalized filtering. It first calibrates the smallest feasible sampling budget
\[
\hat{s}=\inf\Big\{s\in [1,M]:\hat{R}^{+}(s)\le \alpha\Big\},
\]
abstains if \(\hat{R}^{+}(M)>\alpha\), and then calibrates a filtering threshold \(\hat t\) using conformal risk control at level \(\beta\). The combined guarantee bounds final miscoverage by
\[
\alpha+\beta-\alpha\beta.
\]
The paper itself characterizes the method as a risk-constrained sample-then-filter pipeline or a choose-budget-then-sample-then-filter variant, not as a pure choose-then-sample method over answers [2510.10193].

Conversely, some works are best described as sample-then-choose rather than choose-then-sample. "Lookahead and Hybrid Sample Allocation Procedures for Multiple Attribute Selection Decisions" studies a fixed budget \(T\) of attribute measurements, with posterior updating by Bayes’ rule and final choice deferred until after sampling. The decision-maker may maximize expected utility or the probability of selecting the truly best alternative, and hybrid policies uniformly allocate the first \(H\) samples before switching to one-step lookahead. The paper explicitly states that the decision-maker must allocate samples, collect them, update beliefs, and then select the alternative, so the final choice follows sampling rather than preceding it [2007.16119].

Other adjacent uses of the phrase emphasize latent elimination rather than direct online acceptance. "Consideration Set Sampling to Analyze Undecided Respondents" adopts a choose-then-sample view in which respondents first eliminate alternatives and retain a smaller consideration set, which the survey then samples as a set-valued response \(Y_i:\Omega\to\tilde P(\mathcal S)\). The paper models these responses as categorical random sets and transfers multinomial regression, regularization, interpretable machine learning, and clustering to the set-valued outcome space. On the German pre-election poll example, the study uses \(5076\) observations, of which about \(22.8\%\) are undecided and provide non-singleton sets [2307.14333].

Taken together, these boundary cases suggest two clarifications. First, choose-then-sample is not synonymous with any sequential or selective pipeline. Second, the decisive criterion is where the selection operator sits relative to uncertainty resolution: before sampling, inside streaming acceptance, over latent structures that generate subsets, or after a sampled candidate pool has already been produced.

Source: https://www.emergentmind.com/topics/choose-then-sample