Selective Sample Inclusion Methods

• Selective sample inclusion is a method that deliberately includes or excludes data points based on auxiliary or outcome variables to enhance representativeness and reduce bias.
• It employs techniques such as unequal probability sampling, calibration, and adaptive algorithms to optimize efficiency and manage the bias-variance trade-off.
• Practical implementations span survey sampling, computer vision, and LLM fine-tuning, leveraging metrics like majorization and Kullback-Leibler divergence to validate results.

Selective sample inclusion refers to any deliberate process by which specific units, data points, or experimental conditions are preferentially included or excluded from analysis, estimation, annotation, or inference. This concept underpins a spectrum of methodologies across statistics, machine learning, survey sampling, and computer vision, designed to improve efficiency, generalizability, or mitigate bias by exercising explicit control over the selection of units. Selective sample inclusion mechanisms can be implemented via sampling weights, quota systems, selection functions conditioned on model output or covariates, or algorithmic strategies designed to maximize dataset informativeness under practical constraints.

1. Theoretical Foundations of Selective Sample Inclusion

Theory for selective sample inclusion originates in unequal probability sampling, constrained design-based estimation, survey calibration theory, selection models in statistics, and information theory (e.g., majorization and Kullback-Leibler divergence). At its core, a selective inclusion mechanism operates by replacing the default i.i.d. or random selection with a scheme—deterministic or stochastic—that depends on auxiliary or outcome-related variables.

Key results for unequal probability sampling plans include comparison, via majorization, between inclusion-probability vectors obtained under different sampling mechanisms. Let $\alpha = (\alpha_1, ..., \alpha_N)$ be prescribed drawing probabilities for a finite population $U = \{1, ..., N\}$, giving first-order inclusion probabilities $\pi_i$ for each unit $i$. For rejective (Hájek) sampling and successive sampling, these inclusion probabilities form vectors $p^{\rm R}(n)$ and $p^{\rm S}(n)$ respectively. Majorization ordering and Kullback-Leibler divergence provide rigorous measures for comparing their uniformity and fidelity to the original drawing probabilities (Yu, 2010):

• $p^{\rm R}(n) \prec p^{\rm S}(n) \prec \alpha$ (more uniform with increasing $n$)
• $$D(p^{\rm R}(n) \| \alpha) \geq D(p^{\rm S}(n) \| \alpha)$$ (successive sampling maintains closer proportionality)

These principles guide practical sampling, estimation adjustment, and bias mitigation.

2. Methodological Approaches

Selective sample inclusion manifests in various domains as follows:

2.1 Unequal Probability Sampling and Calibration

In survey sampling, unequal probability inclusion is undertaken to optimize estimator variance or target representativeness for key subgroups. Approaches include:

• Rejective (Conditional Poisson) Sampling: Samples are drawn with prescribed probabilities $\alpha$, then conditionally accepted to achieve fixed sample sizes, yielding inclusion probabilities with maximal entropy/majorization (Yu, 2010).
• Calibration Estimation and Inverse Sampling: Weights are constructed to match known auxiliary totals in the population, with a second-phase sample drawn from the big data sample using calibrated probability-proportional-to-size (PPS) weights. The Horvitz-Thompson estimator

$$\hat{Y}_{\rm IS} = \sum_{i \in B_2} \frac{y_i}{\pi_i}$$

is unbiased for the population mean under MAR and correct calibration (Kim et al., 2018).

2.2 Selection Under Informative or Non-Ignorable Designs

Selective inclusion becomes critical when the selection mechanism depends on the outcome or covariates (informative sampling). Correction methods rely on modeling the selection probability $G(y; w, \gamma)$ and forming a pseudo-likelihood or weighted likelihood: $$L(\theta, \gamma) = \prod_{i:S_i=1} [f_Y(y_i|x_i;\theta) G(y_i;w_i, \gamma)] \prod_{i:S_i=0} [1 - \pi_i]$$ where $\pi_i = E_{Y}[G(Y;w_i, \gamma)|x_i;\theta]$ (Azzalini et al., 2016). Bayesian approaches incorporate inverse-probability exponentiation into the likelihood, forming a "pseudo-posterior," and propagate weights into hierarchical or latent-variable models (Savitsky et al., 2015, Sikov, 2015).

2.3 Selective Subset/Quota Strategies in Observational and Experimental Design

For generalizability in field experiments or impact studies, sample inclusion can be constrained to match external distributions of key moderators using quota rules, stratification, or optimization:

• Define population proportions $\hat{p}_c$ for categories $c$ from survey data.
• Set integer bounds $\ell_c, u_c$ for total sample size $J$ and enforce $\ell_c \leq n_c \leq u_c$ during recruitment (Olsen, 2022, Olsen et al., 2023).

2.4 Online, Active, and Adaptive Sampling

In dynamic or spatial sampling, sequential sample inclusion adaptively targets underrepresented regions or auxiliary-balance:

• Sequential Spatially Balanced Sampling: Maintains exact marginals, auxiliary balance, and spatial spread even under streaming arrivals (Jauslin et al., 2021).
• Selective Active Learning/Annotation: In computer vision, samples for annotation are selected via clustering, representativeness, or informativeness metrics to minimize annotation effort while retaining performance (Wang et al., 2023).

2.5 Selective Conformal and Risk Control

Selective inclusion is used post-hoc on model outputs to control risk or abstain on low-confidence predictions, often optimizing prediction set size under conditional coverage and risk constraints:

• Two-stage thresholding determines a selection set $S$ (confidence threshold $\lambda_1$), then calibrates risk inside $S$ ($\lambda_2$), e.g. in Selective Conformal Risk Control (Xu et al., 14 Dec 2025).

3. Algorithmic and Statistical Properties

The efficacy and theoretical properties of selective inclusion mechanisms are characterized by:

• Bias and Variance Control: Design-unbiasedness (e.g., Horvitz-Thompson estimator), asymptotic consistency (under regularity and correct modeling), and conditional risk guarantees.
• Majorization and Uniformity: Rejective sampling produces maximally uniform inclusion probability vectors; successive sampling produces inclusion probabilities closely proportional to initial drawing probabilities (Yu, 2010).
• Finite-Sample Guarantees: Some selective conformal algorithms provide finite-sample or PAC-style risk bounds conditional on the selection event (Xu et al., 14 Dec 2025).
• Asymptotic Validity in Selective Inference: Procedures using external randomization, smoothing, or pivot construction yield asymptotically exact inference without discarding data or splitting datasets, as in selective quantile regression (Wang et al., 3 Apr 2024).

4. Practical Implementations and Applications

Selective sample inclusion is implemented in diverse settings, including:

Context	Selection Mechanism	Objective
Survey sampling	PPS, rejective, strata/quota	Variance, representativeness
Big data finite-pop inference	Inverse calibration/PS weighting	Reduce selection bias
Medical image annotation	Clustering, MCMC, rep. selection	Annotation efficiency
Field experiments/trials	Quota/stratum-limited recruitment	External validity bias control
Spatial data streams	Windowed LP, marginal & spread balance	Spatial coverage and auxiliary-balance
Deep learning/SFT data selection	LLM-guided marginal gain/gain+diversity	Labeling budget and generalization
Uncertainty quantification	Selective conformal sets, abstention	Conditional coverage and compactness

In supervised fine-tuning for LLMs, choice-based greedy paradigms select samples incrementally according to their marginal utility as judged by the model itself, optimizing both efficiency and downstream generalization (Li et al., 4 Mar 2025).

5. Trade-offs, Limitations, and Design Considerations

Implementing selective sample inclusion involves several methodological trade-offs:

• Uniformity vs. Proportionality: Rejective sampling is preferable when uniform coverage of the population is critical; successive sampling better tracks desired inclusion weights (Yu, 2010).
• Bias-Variance Trade-off: Tight quotas or selection rules increase representativeness but can induce higher variance in small populations or when quotas are over-specified (Olsen, 2022, Olsen et al., 2023).
• Computation vs. Performance: Algorithms such as sequential spatially balanced sampling minimize variance but require solving LPs per update step, though practical choices of window sizes and pool sizes can mitigate costs (Jauslin et al., 2021).
• Selective Algorithmic Updates: In meta-prompt learning (e.g., Selective LWE), only cases with signal of inconsistency are used to trigger adaptation, improving efficiency but possibly missing systematic errors on consistently misjudged cases (Jwa et al., 7 Dec 2025).
• Correctness of Model-based Weighting: Model misspecification or failure of MAR/transportability conditions can invalidate calibration or propensity score estimators, necessitating careful diagnostic checks (Kim et al., 2018).

6. Impact and Empirical Evidence

Simulation studies, empirical benchmarks, and real-world deployments consistently show:

• In big data settings, properly weighted inverse sampling and doubly robust integration yield unbiased estimators, correct variance, and proper CI coverage, even under strong selection bias, unlike naïve or unweighted approaches (Kim et al., 2018).
• Quota and stratification-based selective inclusion in randomized controlled trials and impact studies substantially reduces external validity bias (e.g., by more than 0.23 SD in school population means), sometimes at limited variance cost, provided sufficient pool size and accurate auxiliary information (Olsen et al., 2023).
• Selective LWE and similar selective adaptation frameworks offer cost-effective improvements in LLM evaluation consistency, with up to 0.947 consistency and minimal additional compute relative to full sequential updates (Jwa et al., 7 Dec 2025).
• State-of-the-art object detectors leveraging two-stage selective anchor assignment (TS⁴Net) achieve superior mAP with streamlined architectures, confirming the importance of stage-adaptive selective inclusion for both coverage and precision (Feng et al., 2021).

7. Advanced Topics and Current Research Frontiers

Current work continues to expand selective sample inclusion strategies into high-dimensional selective inference, developing pivots and confidence intervals that remain valid post-selection or post-adaptation, exploiting external randomization for consistent and powerful inference (Wang et al., 3 Apr 2024, Tian et al., 2015). Adjustment to non-Gaussian and discrete outcome settings generalizes classic selection models, enabling inclusion correction in a wide range of applications (Azzalini et al., 2016).

Research continues in developing algorithmically efficient, statistically robust, and theoretically grounded selective sample inclusion techniques suitable for large-scale, online, and adaptive data-driven contexts across science and AI.

---

References:

• "On the inclusion probabilities in some unequal probability sampling plans without replacement" (Yu, 2010)
• "Sampling techniques for big data analysis in finite population inference" (Kim et al., 2018)
• "Bayesian Estimation Under Informative Sampling" (Savitsky et al., 2015)
• "Sample selection models for discrete and other non-Gaussian response variables" (Azzalini et al., 2016)
• "Sequential Spatially Balanced Sampling" (Jauslin et al., 2021)
• "Selective inference with a randomized response" (Tian et al., 2015)
• "Select Conformal Risk Control" (Xu et al., 14 Dec 2025)
• "TS4Net: Two-Stage Sample Selective Strategy for Rotating Object Detection" (Feng et al., 2021)
• "Add-One-In: Incremental Sample Selection for LLMs via a Choice-Based Greedy Paradigm" (Li et al., 4 Mar 2025)
• "Convergence of Nearest Neighbor Pattern Classification with Selective Sampling" (Joseph et al., 2013)
• "Using Survey Data to Obtain More Representative Site Samples for Impact Studies" (Olsen, 2022)
• "Using Auxiliary Data to Guide the Recruitment of Sites for Randomized Controlled Trials" (Olsen et al., 2023)
• "Asymptotically-exact selective inference for quantile regression" (Wang et al., 3 Apr 2024)
• "Becoming Experienced Judges: Selective Test-Time Learning for Evaluators" (Jwa et al., 7 Dec 2025)