Balanced Sampling Methods

• Balanced sampling is a statistical design method that selects samples meeting prescribed auxiliary constraints to accurately mirror target population characteristics.
• Techniques like the cube method and pivotal sampling use randomization and combinatorial balancing to achieve unbiased estimators and reduced variance.
• Applications in survey design, machine learning, and federated learning demonstrate its practical value in mitigating bias and ensuring representative, robust data.

Balanced sampling is a family of design strategies and algorithms in which samples are selected from a population or dataset such that prescribed constraints—typically involving auxiliary information or structural balance—are met exactly or approximately. The concept arises in probabilistic survey design, machine learning, knowledge graph embedding, federated learning, large-scale LLM pre-training, and other domains where random sampling alone yields suboptimal estimators or biased training due to imbalanced class, feature, or structural distributions. Core to balanced sampling is ensuring that the sampled data reflects the characteristics of the target population along important auxiliary variables, groupings, or patterns, sometimes involving exact or combinatorial balancing conditions.

1. Theoretical Foundations and Principles

Balanced sampling synthesizes several principles formalized in the probability sampling literature:

• Randomization: The sampling design should maintain a high degree of randomness (entropy) to avoid systematic biases and yield robust inference. The entropy of a sampling design $I(p) = -\sum_{s} p(s) \log p(s)$ quantifies this criterion (Tillé et al., 2016).
• Overrepresentation: Units with higher anticipated variance or influence (e.g., larger size, greater uncertainty) are sampled with higher inclusion probabilities, frequently proportional to auxiliary dispersion or size metrics.
• Restriction (Balancing Constraint): The sample is constrained so that weighted auxiliary variable totals match known population totals—formally, for sample $S$ with inclusion probabilities $\{\pi_k\}$ and vector-valued auxiliary variables $x_k$, the balancing condition reads $\sum_{k\in S} x_k/\pi_k = \sum_{k\in U} x_k$ (Tillé et al., 2016).

In model-assisted frameworks, balancing emerges as optimal for anticipated variance minimization—particularly under working models such as $y_k = x_k^\top\beta + \varepsilon_k$.

2. Core Algorithms: Cube Method, Pivotal Sampling, and Their Variants

Balanced sampling has algorithmic realizations in the cube method and its pivotal variant:

• Cube Method [Deville and Tillé, 2004]: Constructs a sample by sequentially updating an initial vector of inclusion probabilities along directions in the null space of the auxiliary matrix, ensuring the balancing condition is met as units are rounded to selection (1) or omission (0). Variants such as the "flight phase" and "landing phase" adapt the method for complex constraints or high stratification (Jauslin et al., 2021).
• Pivotal Sampling (Chauvet, 2012): A particular case of the cube method, especially when balancing only on inclusion probabilities. Ordered pivotal sampling proceeds with a fixed unit ordering: pairs (or fighting pairs) are iteratively "rounded"—one excluded, one carries forward—until sample size and balancing constraints are achieved. The process is exact and without replacement, leading to closed-form first and second order inclusion probabilities.

A summary of key formulas involved:

• For two non–cross-border units $k$ and $l$ in microstrata $U_i$ and $U_j$ with $i<j$:

$$\pi_{kl} = \pi_k\pi_l\left\{1 - \prod_{l=i}^{j-1}\frac{a_l b_l}{(1-a_l)(1-b_l)}\right\}$$

with $a_i, b_i$ as cumulative sums at cross–border units (Chauvet, 2012).

These designs preserve inclusion probabilities, enable unbiased Horvitz-Thompson estimation, and—if the auxiliary is strongly correlated with the paper variable—minimize variance.

3. Robustness, Variance Properties, and the Role of Ordering

The efficiency of balanced sampling depends on the suitability of unit ordering (for methods like ordered pivotal/systematic sampling) and the appropriateness of auxiliary variables:

• Variance Reduction: When the population is ordered such that the balancing variable and the target variable are highly correlated, microstrata formed are homogeneous, yielding estimators with minimal variance. In the case of pivotal sampling and systematic sampling with appropriate orderings, this induces a form of implicit stratification (Chauvet, 2012).
• Robustness: Ordered pivotal sampling introduces randomization not present in deterministic systematic sampling, leading to greater robustness against unfavorable orderings. For pivotal sampling, the maximum design effect

$\mathit{DMAX}(ops)=\frac{N-1}{N-n}$

is close to 1 for large $N$, while systematic sampling can in the worst case be $n$ times higher (Chauvet, 2012). This underscores the protective effect of additional randomization.

• High-Dimensional and Highly Stratified Populations: Cube-based and pivotal methods may face computational slowdowns when auxiliary variables induce a highly stratified structure and inclusion probabilities are fractional. Enhanced algorithms decompose the balancing step into block-wise updates for scalability (Jauslin et al., 2021).

4. Extensions: Spatial, Multi-Dimensional, and Domain-Aware Balanced Sampling

Balanced sampling methodologies generalize beyond simple vector-based balancing:

• Spatial Balanced Sampling: When spatial coordinates are included as auxiliary variables (or as balancing constraints), special algorithms (local pivotal, generalized random tessellation stratified designs) ensure selected units are spatially well spread, inhibiting spatial clustering and redundancy (Tillé et al., 2016).
• Combinatorial Designs: In survey settings where structural properties—such as avoiding contiguous (adjacent) spatial or temporal units—are required, balanced sampling plans use group divisible designs and cyclic constructions to ensure forbidden configurations are excluded while maintaining balance over allowed pairings (Wang et al., 2014).
• Multi-Domain and Federated Settings: Balanced sampling is central in multi-domain batch scheduling (e.g., medical imaging), where batches are constructed with balanced representation from multiple data sources. In federated learning with non-IID data, frameworks use stratified label schedules and label-aware client selection to ensure balanced gradient aggregation and improved generalization, supported by privacy-preserving aggregation mechanisms (Wong et al., 18 Apr 2025, Tetteh et al., 2021).
• Balanced Subsampling for Categorical Covariates: For big data with categorical predictors, balanced subsampling ensures all levels (and their pertinent combinations) are represented, yielding non-singular information matrices and robust parameter estimation. The imbalance function $f(X_s)$ measures deviation from balanced representation, and sequential algorithms minimize this function for practical scalability (Wang, 2022).

5. Applications: Survey Sampling, Machine Learning, Forecasting, and Deep Model Training

Balanced sampling principles underpin a range of practical applications, tailored to the underlying data structure and modeling goals:

• Official Surveys and Censuses: Cube and pivotal sampling methods are utilized to select representative, variance-reducing samples in large-scale censuses, adapting to available auxiliary data and real-world population structure (Chauvet, 2012).
• Healthcare Analytics and Credit Scoring: Balanced (stratified) sampling addresses class imbalance in supervised learning, especially in cancer prognosis or financial credit prediction tasks, leading to consistent improvements in model accuracy and robustness compared to random or simple stratified sampling (Saleema et al., 2014, Qian et al., 2021).
• Time Series Forecasting: Universal forecasting models benefit from balanced time series corpora created by grid-based sampling over statistical features to ensure all types/patterns of series are equitably represented, which accelerates convergence and improves generalization (Shao et al., 23 May 2025).
• Knowledge Graphs and Representation Learning: Balanced sampling addresses overrepresentation of hub entities in heterogeneous networks and knowledge graphs, leveraging degree-aware sampling and network coarsening to supply high-entropy, informative training pairs. In distributed KGE training, balanced sampling ensures equitable memory and communication loads (Zhan et al., 2021, Cattaneo et al., 2022).
• LLM Training: Cluster-based balanced sampling, sometimes with repetition-clip mechanisms, rebalance long-tailed document distributions during LLM pre-training or fine-tuning, mitigating overfitting and enhancing generalization across rare data or semantic clusters (Shao et al., 22 Feb 2024).
• Quantization for MoE LLMs: Expert-Balanced Self-Sampling (EBSS) constructs calibration sets that achieve low perplexity while preserving balanced expert usage, directly improving quantization accuracy in Mixture-of-Experts models (Hu et al., 2 May 2025).

6. Future Directions and Challenges

Balanced sampling remains an active area for methodological innovation and application:

• Algorithmic Efficiency: Research continues on the development of polynomial-time and scalable algorithms for balanced sampling in complex structures, including high-dimensional, spatial, and networked data (Cannon et al., 2023).
• Incorporation of Complex Constraints: Extensions include integrating demographic, fairness, and operational constraints, as well as adapting to online or streaming populations, sequential sampling settings, and privacy-preserving frameworks (Jauslin et al., 2021, Wong et al., 18 Apr 2025).
• Variance Estimation and Model Assessment: Continued refinement of variance estimation procedures for balanced samples, especially under multi-phase or multi-objective designs, remains essential for inferential reliability (Jauslin et al., 2021, Choi et al., 2023).
• Domain-Generalization and OOD Robustness: Application of balanced sampling in settings characterized by strong distribution shifts (e.g., multi-domain medical imaging, federated learning with extreme non-IID data) is driving advances in generalizable model development (Tetteh et al., 2021, Wong et al., 18 Apr 2025).

Balanced sampling thus provides a rigorously founded, algorithmically diverse, and context-adaptive paradigm for sample selection, model training, and estimation design, with broad applicability across statistical, machine learning, and data science domains.