---
title: Independent Client Sampling in Federated Learning
url: https://www.emergentmind.com/topics/independent-client-sampling
type: topic
---

# Independent Client Sampling in Federated Learning

Independent client sampling refers to a family of methodologies in distributed statistical learning and quality control—most prominently in federated learning (FL)—that select clients (or units) for participation in each round or stage via independent Bernoulli or multinomial draws, often with arbitrary inclusion probabilities. This design contrasts with fully deterministic or coordinated (dependent) sampling and has become foundational for scalability, privacy, fairness, and statistical efficiency in modern multi-device and privacy-aware distributed learning settings.

## 1. Principles and Mathematical Formulation

In independent client sampling, each client $i \in \{1, ..., N\}$ is independently sampled in round $t$ with probability $q_i^t$, possibly dependent on system, statistical, or privacy constraints. Formally:

\[
\mathbb{I}_i^t \sim \mathrm{Bernoulli}(q_i^t)
\]

where $\mathbb{I}_i^t$ indicates client $i$’s participation. The set of sampled clients is $S^t = \{i : \mathbb{I}_i^t = 1 \}$. Aggregation rules are typically designed for unbiasedness; for FL gradient aggregation,

\[
\hat{g}^t = \sum_{i=1}^N a_i \frac{\mathbb{I}_i^t}{q_i^t} g_i^t
\]

where $a_i$ is client's data weight and $g_i^t$ its local (stochastic) update. This structure ensures $\mathbb{E}[\hat{g}^t] = \sum_{i=1}^N a_i g_i^t$ regardless of $q_i^t > 0$.

The analysis of convergence and efficiency depends on the properties of the induced random aggregation weights (variance, covariance), the system and statistical heterogeneity across clients, and possibly additional constraints (privacy budgets, bandwidth allocations). Notable theoretical results show that even non-uniform, arbitrary $q_i^t$ can be accommodated while retaining convergence guarantees, provided the aggregation is appropriately debiased [2107.12211, 2112.11256, 2212.14370, 2402.10097].

## 2. Historical Context and Motivating Applications

Independent client sampling has evolved from earlier acceptance sampling in industrial quality control [1402.2468], where independent test (unit) sampling enables tractable, unbiased inference under unknown distributions. In federated learning, where global data centralization is infeasible, independent sampling supports:

- **Scalability:** Sampling a small subset per round mitigates the costs of global synchronous updates [2107.12211].
- **Straggler Mitigation:** Probabilistically reducing the chance of including slow or unavailable devices [2402.10097, 2505.23555].
- **Statistical Robustness:** Inclusive of non-i.i.d. data, mitigates selection bias over time [2211.13975, 2302.03662].
- **Privacy and Personalization:** Allows individualized privacy guarantees and sampling rates [2501.17634, 2412.05636].
- **Fairness:** Equitable participation among heterogeneous clients [2408.13683, 2211.13975].

Early works focus on random sampling as a default, later extended and refined to stratified, importance-weighted, or fairness-aware variants.

## 3. Advanced Methodologies: Adaptive, Stratified, Privacy- and Fairness-Aware Sampling

A range of strategies build upon the independent client sampling paradigm to address practical challenges:

### a) Adaptive Probability Tuning

Sampling probabilities are optimized to minimize convergence time or estimator variance, leveraging statistical heterogeneity (local data size/gradient norm) and system constraints (bandwidth, computation):

- **Variance-Minimizing Schemes:** Such as K-Vib [2310.02698], OSMD [2112.14332], DELTA [2205.13925], which adapt $q_i^t$ based on observed client gradient magnitudes and/or diversity, yielding superior regret/convergence bounds.
- **Bandwidth-Aware/Computation-Adaptive Sampling:** Probabilities $q_i^t$ reflect client communication/computation speed, balancing fast round completion with inclusion of important clients [2112.11256, 2402.10097, 2505.23555].

### b) Stratified and Importance Sampling

Clients are grouped into strata based on compressed gradients or metadata, with Neyman allocation minimizing estimator variance under round-level sampling quotas [2412.14226]. Importance sampling adjusts $q_i^t$ in proportion to informative statistics (e.g., gradient norms, diversity metrics).

### c) Privacy-Aware and Individualized Sampling

To achieve personalized or heterogeneous differential privacy (DP) guarantees:

- **Individualized DP through Sampling:** Each client sets a privacy budget $\varepsilon_i$. The server computes a group-specific sampling rate $q_i$ such that cumulative participation attains $(\varepsilon_i, \delta)$-DP, with a fixed noise multiplier [2501.17634].
- **Game-Theoretic and Incentive-Aligned Sampling:** Sampling probabilities are coordinated via economic mechanisms—clients declare privacy cost functions, and Stackelberg equilibria yield an optimal trade-off between privacy, participation, and model utility [2412.05636].

### d) Fairness-Driven and Diversity-Promoting Approaches

Limitations of uniform sampling in representing all clients equitably motivate submodular maximization approaches (SUBTRUNC, UNIONFL) [2408.13683], or graph-based diversity constraints [2211.13975], to ensure balanced or diverse participation over time.

## 4. Implications: Convergence, Efficiency, Privacy, and Fairness

Empirical and theoretical studies support clear distinctions between independent and non-independent sampling:

| Dimension         | Independent Sampling (with optimal $q_i^t$)          | Uniform/Dependent Sampling             |
|-------------------|------------------------------------------------------|----------------------------------------|
| **Convergence**   | Tight bounds; rate scales with inverse of $q_i^t$    | Slower for heterogeneous $g_i$ or $t_i$|
| **Wall-clock Time**| Minimized via $q_i^t$ adaptive to slow clients      | Bottlenecked by stragglers             |
| **Variance**      | Can be minimized/adapted round-wise                  | Higher, may include redundant clients  |
| **Privacy**       | Enables individualized budgets; efficient under DP   | Weak for heterogeneous privacy needs   |
| **Fairness**      | Customizable for inclusion/diversity; resilient      | Exclusion or bias possible             |
| **Practicality**  | Robust to client dropout, varying availability       | Less robust                            |

Empirical benchmarks show speedups in wall-clock time of $1.5-7\times$ over uniform sampling [2402.10097, 2112.11256], reduction in regret by a factor proportional to communication budget [2310.02698], and up to several percent improvement in global model accuracy and fairness [2501.17634, 2408.13683].

## 5. Specialized Domains and Extensions

### a) Streaming and Online Sampling

Under streaming, non-i.i.d. client data, sample selection for local labeling must be performed instantaneously and independently, often under memory and budget constraints. Recent work introduces numerically robust online batch selection using volume sampling and Cholesky updates in high-dimensional embedding spaces [2408.17108].

### b) Acceptance Sampling and Quality Control

In industrial settings (e.g., photovoltaics), independent sampling underpins control-inspection schemes where OC curves, decision limits, and sample size planning are derived under arbitrary distributional assumptions using nonparametric quantile estimates [1402.2468].

### c) Privacy-Preserving Aggregation

FedSTaS and related approaches integrate locally differentially private reporting of client data statistics in sampling and aggregation [2412.14226].

### d) Arbitrary Client Availability

FedGS demonstrates independent diversity- and fairness-aware sampling under arbitrary, possibly adversarial, client availability modes using data-distribution graphs and constrained variance optimization [2211.13975].

## 6. Open Challenges and Future Directions

Despite broad utility, several open challenges remain:

- **Non-i.i.d. Data Regimes:** Variance in updates and DP noise have amplified impacts when data is highly skewed and clients have limited, heterogeneous datasets, limiting achievable utility [2501.17634, 2412.14226].
- **Real-time Optimization:** Estimating optimal $q_i^t$ online in resource-constrained or privacy-limited settings (especially for rapidly changing environments) remains an active problem [2310.02698, 2112.14332].
- **Scalability of Complex Sampling Algorithms:** Approaches using submodular maximization, graph constraints, or advanced privacy mechanics must be engineered for massive-scale deployment with thousands to millions of devices [2408.13683, 2211.13975].
- **Usability and User-Centric Privacy:** Enabling informed user selection of privacy budgets and exposing the consequences to end users is an unsolved system and UI problem [2501.17634].

## 7. Summary Table of Representative Algorithms and Theoretical Guarantees

| Method/Class              | Objective (Key Formula)                               | Primary Guarantee/Result      |
|--------------------------|-------------------------------------------------------|-------------------------------|
| Variance-minimizing IS   | $p_i^t \propto \sqrt{\alpha_1\zeta_{G,i,t}^2 + \alpha_2\sigma_{L,i}^2}$ | Optimal variance under unbiasedness [2205.13925] |
| Bandit/Online OSMD       | Minimize $l_t(q)=1/K\sum_{i}\frac{a_{i}^{t}}{q_i}$    | Dynamic regret bounds; adaptivity [2112.14332]   |
| Privacy-aware/IDP        | $q_i = \text{getSampleRate}(\varepsilon_i, \delta, \sigma)$ | Per-client $(\varepsilon_i, \delta)$-DP [2501.17634] |
| Fairness/submodular      | $\max_S G(S) + \lambda\min(b, F(S))$                  | Improved client dissimilarity, strong convergence [2408.13683] |
| Adaptive bandwidth       | Joint $\min_{\mathbf{q}}$ expected wall-clock time    | $1.5-7\times$ speedup, full heterogeneity [2402.10097] |

## References to Seminal Works

- "A General Theory for Client Sampling in Federated Learning" [2107.12211]
- "Adaptive Federated Learning in Heterogeneous Wireless Networks with Independent Sampling" [2402.10097]
- "Federated Learning With Individualized Privacy Through Client Sampling" [2501.17634]
- "Enhanced Federated Optimization: Adaptive Unbiased Client Sampling with Reduced Variance" [2310.02698]
- "FedGS: Federated Graph-based Sampling with Arbitrary Client Availability" [2211.13975]
- "SUBTRUNC and UNIONFL: Submodular Maximization Approaches for Equitable Client Selection in Federated Learning" [2408.13683]
- "LOCKS: User Differentially Private and Federated Optimal Client Sampling" [2212.13071]
- "FedSTaS: Client Stratification and Client Level Sampling for Efficient Federated Learning" [2412.14226]

The field continues to rapidly develop novel independent sampling paradigms optimized for privacy, fairness, statistical efficiency, and practical deployment constraints.

Source: https://www.emergentmind.com/topics/independent-client-sampling