Stochastic Baseline Sampling

• Stochastic baseline sampling is defined as using i.i.d. random methods, such as Monte Carlo, to establish benchmarks for variance reduction and batch selection.
• It is applied in empirical risk minimization and deep active learning to evaluate and compare selection techniques across complex models.
• Adaptive techniques like Bayesian optimization and parallel optimized sampling enhance baseline performance by improving rare event coverage and reducing moment errors.

Stochastic baseline sampling refers to a class of methodologies in which random sampling is employed as the foundational approach for constructing inference ensembles, acquiring data batches, or propagating trajectories in probabilistic models. These procedures serve as benchmarks against which variance reduction techniques, information-theoretic selection, and adaptive or optimized sampling mechanisms are evaluated. In the context of empirical risk minimization, stochastic equations, deep active learning, and generative trajectory prediction, baseline sampling strategies are crucial for both methodological analysis and practical deployment. The most salient instantiation is the Monte Carlo paradigm, wherein samples are drawn independently from an underlying distribution without additional optimization or scoring. Recent advances have introduced stochastic wrappers for batch selection, as well as adaptive enhancements for rare event coverage, yet the random-sample baseline remains the central comparative standard.

1. Monte Carlo Baseline and Its Role

The archetype of stochastic baseline sampling is Monte Carlo sampling: for a latent variable model $G_\theta(X, z)$ parameterized by history $X$ and latent $z$, the baseline approach draws $N$ i.i.d. samples $z_n \sim p(z)$ and computes trajectories $\tau_n = G_\theta(X, z_n)$ (Chen et al., 2023). In the context of empirical risk minimization, stochastic gradient methods sample data points randomly at each iteration, providing an unbiased estimator of the empirical risk (Csiba, 2018).

This baseline is unbiased but suffers several limitations. For sharply peaked or unimodal $p(z)$, MC samples predominantly fall in high-probability modes, rarely exploring low-probability—and often semantically critical—regions. In batch active learning, MC-style top-K selection is simply the batch of highest acquisition scores, neglecting inter-sample dependencies and dynamic score drift as batches are labeled (Kirsch et al., 2021).

2. Sampling Variance: Central-Limit Scaling and Its Implications

Monte Carlo methods exhibit sampling variance scaling as $O(N^{-1/2}_S)$ for $N_S$ samples. For stochastic differential equations (SDEs), the central-limit theorem ensures that the sampling error of moment estimators $\bar o_m = (1/N_S)\sum_{n=1}^{N_S} [x^{(n)}]^m$ decays as $\varepsilon_S \sim \sigma_m / \sqrt{N_S}$, where $\sigma_m^2 = \mathrm{Var}[x^m]$ (Opanchuk et al., 2015). This canonical scaling cannot be improved by simply decreasing the time step or increasing computational precision; structural variance persists.

The prevalence of this error structure in baseline sampling motivates the development of variance reduction methods such as parallel optimized sampling (POS), Bayesian optimization samplers, and information-theoretic batch acquisition algorithms.

3. Stochastic Batch Acquisition as a Baseline

In deep pool-based active learning, a highly efficient stochastic baseline is provided by stochastic batch acquisition (SBA), as formulated by Kirsch et al. (Kirsch et al., 2021). Given a scoring function $s(i; D)$ per candidate $i$, the SBA algorithm samples batch members without replacement according to distributions derived from $s(i)$. Three key variants are proposed:

• Softmax acquisition: $p_i \propto \exp(\beta s(i))$, Gumbel noise is added to scores, and the top-$K$ perturbed points are selected.
• Power acquisition: $p_i \propto s(i)^\beta$, where $s(i) \ge 0$ and Gumbel noise is added to $\log s(i)$.
• Soft-rank acquisition: Sampling probability via ranks, $p_i \propto r_i^{-\beta}$.

The operational complexity is $O(M\log K)$ per batch, matching naïve top-K in runtime but empirically outperforming top-K BALD and BADGE in coverage, minority group accuracy, and fairness, with identical computational cost. SBA remains purely an inference-time method, requiring no additional model retraining during batch selection.

Acquisition Name	Sampling Probability $p_i$	Perturbation Applied
Softmax	$\propto \exp(\beta s(i))$	$s(i) + \epsilon_i$
Power	$\propto s(i)^\beta$	$\log s(i) + \epsilon_i$
Soft-rank	$\propto r_i^{-\beta}$	$-\log r_i + \epsilon_i$

4. Adaptive Baseline Enhancement: Bayesian Optimization in Stochastic Prediction

BOsampler provides an adaptive enhancement to baseline MC sampling by employing Bayesian optimization to mine the long-tail of predicted distributions (Chen et al., 2023). Here, trajectory sampling is treated as a sequential design problem. A Gaussian process surrogate is constructed over pseudo-score functions $f(z)$, balancing exploitation (high mean) and exploration (high uncertainty):

• The acquisition function is $$a_{w-1}(z) = \mu_{w-1}(z) + \sqrt{\beta\,\kappa_{w-1}(z,z)}$$, where $\mu_{w-1}$ and $\kappa_{w-1}$ are the GP posterior mean and variance, respectively.
• After a warm-up random sample phase, further samples are chosen by maximizing the acquisition function, iteratively improving coverage of rare but important trajectory modes.

Experimental metrics on multi-modal human trajectory datasets demonstrate that BOsampler yields an ADE/FDE improvement of $23.7\%/27.5\%$ over MC on "rare event" subsets, with minor additional computational overhead. Combining quasi-Monte Carlo warmup with BOsampler further marginally improves rare mode discovery.

5. Parallel Optimized Sampling: Baseline and Moment-Matching Optimization

Parallel optimized sampling (POS) directly targets the persistent variance in baseline MC estimators for SDEs and statistical equations (Opanchuk et al., 2015). Given knowledge of the exact moments $\mu_m$ of the target distribution, POS constructs an ensemble $X = (x_1, ..., x_{N_S})^\top$ that matches the first $M$ moments exactly up to numerical precision.

• The cost function is $C(X) = \|\bar{\mathbf{o}}(X) - \mu\|^2$ for sample moments $\bar o_m(X)$.
• Optimization uses a Gauss–Newton step: $$\delta X = J^\top u^{-1}[\mu - \bar{\mathbf{o}}(X)]$$, updating $X$ until the deviation vanishes.
• This produces initial ensembles and time-step updates with sampling errors at low-order moments reduced from $O(N_S^{-1/2})$ to machine precision $10^{-16}$ in static cases and up to $10$–$100\times$ lower than MC under nonlinear time-evolution.

POS does not introduce systematic bias in observables and remains unbiased as $N_S \rightarrow \infty$, though in nonlinear dynamics error propagation in higher moments persists over long runs.

6. Comparative Experimental Benchmarks

Empirical results across domains reinforce the utility and limitations of stochastic baseline sampling:

• In active learning (MNIST, EMNIST, Synbols, CLINC-150), stochastic batch methods (PowerBALD, Softmax, Softrank) dominate top-K BALD and BADGE in both accuracy and fairness, often at $10^3$–$10^4\times$ faster runtimes (Kirsch et al., 2021).
• BOsampler demonstrates a substantial boost in rare trajectory coverage on ETH-UCY using Social-GAN, Trajectron++, PECNet, and Social-STGCNN, achieving up to $27.5\%$ lower final displacement error for top $4\%$ outlier modes (Chen et al., 2023).
• POS yields moment error collapse in SDE simulations: for $N_S=10^3$, moment errors drop from $\sim 3\times10^{-2}$ to $\sim 10^{-16}$, with nonlinear cases seeing $10$–$100\times$ error reduction in low-order cumulants (Opanchuk et al., 2015).

These results suggest that stochastic baseline sampling, when enhanced by stochastic wrappers and adaptive optimization, can achieve accuracy, fairness, and computational efficiency that challenge the necessity of complex, compute-intensive batch algorithms.

7. Limitations, Parameters, and Practitioner Recommendations

Stochastic baseline sampling carries several inherent constraints:

• Coverage deficiency: Pure MC samples fail to explore low-probability distribution regions without exponentially increasing budget.
• Score drift: Batch acquisition selection must accommodate score instability and inter-batch redundancies; stochastic methods alleviate but do not eliminate this.
• Parameter tuning: Coldness parameter $\beta$ in SBA methods requires data-dependent tuning for optimal performance; default $\beta=1$ suffices in most practical cases (Kirsch et al., 2021).
• Computational overhead: Adaptive methods (e.g., BOsampler, POS) incur constant-factor cost increases over baseline MC, but remain scalable ($N \lesssim 20$ in BOsampler, small $M$ in POS).
• Bias risk: POS and BOsampler induce no systematic observable bias, reverting to baseline distribution properties as sample size grows.

Practitioner guidance based on experimental results recommends replacing naïve top-K or random MC by stochastic batch selection (Power or Softmax) and, where rare event coverage is critical, supplementing MC with Bayesian optimization or moment-matching initialization. These strategies provide strictly superior or equivalent performance at negligible additional computational cost, ensuring robust benchmarking for novel algorithmic developments.