QuasiMoTTo: Quasi-Monte Carlo Test-Time Scaling

Abstract: Scaling inference compute, by generating many parallel attempts per problem, is a costly but reliable lever for improving LLM capabilities. By default these attempts are generated independently, wasting inference compute on redundant solutions. This waste seems unavoidable. After all, independence is what makes parallel sampling trivial to scale. However, this tradeoff is not fundamental: there is a rich design space of samplers that generate correlated but exact samples entirely in parallel. We explore this design space as an avenue for improving sample efficiency in scaling inference compute and reinforcement learning (RL). Concretely, we introduce QuasiMoTTo, which uses correlated samples as a drop-in replacement for i.i.d. samples. To generate these samples, QuasiMoTTo uses a reparameterization of autoregressive sampling as inverse-CDF sampling and draws the underlying uniforms with quasi-Monte Carlo (QMC); because QMC spreads the uniforms out more evenly than i.i.d., the resulting samples cover the output space with far less redundancy. Even though the batch is correlated, each sample is marginally distributed according to the LLM, so we can use the batch for policy-gradient training. Our empirical analysis focuses on understanding how efficiently QuasiMoTTo can turn compute into performance. To evaluate correlated samplers, whose dependence breaks standard pass@k estimators, we first develop an unbiased bootstrap estimator. Across four reasoning benchmarks, QuasiMoTTo matches i.i.d. pass@k accuracy with 25-47% fewer samples. Strikingly, QuasiMoTTo often saturates an upper bound on pass@k that holds for any marginal-preserving sampler. We also apply QuasiMoTTo to policy-gradient RL (GRPO) where it matches i.i.d. performance with 50% fewer training steps. These gains come from higher coverage, which yields a stronger learning signal per batch.

• The paper introduces QuasiMoTTo, a novel method that applies quasi-Monte Carlo based dependent sampling to improve sample efficiency in large language model inference.
• It details the use of lattice, stratified, and token-level Sobol samplers with inverse-CDF decoding to ensure marginal correctness while maximizing output diversity.
• Empirical results show up to a 50% reduction in RL training steps and 25–47% fewer samples for equivalent pass@k accuracy, highlighting significant compute savings.

QuasiMoTTo: Enhancing Sample Efficiency of LLM Inference via Quasi-Monte Carlo Methods

Motivation and Background

Scaling inference compute in LLMs typically involves parallel sampling—generating $k$ i.i.d. completions for the same prompt and aggregating outcomes via pass@$k$ or as policy gradient minibatches in RL. However, i.i.d. sampling induces substantial solution redundancy across batch members, particularly in low-entropy output spaces—most completions are clustered in high-likelihood regions, which diminishes effective coverage and exacerbates compute cost. The standard approach thus underutilizes parallel compute, imposing strong diminishing returns as $k$ increases.

The QuasiMoTTo method addresses this inefficiency, leveraging the statistical flexibility that only the per-sample marginals must be correct for unbiased average-type estimators and policy-gradient RL (Section 2, Figure 1). By constructing dependent—yet marginally exact—rollouts via quasi-Monte Carlo (QMC) samplers and autoregressive inverse-CDF arithmetic coding, QuasiMoTTo enables parallel batch generation with strong batch-wide coverage guarantees.

Figure 2: QuasiMoTTo contrasts i.i.d. sampling (top) with QMC-based correlated sampling (middle) and demonstrates efficient inverse-CDF decoding (right), leading to pronounced gains in sample efficiency (bottom right).

Methodology

QMC-Based Dependent Sampling

Three QMC samplers are instantiated:

• Lattice: Points are positioned uniformly on $[0, 1)$, shifted by a shared random offset, resulting in $k$ points $U_i = (i/k + \Delta) \bmod 1$. Each $U_i$ is marginally uniform, but batched samples are maximally repulsive, covering $[0,1]$ evenly.
• Stratified: $[0,1]$ is divided into $k$ strata, and a single marginally uniform point is drawn from each. This provides strong local coverage guarantees.
• Token-level Sobol: Sobol low-discrepancy sequences in $k$0 (for sequence length $k$1) are used to systematically generate diverse sequence rollouts.

The critical property is that each QMC-generated $k$2 is still $k$3 marginally. Thus, mapping $k$4 to sequence space via arithmetic/inverse-CDF coding yields batch samples $k$5 that are each distributed as $k$6, but collectively, these samples cover the output space with minimal redundancy.

Figure 1: Joint structure in two-sample batches may be arbitrary while preserving marginal correctness. QMC enables anti-correlated coupling that spreads samples, boosting coverage relative to i.i.d. sampling even when marginals are unchanged.

Arithmetic Sampling: Inverse-CDF Decoding

Given a QMC batch $k$7, each $k$8 is decoded using arithmetic/inverse-CDF coding, recursively partitioning $k$9 into bins on each token step according to $k$0 (Eq. (4) in the paper), preserving exact sampling fidelity.

This pipeline is embarrassingly parallel: once uniforms are sampled, decoding each sequence is independent and requires only a single additional state variable ($k$1) per rollout.

Estimation Procedures for Dependent Samples

Many estimators (policy-gradients, average-type metrics) remain unbiased for any marginally correct batch, but certain combinatorial metrics (notably pass@$k$2) depend on batch independence. QuasiMoTTo develops and proves a bootstrap estimator for pass@$k$3 under lattice/stratified/Sobol QMC constructions, exploiting lattice symmetries to yield unbiased pass@$k$4 estimates even under highly dependent batches.

Theoretical Upper Bound on Pass@$k$5

A key result is a non-improvable upper bound: for any marginal-preserving sampler with pass@1 $k$6, the maximum attainable pass@$k$7 is $k$8. This ceiling is unattainable for i.i.d. sampling except in trivial or degenerate cases, but QMC-based dependent samplers can closely approach it (Section 5).

Figure 3: The i.i.d. pass@$k$9 curve (coral) vs. the upper bound (black). The area between the curves indicates the redundancy cost of i.i.d. sampling, which QuasiMoTTo can minimize.

Empirical Results

Experiments consider four symbolic reasoning tasks (Countdown, Maze, Sudoku, 1D-ARC) using both supervised-finetuned and base LMs (Qwen3.5-0.8B/2B). All sampler variants use identical models and prompts.

Test-Time Scaling: Sample Efficiency in pass@$[0, 1)$0

QuasiMoTTo consistently and significantly outperforms i.i.d. sampling for pass@$[0, 1)$1 (~25–47% reduction in required samples for the same quality). The lattice sampler achieves near-upper-bound pass@$[0, 1)$2 on all benchmarks, indicating minimal further headroom for marginal-preserving sampling improvements.

Figure 4: pass@$[0, 1)$3 for lattice-based QuasiMoTTo (teal) dominates i.i.d. (coral) and closely tracks the upper bound (black) across all reasoning benchmarks.

Figure 5: Sample efficiency—QuasiMoTTo achieves identical accuracy as i.i.d. sampling using 25–47% fewer samples. Lattice is most efficient; Sobol and stratified also deliver gains.

RL Policy Gradient Optimization: Compute Efficiency

Plugging QuasiMoTTo into GRPO-style policy-gradient RL, the method reduces the number of training steps to reach a fixed pass@1 by up to 50% compared to i.i.d. sampling. The gains are attributed to increased within-batch reward variance, thereby improving the effectiveness of group-relative advantage signals and reducing the rate of zero-variance minibatches.

Figure 6: Policy improvement curves: QuasiMoTTo (blue) achieves a target pass@1 in significantly fewer training steps than i.i.d. sampling.

Figure 7: RL training dynamics: QuasiMoTTo produces fewer zero-variance groups and achieves higher training reward earlier than i.i.d. sampling, reflecting greater sample diversity.

Trade-offs and Analysis of Sampler Variants

QuasiMoTTo’s empirical and theoretical analyses reveal a freedom-coverage tradeoff: increasing sample dependence (as in lattice) improves batch coverage but decreases randomness per sample. Lattice yields maximal coverage (pairwise MI $[0, 1)$4), stratified is intermediate, i.i.d. offers maximal independence (MI $[0, 1)$5) but suffers maximal redundancy. Sobol provides efficient token-level coverage but is more suited to continuous or high-dimensional settings.

Implications and Future Directions

QuasiMoTTo demonstrates that sample coordination at inference time is a powerful, practically effective lever for improving both deployment and RL training sample efficiency without retraining the model, altering its semantics, or introducing sampling bias. Notably, this is achievable without loss of parallelism, marginal correctness, or estimator fidelity for key training schemes (e.g., GRPO). The method is orthogonal and complementary to prior work on decoding diversity and RL objective design, focusing on the sampling process itself.

For future research, several directions arise:

• Extending QMC-based coverage to semantic equivalence in open-ended tasks, especially where syntactic diversity underestimates true solution variety
• Adaptive QMC designs that target regions of high epistemic/aleatoric uncertainty for further efficiency
• Application to scientific discovery, code synthesis, or pattern induction tasks where diverse solution generation is essential for exploration and innovation

Conclusion

QuasiMoTTo provides a novel, theoretically grounded, and empirically validated framework for scaling LLM inference compute and accelerating RL-based training via dependent, QMC-based batch sampling. By replacing i.i.d. samples with carefully constructed, marginally-exact but batch-diverse samples, QuasiMoTTo achieves pronounced reductions in inference and RL rollout budgets while maintaining exact sampling semantics and estimation guarantees. This represents a significant refinement in practical LLM deployment and training pipelines, with potential to further amplify capabilities as model and compute scales grow.