Reasoning with Sampling: Your Base Model is Smarter Than You Think

Abstract: Frontier reasoning models have exhibited incredible capabilities across a wide array of disciplines, driven by posttraining LLMs with reinforcement learning (RL). However, despite the widespread success of this paradigm, much of the literature has been devoted to disentangling truly novel behaviors that emerge during RL but are not present in the base models. In our work, we approach this question from a different angle, instead asking whether comparable reasoning capabilites can be elicited from base models at inference time by pure sampling, without any additional training. Inspired by Markov chain Monte Carlo (MCMC) techniques for sampling from sharpened distributions, we propose a simple iterative sampling algorithm leveraging the base models' own likelihoods. Over different base models, we show that our algorithm offers substantial boosts in reasoning that nearly match and even outperform those from RL on a wide variety of single-shot tasks, including MATH500, HumanEval, and GPQA. Moreover, our sampler avoids the collapse in diversity over multiple samples that is characteristic of RL-posttraining. Crucially, our method does not require training, curated datasets, or a verifier, suggesting broad applicability beyond easily verifiable domains.

• The paper introduces a novel, training-free MCMC-based power sampling algorithm that elicits latent reasoning capabilities from base LLMs.
• Empirical results demonstrate that power sampling matches or exceeds RL-posttraining on both in-domain and out-of-domain tasks while maintaining sample diversity.
• The study challenges the need for expensive RL-posttraining by revealing that base models can achieve strong reasoning performance through advanced inference-time sampling.

Reasoning with Sampling: Your Base Model is Smarter Than You Think

Introduction

This paper investigates the extent to which base LLMs possess latent reasoning capabilities that can be elicited at inference time via advanced sampling, without any additional posttraining. The authors challenge the prevailing assumption that reinforcement learning (RL) posttraining is necessary to unlock strong single-shot reasoning performance. They introduce a novel, training-free sampling algorithm inspired by MCMC techniques, specifically targeting the power distribution $p^\alpha$ of the base model. Empirical results demonstrate that this approach matches or even surpasses RL-posttraining (GRPO) on both in-domain and out-of-domain reasoning tasks, while maintaining sample diversity and avoiding the mode collapse characteristic of RL methods.

Figure 1: Our sampling algorithm matches and can outperform RL-posttraining across multiple reasoning tasks and domains.

Power Distribution and Sampling

The central theoretical contribution is the identification of the power distribution $p^\alpha$ as a superior target for reasoning tasks. Unlike low-temperature sampling, which greedily sharpens next-token distributions, sampling from $p^\alpha$ upweights sequences with few but high-likelihood completions, implicitly planning for future high-likelihood tokens. This is formalized by contrasting the sum-of-exponents (power sampling) with exponent-of-sums (low-temperature sampling), showing that only the former properly accounts for pivotal tokens and critical windows in reasoning traces.

The authors provide a rigorous analysis and toy examples illustrating that power sampling favors tokens leading to high-likelihood completions, even when their marginal probability is lower than alternatives with many low-likelihood completions. This property is crucial for reasoning tasks, where a small number of pivotal tokens can determine correctness.

MCMC-Based Power Sampling Algorithm

To sample from the intractable $p^\alpha$ distribution over sequences, the authors employ a Metropolis-Hastings (MH) MCMC algorithm. The proposal distribution is constructed by randomly resampling subsequences using the base model, ensuring irreducibility and aperiodicity. The acceptance ratio is computed using unnormalized likelihoods, leveraging the autoregressive structure for efficient computation.

The algorithm proceeds in blocks, progressively sampling from intermediate distributions and running $N_{\text{MCMC}}$ MH steps per block. This sequential approach mitigates exponential mixing times and leverages the structure of language generation for practical efficiency.

Figure 2: Metropolis-Hastings sampling with random resampling of subsequences, enabling efficient exploration of the sequence space.

Experimental Results

The authors benchmark their algorithm on MATH500, HumanEval, GPQA, and AlpacaEval 2.0, using Qwen2.5-Math-7B, Qwen2.5-7B, and Phi-3.5-mini-instruct as base models. The RL baseline is GRPO, the standard for posttraining LLMs on reasoning tasks.

Key findings include:

• Single-shot accuracy: Power sampling achieves near-parity with GRPO on in-domain tasks (MATH500) and outperforms GRPO on out-of-domain tasks (HumanEval, AlpacaEval 2.0).
• Sample diversity: Unlike RL-posttraining, which collapses diversity, power sampling maintains high pass@$k$ performance for large $k$, matching the base model's diversity while improving accuracy.

  Figure 3: Power sampling and GRPO both sample from high-likelihood and high-confidence regions, but power sampling maintains greater spread and diversity.

  

  Figure 4: Pass@$k$ accuracy curves show power sampling strictly dominates both GRPO and the base model, with sustained diversity at high $k$.

  

  Figure 5: MATH500 accuracy as a function of $\alpha$ and number of MCMC steps, demonstrating robustness and scalability of power sampling.

Analysis and Implementation Considerations

The algorithm is training-free, dataset-free, and verifier-free, making it broadly applicable and avoiding the instabilities and resource requirements of RL methods. The main computational cost is increased inference-time sampling, scaling as $\mathcal{O}(N_{\text{MCMC}} T^2 / B)$ tokens per output sequence. Empirically, moderate values of $N_{\text{MCMC}$ and block size $B$ suffice for strong performance, with diminishing returns beyond $N_{\text{MCMC}} \approx 10$.

Hyperparameter sensitivity is low: $\alpha=4.0$ is optimal across models and tasks, and the method is robust to variations. The proposal distribution can be tuned (e.g., higher temperature for general helpfulness tasks) for further gains.

Qualitative analysis shows that power sampling produces longer, more detailed reasoning traces, matching RL in response length without explicit incentives. It also solves problems missed by GRPO, indicating that base models possess untapped reasoning capacity.

Implications and Future Directions

The results imply that much of the reasoning improvement attributed to RL-posttraining is recoverable by inference-time sampling from the base model, provided the sampling algorithm targets the right distribution. This challenges the necessity of expensive RL pipelines for many reasoning applications and suggests that base models are underutilized.

Practically, this opens avenues for deploying strong reasoning capabilities in resource-constrained settings, or in domains lacking verifiable reward signals. Theoretically, it motivates further study of the relationship between base model likelihoods, reasoning capacity, and the geometry of the sequence space.

Future work may explore adaptive or learned proposal distributions, integration with other inference-time steering methods, and extension to multimodal or non-autoregressive architectures. The approach also invites investigation into the limits of base model reasoning and the design of sampling algorithms that can elicit even more sophisticated behaviors.

Conclusion

This paper demonstrates that base LLMs are substantially more capable at reasoning than previously revealed by standard sampling methods. By targeting the power distribution via MCMC-based sampling, single-shot reasoning performance matches or exceeds RL-posttraining, while maintaining diversity and generalizability. The findings suggest a paradigm shift in how reasoning capabilities are elicited from LLMs, emphasizing inference-time algorithms over posttraining, and highlight the importance of understanding and leveraging the latent structure of base model distributions.