Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning

Abstract: LLMs increasingly tackle hard reasoning problems by spending more test-time compute, yet the dominant strategy remains naive repeated sampling: draw many independent solutions and hope one is correct. Because such sampling explores only through local decoding noise, it tends to produce many near duplicate attempts rather than genuinely different ideas. We ask whether exploration can instead be steered at a semantic level, by first sampling problem specific concepts, hints, or strategies and then conditioning answer generation on them. We refine this into a simple, more exploratory procedure that emits many diverse concepts in a single trajectory, and evaluate it on hard problems where repeated sampling struggles. We then go a step further and make concept generation trainable: a small concept generator is optimized with reinforcement learning so that its concepts maximize the downstream success of a larger, frozen answer generator. On hard mathematical reasoning problems, the trained concept generator substantially improves the answer generator's pass@k over naive repeated sampling at the same answer generation allocation, surpasses concepts drawn from much larger untuned models, and transfers to answer generators it was never trained against, including a model from a different family. A small model can thus be trained into an effective, reusable search policy for a much larger one.

• The paper introduces a methodology to train concept generators (CG) that produce problem-specific concepts to guide large language model (LLM) answer generators (AG), improving reasoning exploration.
• The paper shows that using a single-trajectory generation of diverse concepts significantly improves pass@128 on hard mathematical reasoning problems, achieving 39.20% vs. 19.00% for naive repeated sampling.
• The study finds that concept conditioning promotes broader coverage of solution paths. Evaluation on different LLM families confirms meaningful transfer of the trained CG policies.

The paper studies whether inference-time exploration for LLM reasoning can be improved by moving beyond independent repeated sampling. Its central claim is that repeated sampling explores primarily through token-level decoding noise and therefore often produces near-duplicate solutions. The proposed alternative is to generate problem-specific concepts, hints, or strategies first, then condition a larger answer generator on those concepts. The paper goes further by training the concept generator with reinforcement learning against the downstream performance of a frozen answer generator. In this formulation, exploration becomes a trainable search policy rather than an incidental consequence of sampling temperature (2609.26704).

Problem formulation and motivation

The paper frames exploration as a common bottleneck in both inference-time scaling and RL-based reasoning. In repeated sampling, a model produces many complete answers and a verifier retains any correct solution. If the model's probability of producing a correct answer is low, increasing the rollout budget may still yield many semantically redundant attempts. The authors therefore separate the selection of a reasoning direction from the execution of that direction. A concept generator (CG) proposes multiple high-level ideas, while an answer generator (AG) produces solutions conditioned on those ideas.

This decomposition is specifically targeted at hard problems. On easy instances, repeated sampling already leaves little headroom, whereas a hard-problem regime exposes whether concept conditioning can discover solution paths absent from the unconditional sampling distribution. The paper consequently constructs model-specific hard subsets for preliminary experiments and evaluates the principal method on DeepMath-103k-derived problems for which the Qwen2.5-32B AG obtains no correct answer in an initial batch of 128 rollouts. The held-out evaluation set contains 1,000 such problems, while an additional filtered subset of Omni-MATH~2 measures out-of-distribution transfer.

A useful theoretical perspective developed in the paper is that concepts alter the effective solve rate of the answer generator. If concept $c_j$ induces a conditional success probability $p_j$, and the fixed answer budget is split uniformly among $M$ concepts, then performance depends on the aggregate failure probability across concepts rather than on the average raw success rate alone. Consequently, adding a concept can be harmful when it receives rollouts but has lower yield than the existing concepts. This formalizes the dilution problem inherent in distributing a fixed budget over multiple guidance signals and motivates learning concepts that are both useful and diverse.

Reassessment of prior concept-guided sampling

The paper first revisits a previous concept-guided sampling protocol in which concepts are generated iteratively by one model and used to prompt another. The authors evaluate all 25 concept-generator/answer-generator combinations formed from Qwen2.5-Instruct models ranging from 1.5B to 32B parameters on MATH500.

The principal finding is that the reported advantage of the prior method largely disappears when repeated sampling is given exploratory decoding parameters. The original baseline used temperature $0.8$ and top-$p=0.5$, which substantially restricts diversity. With temperature $1.0$, top-$p=0.95$, and top-$k$ disabled, concept guidance slightly underperforms repeated sampling across nearly all model pairs. The baseline therefore reaches 96.2% pass@50 with a 32B AG, while concept-guided differences are negative for nearly every CG size.

The paper identifies two additional weaknesses in the iterative protocol. It generates concepts one at a time, introducing unnecessary sequential overhead, and produces approximately one concept per problem for the Qwen models. The latter is especially problematic because the proposed method is intended to diversify exploration. Even a Llama-3.2-3B concept generator that produces an average of 3.18 concepts per problem fails to outperform exploratory repeated sampling, indicating that the observed weakness is not attributable solely to low concept counts. Rather, the baseline decoding configuration is a major determinant of the earlier reported gains.

Single-trajectory concept generation

The authors address these limitations by prompting the CG to analyze the problem once and emit a list of non-duplicative, problem-specific concepts. Up to ten concepts are parsed from a single autoregressive trajectory. Each concept receives a portion of the fixed answer-rollout budget, preserving the computational comparability with repeated sampling.

Evaluation on model-specific hard subsets of MATH produces consistent improvements. With a 32B AG, a 7B CG raises pass@50 by 8.8 percentage points relative to exploratory repeated sampling. Across all CG/AG pairs, improvements reach as high as 9.7 points. Larger CGs generally perform better, but the result that a 7B CG can improve a 32B AG is already important: the search policy need not match the scale of the solver it controls.

The single-trajectory protocol also produces substantially more concepts, ranging from approximately four to the ten-concept cap. On a cross-family experiment using Llama-3.2-3B as both CG and AG, the method increases pass@50 from 25.2% to 26.1%, whereas the original iterative procedure had degraded the exploratory baseline. This result supports the claim that generating multiple concepts in one trajectory is not merely an implementation optimization; it changes the quality and breadth of the search distribution.

Reinforcement learning over concepts

The main methodological contribution is to train the CG for downstream utility while keeping the AG frozen. For each problem, the CG samples eight concept trajectories. Each trajectory contains an analysis and up to ten parsed concepts. The frozen AG then receives a total of 128 answer rollouts per trajectory, distributed as evenly as possible among the concepts. Answers are graded by a Qwen2.5-32B-Instruct judge, and the resulting correctness scores are aggregated into a trajectory-level reward.

The paper compares two reward functions. Max-of-max assigns a binary reward if any rollout conditioned on any concept is correct. This is directly aligned with pass@128 but provides a sparse, one-bit learning signal. Max-of-mean takes the highest per-concept accuracy among the concepts in a trajectory. It is not exactly aligned with uniform rollout allocation, since it favors the best individual concept and implicitly approximates oracle allocation, but it provides a substantially more informative continuous signal.

The CG is optimized with a GRPO-style objective using group-relative trajectory advantages. The entire trajectory receives the same advantage, rather than individual concepts receiving separate credit. This choice encourages the policy to produce useful concept sets but creates a potential credit-assignment problem: weak concepts may share credit with strong concepts, and concept quality could vary systematically by position.

The complete training loop is summarized below.

Figure 1: The concept generator samples concept trajectories, the frozen answer generator produces conditioned rollouts, a judge scores them, and the aggregated trajectory reward updates the concept generator.

The training procedure is computationally expensive. Each problem requires eight concept trajectories, 128 AG rollouts per trajectory, and corresponding judge calls, so reward estimation dominates optimization. In a representative training step, AG generation and judging account for approximately 94% of the 413-second step, whereas CG generation and the actor update require roughly 18 and 10 seconds, respectively. This cost is paid during training only; at inference, one 7B CG call is estimated to cost less than one quarter of a 32B AG rollout and under 0.3% of a 128-rollout answer allocation.

Main results on hard mathematical reasoning

The central results are obtained with Qwen2.5-7B-Instruct as the trainable CG and Qwen2.5-32B-Instruct as the frozen AG. The trained max-of-mean policy reaches 39.2% pass@128 on the DeepMath held-out hard set, compared with 19.0% for naive repeated sampling. At pass@64, it reaches 29.64%, compared with 11.35% for the baseline. The max-of-max objective also improves performance, reaching 35.3% pass@128, but is consistently weaker than max-of-mean.

Method	DeepMath pass@64	DeepMath pass@128	Omni-MATH~2 pass@64	Omni-MATH~2 pass@128
Naive repeated sampling	11.35%	19.00%	6.81%	11.29%
Generic prompt modification	14.54%	22.60%	6.93%	11.22%
Untuned 7B CG	20.62%	28.90%	9.64%	14.98%
Untuned 32B CG	23.77%	33.80%	10.18%	15.27%
Trained 7B CG, max-of-max	26.66%	35.30%	10.04%	15.27%
Trained 7B CG, max-of-mean	29.64%	39.20%	12.75%	18.60%

The improvement is not reducible to generic prompt perturbation. Ten fixed, problem-agnostic hints raise DeepMath pass@128 only to 22.6%, substantially below the 39.2% obtained by matched, trained concepts. The paired-bootstrap analysis reports a +20.2-point gain over naive repeated sampling at pass@128, with a 95% confidence interval of [+17.0, +23.4]. On Omni-MATH~2, the trained max-of-mean policy still improves pass@128 by 7.31 points, from 11.29% to 18.60%, demonstrating transfer beyond the training distribution.

The full allocation curves show that the gains become larger as the answer-rollout budget increases. This is consistent with a search-diversification mechanism: concept conditioning has only a modest effect on pass@1, but it substantially improves the probability that at least one of many attempts discovers a correct solution.

Figure 2: Pass@k curves show that trained concept guidance provides its largest advantage at larger answer-rollout allocations on both DeepMath and Omni-MATH~2.

The result also contradicts a simple scale-based explanation. The untuned 32B CG reaches 33.8% pass@128, while the trained 7B CG reaches 39.2%. Paired bootstrap intervals confirm a +5.4-point advantage for the trained 7B CG over the untuned 32B CG on DeepMath and a +3.3-point advantage on Omni-MATH~2, with both intervals excluding zero. RL training therefore provides a specialization benefit that exceeds the effect of increasing the untuned CG's parameter count.

Evidence that the method improves exploration

The authors conduct several analyses to establish that the CG is not merely solving the problem directly or inserting the final answer into the prompt. First, concepts are randomly deranged across held-out problems. Mismatched concepts yield 23.9% pass@128, only slightly above the 22.6% generic-prompt baseline, whereas matched concepts yield 39.2%. Thus, approximately 15 of the 20 percentage-point improvement over naive sampling is attributable to problem relevance.

Second, answer leakage is rare. Only 0.44% of generated concepts are flagged as containing the ground-truth answer, and only 3% of problems contain at least one such concept. The gains therefore cannot plausibly be explained by systematic transmission of final answers from the CG to the AG.

Third, the authors measure semantic diversity using the Vendi Score over embeddings of chains of thought with the problem statement and final answer removed. At the selected RBF bandwidth, the naive baseline has a score of 50.38, generic prompt modification 53.54, untuned 7B and 32B CGs 63.33 and 62.85, and trained max-of-max and max-of-mean 66.32 and 73.04. The max-of-mean policy consequently produces a 45% increase in effective distinct rollouts over naive sampling.

Figure 3: Vendi Score analysis indicates that trained concept guidance produces substantially more semantically distinct reasoning trajectories than repeated sampling.

The relationship between concept count and performance provides further evidence. With a fixed total budget of 128 AG rollouts, increasing the number of concepts from one to ten raises pass@128 from 22.70% to 39.80%. Intermediate counts of two, four, and eight concepts produce 28.10%, 32.70%, and 36.50%, respectively. Pass@1 remains approximately constant, between 2.15% and 2.44%. The divergence between stable single-shot accuracy and increasing pass@k demonstrates that the primary effect is broader coverage of solution paths rather than improved unconditional answer quality.

Transfer to a different answer-generator family

The trained CG is optimized exclusively against Qwen2.5-32B-Instruct, yet it also transfers to Llama-3.3-70B-Instruct. On the DeepMath hard set, the transferred policy reaches 34.3% pass@128, compared with 26.2% for naive repeated sampling and 28.9% for Llama's own self-generated concepts. It is also better than the untuned Qwen 7B CG, which reaches 31.3%.

On Omni-MATH~2, transfer is weaker but remains positive: the trained CG reaches 13.97% pass@128, compared with 12.50% for naive sampling and 11.94% for Llama self-concepts. These results support the interpretation that the CG learns a reusable semantic exploration policy rather than a policy narrowly overfit to the token distribution or prompt conventions of the Qwen2.5-32B AG.

This transfer result is particularly consequential under the paper's deployment assumptions. The AG may be frozen, expensive to fine-tune, or accessible only through an API. Training a smaller CG provides a mechanism for adapting the search behavior around such a model without changing its parameters.

Reward objectives and training dynamics

Max-of-mean outperforms max-of-max throughout the principal evaluation. Its advantage follows from the greater information content of the reward: max-of-max records only whether any of 128 rollouts succeeded, while max-of-mean preserves per-concept empirical success rates. This makes relative ranking among CG trajectories possible even when all trajectories contain at least one success or all fail under the binary objective.

The max-of-mean objective reaches 38.6% pass@128 after 200 training steps, compared with 35.3% for max-of-max. Both objectives eventually enter a performance plateau after several hundred steps. Longer training does not yield sustained improvement, and max-of-mean becomes unstable at approximately 750 steps, when entropy and KL divergence increase, overlong trajectories become truncated, and pass@128 temporarily falls to approximately 28%.

Figure 4: Training curves show rapid early improvement, saturation after a few hundred steps, and late instability for the max-of-mean objective.

The theoretical analysis makes the trade-off explicit. Max-of-max is an unbiased estimator of the deployed pass@k objective conditioned on a concept trajectory, but its variance and binary nature produce weak policy-gradient signals. Max-of-mean is biased toward the strongest concept and is not fully aligned with uniform allocation, but its lower-variance graded signal is empirically more effective. The superiority of max-of-mean therefore depends on a practical exchange between objective alignment and reward informativeness.

Limitations and open questions

The principal limitation is computational. Training requires $8 \times 128 = 1{,}024$ answer-generator rollouts and 1,024 judge calls per problem before each policy update. Although the cost is amortized over inference, the training procedure is substantially more expensive than ordinary SFT or direct RL on the CG alone.

Credit assignment is also coarse. Every concept in a trajectory receives the same trajectory-level reward, so the method cannot identify which concepts were causally responsible for successful answers. The position analysis mitigates this concern: per-concept accuracy ranges from 1.3% at the first position to 2.7% at the fifth, with later positions near 2%, and no position is inactive. Nevertheless, the analysis does not establish that all concepts contribute independently.

The fixed uniform rollout allocation is another unresolved issue. The formal analysis shows that an oracle would allocate more rollouts to concepts with higher conditional solve rates, whereas the implementation gives each concept approximately the same number of attempts. The gap between uniform and adaptive allocation is therefore an explicit source of lost performance. The paper leaves open whether a bandit-style allocation strategy can exploit concept-level feedback without introducing prohibitive sequential latency.

Evaluation also relies on an LLM judge from the same Qwen2.5-32B family as the primary AG. This creates a possible shared-model bias in both filtering and correctness assessment, even though the authors use a separate scoring prompt and perform leakage audits. The experiments are concentrated on mathematical reasoning, and the transfer results, while positive, do not establish whether the method behaves similarly in domains with weaker verifiers or less clearly decomposable search spaces.

Finally, the hard-set construction is based on a finite sample of 128 initial rollouts. A nominally zero-success problem can still have a nonzero underlying AG solve probability, which explains why the naive baseline recovers approximately 19.5% of the held-out instances under fresh sampling. The authors quantify this filtering variance, but it remains important when interpreting absolute pass@128 values.

Conclusion

The paper presents concept generation as a trainable semantic search policy for a frozen LLM answer generator. Its strongest evidence comes from hard mathematical reasoning problems: a trained 7B CG raises Qwen2.5-32B pass@128 from 19.0% to 39.2%, exceeds an untuned 32B CG, increases semantic rollout diversity, and transfers to Llama-3.3-70B without retraining. The results show that inference-time exploration can be improved not only by increasing rollout count or model scale, but by learning which problem-specific reasoning directions should receive that fixed budget. The remaining technical questions concern compute-efficient reward estimation, finer credit assignment, adaptive rollout allocation, judge robustness, and performance outside mathematical reasoning.