Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries

Abstract: Open-ended scientific discovery often requires repeatedly proposing and evaluating candidate solutions. LLM-based systems can support this process by generating and refining executable solutions from verifier feedback. Methods such as TTT-Discover use test-time training (TTT) to update the solution-generating LLM from verifier feedback, adapting its generation policy to improve subsequent proposals on the target problem. However, this becomes expensive when reliable execution requires a large model, since training must maintain gradients, optimizer states, and policy statistics while repeatedly generating long, structured outputs. It also complicates credit assignment: outcome-level verifier feedback must jointly evaluate the high-level strategy and its low-level implementation. In this work, we introduce Guidance-TTT, which separates these roles. A compact guidance model is trained at test time to propose high-level strategic changes, while a frozen execution model implements them as complete executable solutions. At each step, the system selects a promising previously discovered solution, proposes a change, executes and verifies it, and updates only the guidance model using an adaptive group-relative RL objective. This concentrates test-time learning on short strategic decisions while retaining the implementation capability of a substantially stronger model without adapting it. Without web access, Guidance-TTT produces strong solutions across four distinct domains: combinatorial optimization (Polyomino Packing), heuristic programming (AHC058), machine learning (Lasso), and GPU kernel optimization (TriMul). Across these tasks, it outperforms the best solutions reported in prior work while remaining competitive with state-of-the-art results on public online leaderboards. Code is available at https://github.com/Human-Agent-Society/reef/tree/guidance-ttt-support.

• The paper introduces Guidance-TTT, a hierarchical test-time RL framework that adapts a compact guidance model for strategic proposals while keeping a larger frozen executor for complete program generation.
• Guidance-TTT demonstrates improved performance across various domains, including a 39.8% increase in Lasso optimization and achieving 91.89 in Polyomino Packing, outperforming previous benchmarks.
• The methodology shows that adapting strategic guidance models can be more effective and cost-efficient than directly adapting large solution-generating models, while handling challenges like credit-assignment and linenic concentration.

Research problem and central claim

“Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries” (2610.06269) addresses a specific limitation of test-time training for open-ended program discovery. Existing systems either keep the solution-generating LLM frozen and use verifier feedback for archive search, or adapt the same model that must both select a strategic modification and implement it as a complete executable program. The latter coupling is expensive when reliable execution requires a large model, and it creates a credit-assignment problem: a low verifier score may reflect a poor strategic idea, an implementation error, or an incompatibility between the proposed change and the parent program.

The paper introduces Guidance-TTT, a hierarchical test-time RL framework that separates these roles. A compact guidance model proposes parent-conditioned, high-level modifications in natural language, while a substantially stronger execution model remains frozen and translates those modifications into complete programs. Verifier feedback updates only the guidance model. The paper’s central claim is therefore stronger than a simple parameter-efficiency result: test-time learning is more effective when it adapts strategic proposal generation rather than the model responsible for complete solution generation.

The distinction is between learning in solution space and learning in “thought space.” Guidance-TTT does not train on complete programs as the policy action. Its action is an explicit strategic instruction such as changing a search schedule, introducing a screening mechanism, specializing a kernel dispatch rule, or modifying a planning heuristic.

Figure 1: Guidance-TTT applies verifier feedback to a compact strategic guidance model while a frozen executor realizes the proposed modification.

This decomposition preserves the implementation capability of the large executor while concentrating adaptation on shorter, lower-dimensional decisions. It also changes the semantics of the reward: the guidance policy is optimized for changes that are both strategically useful and executable by the frozen model.

Guidance-TTT architecture and optimization loop

The method maintains a parent-linked archive of evaluated candidate programs. Each archive entry contains the exact executable solution, a verifier score, a compact solution summary, and a summary of the change that produced it. At every update, PUCT selects one or more promising parent solutions. For each selected parent, the guidance model samples multiple strategic modifications. The frozen executor receives the task description, the exact parent program, and one proposed modification, and returns a new complete candidate. The verifier evaluates the candidate, and the resulting sibling rewards update the guidance policy.

Figure 2: A PUCT-selected parent is modified by the guidance model, realized by the frozen executor, verified, and used to update only the guidance policy.

The default configuration uses 30 updates, eight selected parent groups per update, and 16 rollouts per parent, for 3,840 candidate attempts. The guidance model is Qwen3-8B or Qwen3-14B with rank-32 LoRA; the principal executor is frozen GLM-5.2. Thus, the expensive model generates and implements complete programs but does not carry test-time optimizer states or gradients.

Parent selection uses a maximum-child PUCT value, with an archive structure that supports evolutionary reuse while retaining lineage information. The guidance policy conditions on the task, the parent’s summaries, and its score, but the executor receives the exact source program. This distinction is important: summaries provide a strategic interface, whereas exact parent code prevents the executor from reconstructing the program approximately from a compressed description.

The policy objective is an entropic reward objective that emphasizes high-reward outcomes rather than merely increasing the expected reward. For each parent, rewards from the 16 sibling rollouts are converted into an adaptive group-relative advantage. Constant-reward groups are masked, and each complete rollout batch produces one policy update. The update uses a REINFORCE-style actor loss with importance weighting and a centered base-policy correction; it is explicitly not a clipped PPO objective. Only the guidance LoRA parameters are optimized, with Adam at learning rate $4\times 10^{-5}$ and no weight decay.

Figure 3: Verifier feedback drives archive search in self-evolution, complete-solution adaptation in TTT-Discover, and strategic guidance adaptation in Guidance-TTT.

This design makes a substantive assumption: the frozen executor must be capable of implementing useful strategic instructions. If the executor cannot reliably translate a proposed modification into code, guidance learning cannot recover the missing implementation capability. Conversely, if the executor is sufficiently capable, adapting it directly may waste computation on low-level behaviors that need not change across the target problem.

Evaluation across four discovery domains

The evaluation spans four structurally different tasks: Polyomino Packing, Lasso regularization-path optimization, AHC058 apple production planning, and TriMul GPU-kernel optimization. All discovery runs are offline and begin from executor-generated seed solutions rather than inherited source code or web search. The headline results are:

Task	Prior-work reference	Guidance-TTT result	Relative significance
Polyomino Packing	89.40	91.89	Exceeds a web-enabled four-agent CORAL result
Lasso	0.1243	0.1739	Approximately 39.8% higher than SimpleTES
AHC058	849,325,750	850,082,731	Exceeds the cited SimpleTES submission
TriMul latency	1131 $\mu$s	1129 $\mu$s	Slightly better than K-Search

The Polyomino result is particularly notable because the reported score of 91.89 exceeds both the cited four-agent, web-enabled CORAL score of 89.40 and the released human reference of 89.10. The Lasso result improves from 0.1243 to 0.1739 under the paper’s reciprocal geometric-mean-time metric, a 39.8% gain under the same correctness tolerance. On AHC058, the official submission score reaches 850.08 million, exceeding SimpleTES by approximately 0.09%. TriMul reaches 1129 microseconds, within 1 microsecond of the eligible public rank-1 Triton reference at 1128 microseconds.

These comparisons require careful interpretation. They mix historical baselines, public leaderboard entries, official submissions, and measurements taken under different execution environments. In particular, the AHC058 values are submission scores rather than matched repeated trials, and several ablation values are archive endpoints rather than fixed-program re-evaluation means. The results establish strong task performance, but they do not provide uniform uncertainty estimates across methods.

The discovered programs also exhibit recurring structural patterns rather than arbitrary code variation. The Polyomino solver combines skyline construction with simulated annealing over piece priorities and board width. The Lasso solver combines a homotopy-style path method with active-set factorization, segment-tree screening, and interpolation between path knots. The AHC058 solver uses phase-dependent scheduling, state-dependent valuation, and cascade-aware investment decisions. The TriMul kernel uses fused Triton front ends and shape-dependent dispatch between cuBLAS and persistent Triton kernels.

Figure 4: Guidance-TTT solutions retain strong algorithmic backbones while introducing targeted changes to search, computation, or adaptation allocation.

The implication is that guidance learning often discovers modifications to an existing computational backbone rather than replacing the entire algorithm. This is consistent with the method’s parent-conditioned design: the policy learns which structural change is promising for a particular implementation state.

Ablations and adaptation cost

The matched ablations support the paper’s claim that both strategic adaptation and the location of adaptation matter. With the same 8B guidance configuration and rollout budget, the full method substantially outperforms frozen guidance and direct solution training:

Configuration	Polyomino	TriMul latency
Guidance-TTT	91.89	1225.58 $\mu$s
Frozen guidance	84.85	1256.62 $\mu$s
Direct solution training	41.63	9664.91 $\mu$s
Additional parent context	82.12	1166.82 $\mu$s

Freezing the guidance policy while retaining the hierarchy reduces Polyomino from 91.89 to 84.85 and worsens TriMul latency from 1225.58 to 1256.62 microseconds. The hierarchy alone is therefore insufficient; test-time adaptation of the guidance model contributes materially.

The direct-solution-training control is substantially worse: 41.63 on Polyomino and 9664.91 microseconds on TriMul. This is a strong and somewhat contradictory result because the directly adapted model is asked to produce more information, not less. The outcome suggests that adapting a compact strategic policy can be more effective than adapting a model that must simultaneously identify a useful change, preserve program validity, and regenerate a complete solution.

Adding a second parent’s summary produces task-dependent behavior. It improves TriMul to 1166.82 microseconds but reduces Polyomino to 82.12. The paper interprets this as evidence that cross-parent mechanisms transfer more readily in TriMul than in Polyomino. However, this conclusion is not causal: the experiments use individual search trajectories, and the two tasks differ in both search structure and implementation constraints.

The cost comparison is favorable but not absolute. With a frozen GPT-OSS-120B executor on Polyomino, Guidance-TTT reaches 83.19 versus 83.72 for directly adapting the same 120B model through TTT-Discover. The reported cost is approximately $120 versus $283, a reduction of roughly 58%, although the Guidance-TTT score is slightly lower. This establishes a practical trade-off rather than unconditional dominance: freezing the executor can reduce adaptation cost substantially while preserving comparable, but not always equal, performance.

Guidance scale and search dynamics

Scaling the guidance model from 8B to 14B improves some saturated tasks but does not produce a monotonic benefit across all domains. With GLM-5.2 held fixed, the retained results are:

Task	8B guidance	14B guidance
Polyomino	91.89	89.80
Lasso	0.1739 $\pm$ 0.0015	0.1730 $\pm$ 0.0023
AHC058	849,063,233	850,082,731
TriMul latency	1225.58 $\mu$0s	1128.91 $\mu$1s

The 14B model improves AHC058 and TriMul but underperforms the 8B model on Polyomino and has a slightly lower Lasso fixed-solver mean. For Lasso, the 14B run reaches a higher archive peak, 0.23467 versus 0.21259, yet its fixed-solver mean is lower. This distinction shows why search-time archive maxima should not be treated as equivalent to robust final-program performance.

Figure 5: Larger guidance can increase archive peaks without increasing the fixed-program evaluation mean, as illustrated by the Lasso comparison.

The paper’s diagnostic probes suggest that the advantage of larger guidance is not simply improved code-condition compliance. In three TriMul explanation samples, the 8B model correctly described the relevant conditions without adding incorrect restrictions once, whereas the 14B model did so three times. The 8B model incorrectly imposed extra divisibility requirements on a persistent-contraction fallback that only required $\mu$2 not to be divisible by 128. Similar probes found 14B advantages in following objective definitions, distinguishing element-level sparsity from tile-level pruning, handling strict phase boundaries, and reasoning about coupled decisions.

Figure 6: The 14B guidance model more reliably describes the conditions under which discovered mechanisms apply, although the probe contains only three samples per model.

These diagnostic results are suggestive rather than statistically conclusive. They support the interpretation that additional guidance capacity can improve mechanism-level reasoning, but they do not isolate whether the observed performance differences arise from reasoning quality, sampling variance, archive dynamics, or model-specific generation behavior.

The search trajectories show a common pattern: substantial structural changes are followed by local refinements and plateaus. TriMul reaches its best search-time latency at update 25; Polyomino reaches its best at update 27. Lasso and AHC058 also show intervals in which valid children improve their selected parents without improving the global archive frontier.

Figure 7: Best-so-far trajectories exhibit structural improvements followed by local refinement and temporary plateaus.

The relevant learning unit is therefore not an abstract strategic proposal in isolation but an executable, parent-conditioned change. A proposal can be useful for one parent and ineffective for another, even when its textual description is similar. This property explains why the additional-parent-context ablation is unstable and why local parent-relative improvements do not necessarily translate into global discovery progress.

Robustness and transfer evaluation

The paper evaluates fixed discovered programs beyond the primary search environments. For Lasso, the Qwen3-8B plus GLM-5.2 solver passes all 15 cases in a held-out suite consisting of four real datasets and 11 stress cases. SimpleTES passes 13 of 15, failing two high-dimensional correlated stress cases. The 8B solver is slower on the four real datasets than SimpleTES, but it is the fastest discovered solver among those that pass every stress case.

On an extended suite of 11 real datasets, both the 8B and 14B Guidance-TTT solvers pass all cases. On the nine datasets where SimpleTES also passes, the 8B solver is faster on all nine, with a 1.28-times geometric-mean speedup. The 14B solver is faster on seven of nine, but its large slowdown on RCV1 reduces its geometric-mean speedup to 1.05 times. Thus, the smaller guidance model produces the more consistent transfer result in this evaluation.

For TriMul, the selected kernel is evaluated unchanged under Triton 3.4.0 and 3.6.0. Its latency is 1124 microseconds and 1072 microseconds, respectively, compared with 1129 microseconds under Triton 3.3.1. The kernel remains the fastest automated discovery result among the compared methods in each tested compiler environment, although the public rank-1 submission remains faster at 1112 and 1062 microseconds under the two out-of-distribution compiler versions.

These experiments strengthen the claim that the discovered programs are not merely overfit to a single timing or input configuration. They do not, however, establish broad distributional robustness: the transfer tests use fixed, selected solutions and do not continue search under the new environments.

Limitations and open questions

The paper identifies two central failure modes. First, verifier feedback conflates guidance quality with execution quality. In Polyomino, 404 of 3,840 candidates are invalid, primarily due to compilation or parsing errors; in TriMul, 1,390 candidates fail, mostly correctness tests. A useful strategic proposal can receive zero reward because the executor realizes it incorrectly.

The paper gives concrete paired examples. Two Polyomino proposals request a similar density-adaptive simulated-annealing mechanism; one fails because generated calls precede required variable declarations, while another compiles and reaches 91.3173. Analogous cases appear in TriMul, Lasso, and AHC058. These examples establish that the reward is a joint measure of strategy and realization. The entropic objective correctly favors realizable high-reward guidance, but it does not identify whether a low reward reflects bad guidance or execution failure.

Figure 8: Execution errors create ambiguous negative feedback, while archive-policy interaction can concentrate search within a narrow lineage.

Second, PUCT selection and policy updating can concentrate both search and learning around successful lineages. After Polyomino reaches its global best at update 27, updates 28–30 produce 341 valid candidates and 17 parent improvements, but no global improvement. In TriMul, 24 of 530 valid candidates improve their selected parents after the update-25 frontier, yet none improves the global best. The selected parents descend from a common ancestor, and subsequent changes remain within the same algorithmic family.

This concentration is not equivalent to terminal failure. Lasso improves again after a plateau, and AHC058 also resumes progress. Moreover, AHC058 includes 167 HTTP 402 failures during one analyzed plateau, so service interruptions confound attribution. The experiments therefore reveal mechanisms of credit ambiguity and lineage concentration, but they do not quantify how much each mechanism contributes to final performance.

Several broader methodological limitations remain. The principal scaling comparisons use one search trajectory per model size, preventing reliable estimation of between-run variance. Baseline comparisons are heterogeneous, combining historical measurements, public leaderboard entries, and official submissions. The method also depends strongly on executor quality, as shown by Polyomino scores of 91.89 with GLM-5.2, 85.32 with GPT-5.4, and 83.19 with GPT-OSS-120B under the same 8B guidance configuration. Finally, the paper does not establish a general rule for choosing the appropriate guidance context, archive diversity mechanism, or executor size for a new discovery task.

The open technical question is therefore specific: how should verifier feedback be decomposed into strategic value and realization quality when the executor is itself a stochastic, imperfect compiler from guidance to executable programs? Closely related is whether diversity-preserving archive policies can improve global discovery without reducing the exploitation benefits of PUCT.

Conclusion

Guidance-TTT separates test-time strategic adaptation from complete-solution generation. A compact LoRA-adapted guidance model learns which parent-conditioned changes to propose, while a stronger frozen executor implements and verifies them. Across Polyomino, Lasso, AHC058, and TriMul, the method obtains strong results, including 91.89 on Polyomino, a 39.8% improvement over the cited Lasso reference, 850.08 million AHC058 points, and 1129-microsecond TriMul latency.

The ablations support the paper’s main methodological claim: learning strategic guidance can be substantially more effective than directly adapting a solution-generating model, while also reducing adaptation cost. The evidence is qualified by heterogeneous baselines, single-trajectory scaling comparisons, executor dependence, and unresolved credit assignment when implementation failures obscure guidance quality. Nevertheless, the work establishes a technically coherent framework in which test-time RL adapts a small policy over executable strategic changes rather than retraining the full program-generating system.