PaT: Planning-after-Trial for Efficient Test-Time Code Generation

Abstract: Beyond training-time optimization, scaling test-time computation has emerged as a key paradigm to extend the reasoning capabilities of LLMs. However, most existing methods adopt a rigid Planning-before-Trial (PbT) policy, which inefficiently allocates test-time compute by incurring planning overhead even on directly solvable problems. We propose Planning-after-Trial (PaT), an adaptive policy for code generation that invokes a planner only upon verification failure. This adaptive policy naturally enables a heterogeneous model configuration: a cost-efficient model handles generation attempts, while a powerful model is reserved for targeted planning interventions. Empirically, across multiple benchmarks and model families, our approach significantly advances the cost-performance Pareto frontier. Notably, our heterogeneous configuration achieves performance comparable to a large homogeneous model while reducing inference cost by approximately 69\%.

• The paper introduces Planning-after-Trial, a trial-first pipeline that uses verified Best-of-5 generation and invokes recursive decomposition only after all candidates fail.
• PaT raises Qwen3-4B average Pass@1 to 83.13% at 4.85× relative cost, outperforming FunCoder’s 81.18% at 8.31× and achieving about 2.4× higher return on extra cost.
• A heterogeneous Qwen3-8B generator with a Qwen3-32B planner reaches 87.39% Pass@1 at 31% of homogeneous 32B cost, showing that adaptive escalation can approach frontier quality efficiently.

Motivation: the cost of unconditional planning

Test-time scaling methods for LLM-based code generation—hierarchical decomposition pipelines such as FunCoder and CodeChain, and iterative repair loops such as Reflexion—improve Pass@1 but at a substantial inference-cost premium. The authors identify the root cause as a rigid Planning-before-Trial (PbT) policy: every problem, regardless of difficulty, pays for an explicit decomposition step. On foundational benchmarks this is largely wasted compute; a Qwen3-4B model with standard few-shot inference already solves roughly 76% of HumanEval/MBPP problems directly, so only a minority of instances genuinely require planning. The paper's motivating observation is stark: FunCoder applied to a 4B model costs more than standard inference with a 32B model while performing worse, meaning prior state-of-the-art test-time methods can lose to plain parameter scaling on a cost-performance basis.

The PaT policy

Planning-after-Trial (PaT) inverts PbT. Given a specification $x$, a generator $M_G$ first performs a Best-of-5 trial (temperature 0.8), verified against a generated test suite $\mathcal{T}(x)$ in a sandboxed Python runtime. If any candidate passes all tests, the pipeline terminates immediately with zero planning overhead. Only when all candidates fail does the planner $M_P$ decompose $x$ into subproblem specifications $\{x_i\}$, which are solved recursively under the same trial-first policy. Successful subsolutions are composed into a parent-level candidate and re-verified; on failure, the planner is re-invoked with previously successful helpers as context, enabling component reuse. A plateau rule halts recursion when the pass count fails to improve ($p^{(t)} \le p^{(t-1)}$), returning the best-known solution.

Two design choices deserve emphasis:

• Strict success criterion over consensus scoring. Unlike CodeT-style consensus selection—which is ill-suited because PaT needs a binary trigger rather than a ranking—the method accepts a candidate only if it passes all tests. This is justified empirically: self-generated test suites (averaging 6.7 cases per problem) are entirely error-free for 63.4% of HumanEval problems under Qwen3-4B, with severe noise (3+ false positives) confined to ~10% of problems.
• Failure as a grounded escalation signal. Whereas FunCoder's prompt asks the model to conditionally decompose based on its own subjective complexity assessment, PaT's prompt states that direct implementation has verifiably failed and mandates decomposition. The authors attribute PaT's more aggressive and better-targeted decomposition behavior on hard problems (e.g., 71.2% decomposition rate on Expert-level xCodeEval versus 25.4% for FunCoder) partly to this imperative framing.

Homogeneous results

Across Qwen3 (4B–32B), Llama-3.1-8B-Instruct, and DeepSeek-Coder-V2-Lite, evaluated on HumanEval/MBPP with EvalPlus variants and 500 Codeforces-sourced xCodeEval problems, PaT outperforms Standard, Best-of-N, CodeT, and FunCoder in nearly all configurations. Representative results on the foundational benchmark average:

Model	Method	Avg. Pass@1	Rel. cost
Qwen3-4B	Standard	76.05	1.00
Qwen3-4B	FunCoder	81.18	8.31
Qwen3-4B	PaT	83.13	4.85
Qwen3-32B	Standard	83.35	1.00
Qwen3-32B	PaT	88.37	5.09

The headline claim is that PaT lifts Qwen3-4B to 83.13% average Pass@1—essentially matching 32B standard inference—at comparable relative cost, while beating FunCoder everywhere at roughly 60% of its cost. On xCodeEval, PaT again leads on performance across all models (+16.2 points over Standard for Qwen3-4B). Notably, for weaker models on this harder benchmark PaT's cost exceeds FunCoder's, which the authors frame not as inefficiency but as correct adaptive escalation: frequent failures legitimately warrant planner investment, and per-difficulty cost analysis shows PaT's average cost remains lower or comparable despite much higher decomposition rates on Hard/Expert tiers. A supplementary ROI metric (Pass@1 gain per unit of extra cost) puts PaT at 1.58 on average, about 2.4× FunCoder's 0.67 and 4.6× CodeT's 0.34. Wall-clock latency measurements corroborate the token-cost story: PaT incurs ~3× overhead over Standard versus ~5× for FunCoder.

Heterogeneous configuration

Because PaT invokes the planner rarely, the generator and planner roles can be assigned to different models: a small LLM handles high-volume generation, while a large model is reserved for infrequent planning interventions. The authors formalize this trade-off theoretically under a cost model with power-law scaling ($p_M = \alpha c_M^\beta$): they prove existence conditions under which a heterogeneous PaT strictly dominates homogeneous LLM-only inference in expected cost (Theorem 1), asymptotic dominance for complex tasks given a bound on decomposition overhead (Theorem 2), and a closed-form optimal generator cost $$c_s^* = \min\{(\beta D_L/\alpha)^{1/(\beta+1)}, c_L\}$$ (Theorem 3). Fitting the scaling law to their data yields a predicted optimum of $c_s^* \approx 0.114$, closely matching the actual 4B model cost of 0.11—an encouraging validation, though the fit rests on simplified assumptions (uniform problem complexity, fixed $M_G$0-way decomposition, idealized success/failure behavior).

Empirically, pairing a Qwen3-8B generator with a Qwen3-32B planner achieves 87.39% average Pass@1—a gap under 1% from the homogeneous 32B configuration—at 31% of its cost. The abstract reports this as approximately 69% inference-cost reduction for comparable performance. The heterogeneous curve also exhibits a steeper performance-per-dollar slope than the homogeneous one, indicating that upgrading only the planner is the most capital-efficient intervention. Latency analysis reinforces this: the 4B+32B heterogeneous setup runs at 11.90s per problem, faster than 32B standard inference (14.21s).

Limitations and open questions

The paper is candid about several constraints. Self-generated test cases introduce noise, which can cause both false failures (triggering unnecessary planning) and false successes; the strict pass-all criterion and plateau heuristic mitigate but do not eliminate this, and the authors concede that a noise-free oracle is unlikely. The heterogeneous configuration carries a higher static memory footprint, trading memory for reduced dynamic inference cost. The theoretical analysis depends on idealized assumptions (uniform complexity distribution, deterministic capability thresholds, power-law scaling) whose fidelity to real workloads is unverified beyond a single empirical calibration. An open question the paper leaves unresolved is whether learned policies for deciding when to plan—as explored by Paglieri et al.—could outperform the purely reactive execution-failure trigger, particularly in settings without executable verification signals.

Conclusion

PaT demonstrates that the dominant source of inefficiency in test-time code generation is not planning itself but its unconditional application. By gating decomposition on verified trial failure, PaT improves Pass@1 over strong PbT baselines while cutting their cost substantially, and its reactive structure composes naturally with heterogeneous model assignment, yielding near-frontier performance at roughly a third of frontier cost. The result reframes test-time compute allocation as a difficulty-adaptive resource-allocation problem, with execution feedback serving as a free, learning-free trigger signal.