Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models

Abstract: Routing among LLMs trades response quality against serving cost, motivated by the reported gap between deployed routers and a per-instance oracle. Recent analysis shows that test-time resampling can recover per-instance selection headroom that no single-commit router captures; however, that guarantee holds only under an idealized oracle equipped with correctness labels and an unconstrained budget, neither of which a deployed system has. To the best of our knowledge, no previous work treats resampling the committed model and rerouting to an alternative model as competing uses of a single per-query cost budget. Therefore, this work formulates budget-aware test-time model selection: given a per-query budget and an imperfect verifier, allocate each unit of budget between resampling and rerouting so that expected correctness is maximized. An online resample-or-reroute (RoR) allocation policy driven by estimated marginal correctness per unit cost is proposed, and its behavior is grounded in the recoverability asymmetry between selection and sampling. Replay experiments on newly regenerated multi-draw correctness tensors from an eleven-model open-weight pool over four benchmarks of differing difficulty show that the proposed RoR policy attains a favorable cost-quality Pareto front relative to single-route, one-commit-router, budget-aware best-of-K, cascade, and random-allocation baselines for the tested pools, with the largest gains on the most heterogeneous benchmark; an ablation further shows the gains are verifier-gated, shrinking as verifier quality degrades, and robustness replays under a provider price vector and a label-free agreement verifier delineate where the conclusions carry over.

• The paper introduces Resample-or-Reroute (RoR), which uses posterior accuracy estimates and marginal correctness per cost to decide whether each additional budget unit should resample a model or test an alternative.
• RoR matches or surpasses budget-scalable baselines on three of four benchmarks, reaching 0.968 accuracy on GPQA-Diamond and reducing GSM8K cost by roughly 24–34% while achieving 0.993 accuracy.
• The results show that RoR’s gains depend on verifier quality: agreement verification works for open-ended tasks, while multiple-choice problems require stronger-than-consensus verification, and sequential execution increases latency relative to parallel sampling.

Motivation and problem statement

Model routing for LLMs is typically motivated by the gap between a deployed router and a per-instance oracle that always selects a correct model. Prior analysis, however, established two complications: part of that gap is single-draw label noise irrecoverable by any single-commit router, and the recoverable remainder can be reached without any router at all via test-time resampling (best-of-$K$ on one committed model) — but only under an idealized oracle with correctness labels and an unconstrained budget (2607.08665). The paper under review addresses the operational question this leaves open: given a per-query cost budget $B$ and an imperfect verifier, should a system spend budget resampling the model it already committed to, or rerouting to an alternative (possibly more expensive) model?

The contribution is framed as budget-aware test-time model selection: maximize expected correctness subject to a per-query cost constraint, treating each unit of budget as competing between one more draw of an already-used model and the first draw of a new candidate. The author claims this is the first formulation to treat resampling and rerouting as competing uses of a single explicit per-query budget under an imperfect verifier.

The RoR policy

The proposed resample-or-reroute (RoR) policy maintains a per-query posterior mean $$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$, where $\bar{p}_m$ is an offline train-split accuracy prior, $n_{im}$ and $w_{im}$ are draws and verified-correct counts for model $m$ on query $i$, and pseudo-count $s$ controls how strongly failures trigger rerouting. At each step it takes the action maximizing estimated marginal correctness per unit cost; under early stopping this reduces to selecting the most affordable model by $\hat{p}_{im}/c_m$. A UCB variant adds an exploration bonus scaled by inverse cost. Per-query overhead is $B$0 time and $B$1 memory — negligible relative to a single LLM call.

The policy is grounded in the recoverability asymmetry from the companion theoretical work: when a committed model's reproducible success probability is high, resampling has larger marginal correctness-per-cost than rerouting, so the greedy rule prefers resampling exactly where the asymmetry predicts.

Experimental design

Evaluation uses newly regenerated multi-draw correctness tensors: four benchmarks (GSM8K, MATH-500, GPQA-Diamond, HumanEval+), eleven open-weight models spanning eight pretraining lineages, $B$2 seed-aligned draws per (query, model) cell at $B$3. Policies are replayed offline against these tensors, making every reported number CPU-reproducible without model inference. Baselines include single-route, a one-commit learned router, budget-aware best-of-$B$4, a FrugalGPT-style cascade, random allocation, and a non-deployable oracle-allocation ceiling. Costs are proxied by parameter count in the main experiments, with robustness replays under real provider prices and deployable verifiers.

Main results

At matched mid-budget operating points, RoR dominates or matches every budget-scalable baseline on three of four benchmarks:

Policy	GSM8K acc	MATH-500 acc	GPQA acc	HumanEval+ acc
RoR	0.993	0.887	0.968	0.952
Best-of-$B$5 (budget-aware)	0.983	0.867	0.861	0.852
Cascade	0.992	0.847	0.926	0.952
Random allocation	0.992	0.846	0.706	0.959
Router (one commit)	0.976	0.784	0.566	0.817
Oracle ceiling	1.000	0.944	1.000	0.988

Three regimes emerge. On near-saturated GSM8K, the margin is chiefly cost: RoR reaches 0.993 at mean cost 9.2, roughly 24–34% cheaper than the cascade and best-of-$B$6, and 3.5× cheaper than single-routing the best model while being 2.7 points more accurate. On intermediate MATH-500, RoR leads the strongest baseline by 2.1 points at matched cost and the one-commit router by 10 points at 18% lower cost. On heterogeneous GPQA-Diamond — where pool specialists genuinely differ — rerouting matters most: +10.7 points over best-of-$B$7 and roughly +40 over either single-commit policy. One honest exception is noted: at the very lowest GPQA budgets (~cost 8), single-commit policies beat RoR's cheapest-first exploration until the budget affords a second draw (~cost 11).

On HumanEval+, the advantage concentrates at low budget rather than high-budget accuracy: with a near-perfect execution verifier and a high union ceiling (0.988), the undirected baselines catch up at mid budget (random allocation slightly exceeds RoR at 0.959 vs 0.952). The paper concedes this directly rather than claiming uniform dominance.

Sensitivity analyses support robustness: varying the prior pseudo-count $B$8 over an order of magnitude changes accuracy by at most 1.7 points; calibrating priors on 30% instead of 70% of queries costs at most 1.3 points; standard deviations across 20 draw orderings are small relative to reported margins (largest on GPQA, whose test half contains only 99 queries).

Verifier dependence and realistic verifiers

A central finding is that the gains are verifier-gated. Under the parametric degradation from perfect ($B$9) toward random selection ($$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$0), GPQA accuracy falls 0.968 → 0.710 → 0.675 as $$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$1 drops 1.0 → 0.8 → 0.6; at $$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$2 the margin over budget-aware best-of-$$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$3 slightly inverts on MATH-500 (0.794 vs 0.803) while persisting on GPQA (0.675 vs 0.648). Policies spreading budget across many models degrade fastest, since a weak verifier turns each extra candidate into a chance of selecting a wrong answer.

Two deployable-verifier instantiations delineate where the mechanism survives in practice:

• Agreement (self-consistency) verification works for open-ended answer spaces — RoR retains a lead on GSM8K (0.958 vs 0.942) — but is nearly uninformative on multiple-choice GPQA, where spurious consensus accumulates and the ordering reverses (0.552 vs 0.572), with RoR degrading further as budget grows.
• Partial test suites for code are highly reliable here (measured false-accept rate 1.0%); RoR under base-test gating nearly matches its perfect-verifier ceiling (0.897 vs 0.897 at $$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$4).

This yields a concrete deployment rule stated in the paper: agreement-based verification suffices for open-ended answer spaces, while multiple-choice tasks require a stronger-than-consensus verifier.

Cost realism and latency

Replaying under a provider-calibrated price vector (OpenRouter snapshot, July 2026) preserves the conclusions and reveals an additional effect: budget-aware best-of-$$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$5 weakens when price ordering no longer tracks ability, whereas RoR's per-query reallocation is less sensitive to the reshuffle. These numbers are specific to one provider snapshot, though the replay re-runs in minutes for any other price vector.

On latency, RoR is inherently sequential (2.5–3.6 round-trips per query depending on benchmark), higher than parallel best-of-$$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$6 but consistently fewer rounds than the cascade at equal-or-better accuracy, placing it Pareto-efficient among adaptive policies on the accuracy–latency trade-off. Notably, the UCB exploration variant is at best neutral and loses on GPQA at low budgets; greedy exploitation of the offline prior is recommended as the default.

Limitations and open questions

The paper is explicit about several constraints. The main experiments assume a reliable verifier with early stopping — precisely the setting where the recoverability asymmetry is fully available — so the headline gains should be read as conditional on verifier quality. Costs are proxied by parameter count, preserving ordering but not exact dollar ratios; batched and latency-constrained serving, where draws can be parallelized, is deferred. The offline replay does not capture prompt-adaptive behaviors such as revising prompts between draws. When the pool is homogeneous or the benchmark near-saturated, simpler budget-aware best-of-$$\hat{p}_{im} = (s\,\bar{p}_m + w_{im})/(s + n_{im})$$7 captures most of the benefit, limiting the case for full RoR to heterogeneous pools with trustworthy verifiers. Open questions left by the paper include instantiation of stronger learned verifiers such as process-reward models, and extension of the replay protocol to latency-constrained serving regimes.

Conclusion

This work formulates resampling and rerouting as a single budgeted decision, proposes a marginal-gain greedy allocation policy grounded in the recoverability asymmetry between selection and sampling, and demonstrates via reproducible replay on regenerated multi-draw tensors that the policy attains a favorable cost–quality Pareto front across four benchmarks — winning on cost where benchmarks saturate and on accuracy where pools specialize. The gains are demonstrably verifier-gated, and the robustness replays with real prices and label-free verifiers map out both where the conclusions carry over (real prices, open-ended answer spaces) and where they weaken (consensus verification on multiple-choice tasks).