Sharpening Tax in Post-Training

Abstract: An emerging hypothesis about reinforcement learning (RL) post-training of LLMs is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@1), they often surpass their post-trained counterparts in solution coverage (pass@K) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.

• The paper shows that post-training large language models (LLMs) often improves immediate performance when sampling, but decreases the variety of reliably successful tasks when the sampling is repeated.
• Larger base models (e.g., those with 31+ billion parameters) often exhibit higher variability in successful tasks when compared post-training, meaning they find more successes over multiple attempts; Conversely, smaller models have less variation when compared before and after training.
• Post-Training increases sampling efficiency but also sees a resulting trade-off between task consistency, whereby policies maintain a high probability of discovering tasks but lose variability in alternative options that can be

The paper investigates whether post-training expands the set of agentic problems an LLM can solve or primarily concentrates probability mass on a smaller set of already available behaviors. Its central claim is that, across a broad evaluation of open checkpoint pairs, post-training improves immediate sampling efficiency and consistency but often reduces the value of repeated sampling. The authors call this reduction in test-time scalability the Sharpening Tax (2610.01509).

Research question and evaluation design

The study addresses a limitation in prior analyses of post-training: most evidence for distribution sharpening comes from mathematical reasoning and code generation, where base models have extensive pretraining exposure and where outcome-only verification can award credit to trajectories that arrive at a correct final answer for the wrong reasons. The paper instead evaluates process-centric agentic tasks in which intermediate tool calls and environment transitions determine success.

The main evaluation uses 14 base/post-trained checkpoint pairs from four model families—Gemma-4, Ministral-3, Qwen2.5, and Qwen3.5—spanning approximately 3B to 35B parameters or active parameters. The three benchmarks are:

• BFCL v4 multi-turn base split, evaluating structured function calling and stateful interaction;
• WebShop, requiring search and click actions in a simulated shopping environment;
• ACEBench, covering single- and multi-step tool-use tasks.

Each model-task pair is evaluated with up to 128 independent rollouts. The primary metrics are:

• pass@1, the single-rollout success probability;
• pass@K, the probability that at least one of $K$ rollouts succeeds;
• pass$^K$, the probability that all $K$ rollouts succeed.

The comparison gives the post-trained models their native chat templates and tool parsers, while base models receive a model-agnostic harness consisting of a system prompt, tool descriptions, a JSON tool-call format, parsing logic, and episode stop conditions. This harness is not task-specific and does not include solved demonstrations. The asymmetry is operationally motivated, but it is also an important part of the experimental design: the reported result concerns deployable base models with lightweight scaffolding rather than raw unformatted completions.

The overall study is organized around three linked components: empirical test-time scaling curves, a scalar diagnostic for the loss of scalability, and an adaptive rollout sampler intended to reduce that loss.

Figure 1: The paper’s evaluation framework connects post-training-induced sharpening, the Sharpening Tax diagnostic, and posterior-tempered group sampling.

Base models retain substantial agentic coverage

The first empirical result is that harness-equipped base models are substantially more capable as agents than their single-sample accuracy suggests. Post-trained models generally dominate at small rollout budgets, but base models often have steeper pass@K curves and eventually overtake them.

Figure 2: Post-trained models lead at small rollout budgets, whereas harness-equipped base models often catch up or surpass them under repeated sampling.

The result is particularly pronounced for larger backbones. For example, on WebShop with Gemma-4 31B, the base model exceeds 85% pass@128, compared with 56% for the post-trained model. The post-trained model is much stronger at pass@1, but its advantage is not preserved under sufficiently large parallel sampling. When the largest backbone from each family is aggregated across the three benchmarks, the mean base curve overtakes the mean post-trained curve at approximately $K=22$ and finishes clearly ahead at $K=128$. For the smallest backbones, the base curve does not catch up within the tested budget.

Figure 3: Increasing model scale moves the base/post-trained crossover to smaller rollout budgets.

This scale dependence is central to the paper’s interpretation. Smaller base models frequently lack basic agentic behaviors, such as reliably formatted tool calls, so post-training can genuinely repair failure modes and expand effective coverage. Larger base models, by contrast, already contain a wider repertoire of partially successful behaviors. Post-training can then improve the probability of selecting a high-reward behavior on the first attempt while suppressing lower-probability alternatives that remain useful under repeated sampling.

The harness itself has a large effect on base-model performance. Averaged over eight base models, it raises BFCL pass@1 from 5.59% to 15.63% and pass@32 from 15.19% to 49.13%. On WebShop, the corresponding gains are from 6.32% to 9.73% and from 39.48% to 49.15%. On ACEBench, where the base models are already more functional, the harness has little benefit. The authors also report that applying the same harness to post-trained models often harms performance, which means that the comparison is not simply between “raw base” and “optimized post-trained” systems: interface compatibility is itself policy-dependent.

Post-training bimodalizes task success

The paper attributes the coverage loss to a distributional transformation in per-task success probabilities. Let $p_x$ denote the probability that a policy succeeds on task $x$ in one rollout. Tasks with intermediate $p_x$ benefit most from repeated sampling: they sometimes fail and sometimes succeed, so additional attempts can recover failures. Tasks near $p_x=0$ or $p_x=1$ have little test-time scalability because they either remain unsolved or are solved immediately.

The empirical finding is that post-training moves probability mass away from intermediate task success rates and toward the two extremes. This is not merely a reduction in average diversity; it is a bimodalization of task-level solvability.

Figure 4: Gemma-4 31B exhibits a broad distribution of base-model task success rates and a sharply polarized post-trained distribution.

The effect is especially clear for Gemma-4 31B on WebShop. Among 128 rollouts per task:

• the base model has 87.6% of tasks in the intermediate “pass given compute” category;
• the post-trained model has only 30.0% in that category;
• always-solved tasks increase from 0.0% to 26.0%;
• always-failed tasks increase from 12.4% to 44.0%.

Thus, post-training both solves some tasks almost deterministically and destroys the residual solvability of others. The paper explicitly links the latter effect to amplification of incorrect modes already preferred by the base policy, rather than to a simple transfer of all probability mass toward successful trajectories.

The same qualitative pattern appears across model families, although its strength varies. Gemma-4 shows the strongest polarization and the largest average tax; Qwen3.5 shows the softest sharpening. Ministral-3 displays a scale-dependent transition: its 3B model benefits from post-training on difficult tool-use tasks, whereas its 8B and 14B models exhibit the more typical migration of intermediate mass toward the extremes.

Figure 5: Post-training trades coverage for consistency across model families and rollout budgets.

The coverage-consistency curves provide a complementary view. The post-trained policies maintain relatively high pass$^K$0, because their rollouts tend to agree. Base models have much lower consistency at large $^K$1, but their pass@K continues rising as repeated attempts explore more of their policy support. Consequently, post-training buys sampling efficiency and reliability conditional on the selected behavior, while sacrificing the diversity that enables recovery through retries.

The paper’s strongest interpretation is therefore conditional rather than universal: sharpening is beneficial when rollout budgets are small or when base models lack basic task competence, but it is detrimental when large models are paired with substantial test-time sampling budgets and the objective is broad task coverage.

Sharpening Tax as a test-time scalability diagnostic

To quantify the effect without inspecting an entire pass@K curve, the authors define the raw scalability of a policy up to budget $^K$2 as the area between its pass@K ceiling and its pass@k values for smaller $^K$3:

$^K$4

They also define a calibrated version,

$^K$5

which normalizes by single-shot failure headroom. The Sharpening Tax is the base-model value minus the post-trained value:

$^K$6

A positive tax means that post-training reduces the amount of success recoverable through additional rollouts. The raw form emphasizes absolute retry value; the calibrated form asks how much of the available single-shot failure mass is converted into success through test-time scaling.

The theoretical interpretation is particularly useful. If $^K$7 is the index of the first successful rollout, then $^K$8 equals the expected number of failed attempts before the first success, counting only tasks solved within the budget. A policy has high scalability when many tasks are neither immediately solved nor permanently unsolved. Under this interpretation, sharpening removes retry value by driving tasks toward $^K$9 or $K$0.

The empirical tax is widespread. At $K$1, the calibrated tax is positive in 36 of 42 model-benchmark combinations. It is almost uniformly positive for the largest backbones, while it can be negative for smaller models at low budgets or on BFCL, where post-training repairs tool-call formatting and other basic competencies.

Figure 6: An early Sharpening Tax estimate predicts later scalability loss and related consistency metrics.

The tax is also operationally predictable. An estimate based on eight rollouts, computed on one subset of tasks, predicts the tax at 32 rollouts on a disjoint task subset with Spearman correlation $K$2 across the 42 model-benchmark combinations. The early tax also correlates with the later consistency gap and with future coverage differences. This makes the metric useful not only as a retrospective summary but also as a low-budget diagnostic for deciding whether post-training has narrowed a policy’s test-time scaling behavior.

The paper further shows that the tax can support routing. A small pilot estimate of task-specific tax is used to choose between the base and post-trained policy for each task. The resulting router exceeds the post-trained model’s pass@128 in five of six aggregate settings, by as much as 13.6 percentage points on WebShop for the largest backbones, while exceeding the base model’s pass@1 in every setting. This result indicates that the base and post-trained policies are complementary: the post-trained policy is preferable for many easy or immediately solvable tasks, whereas the base policy is preferable for tasks where repeated sampling can recover rare successes.

Why global temperature scaling is insufficient

A natural intervention is to increase the temperature of the post-trained model at inference time. The paper finds that this partially restores coverage but does not solve the accuracy-coverage conflict. Higher temperature generally improves pass@K at larger budgets, but pass@1 does not improve and can decline substantially. The response is also non-monotonic across benchmarks and model families; the most sharply post-trained Gemma-4 31B policy is relatively insensitive to heating.

This result rules out a simple global flattening explanation. The problem is not merely that every prompt should be sampled more broadly. Easy prompts benefit from conservative sampling, while hard prompts require exploration. A single temperature applies the same intervention to both populations and therefore moves the model along an accuracy-coverage frontier rather than shifting that frontier outward.

Posterior-tempered group sampling

The paper proposes Posterior-Tempered Group Sampling (PTGS), which adapts rollout temperature separately for each prompt during RL training. For each prompt $K$3, PTGS maintains discounted success and failure counts and interprets them as a Beta posterior over the prompt’s current success rate. Thompson sampling produces a posterior draw $K$4, which is mapped to a temperature in $K$5:

• hard prompts with low estimated success rates are heated;
• easy prompts with high estimated success rates are cooled;
• prompts near a target success rate are sampled near the reference temperature.

A forgetting factor allows the difficulty estimate to track policy evolution. PTGS does not alter PPO or GRPO’s optimization objective and computes policy log-probabilities at the same temperature used for rollout generation, preserving an on-policy update.

Figure 7: PTGS uses a Bayesian per-prompt difficulty estimate to heat hard prompts and cool mastered prompts during RL training.

The mechanism is motivated by group-based RL. For a group of $K$6 rollouts, increasing a hard prompt’s success probability raises the probability that the group contains at least one success. For prompts with success probability below one-half, it also raises the probability of a mixed success/failure group, which is important for GRPO because group-relative advantages vanish for all-success and all-failure groups. For mastered prompts, cooling increases the frequency of all-success groups and suppresses unnecessary variation.

The paper evaluates PTGS by fine-tuning Qwen2.5-7B-Instruct with PPO and GRPO on Sokoban and FrozenLake. The results are consistently favorable:

Environment and method	pass@1	pass@128	Calibrated tax
Sokoban, PPO	46.5	55.0	0.094
Sokoban, PPO + PTGS	61.1	69.7	0.081
Sokoban, GRPO	36.5	55.3	0.081
Sokoban, GRPO + PTGS	39.1	72.5	0.025
FrozenLake, PPO	63.7	74.1	0.039
FrozenLake, PPO + PTGS	65.0	80.0	0.020
FrozenLake, GRPO	63.4	77.8	0.029
FrozenLake, GRPO + PTGS	67.3	81.2	0.028

The results support the paper’s claim that adaptive sampling can improve both immediate accuracy and repeated-sampling coverage. The strongest gain occurs for GRPO on Sokoban, where PTGS raises pass@128 from 55.3% to 72.5% and reduces the tax from 0.081 to 0.025.

The training dynamics indicate that this is not equivalent to globally increasing temperature. Fixed-temperature PPO either improves one metric at the expense of another or leaves the tax unchanged. PTGS maintains higher policy entropy, but its mean temperature can be above or below the baseline depending on the environment: it heats difficult prompts and cools easy ones simultaneously. The per-prompt posterior estimates correlate with realized group success at 0.81 on Sokoban and 0.71 on FrozenLake, and PTGS reduces the frequency of zero-success rollout groups that provide weak learning signals.

A further inference-time variant applies the same estimate-then-sample strategy to a frozen policy using pilot rollouts. On FrozenLake, PTGS approaches the accuracy of low-temperature sampling and the coverage of high-temperature sampling simultaneously, exceeding the global-temperature curve. On Sokoban, the improvement is smaller and appears only for some PTGS configurations, which appropriately qualifies the generality of the inference-time result.

Theoretical account

The theoretical analysis formalizes three aspects of the empirical findings.

First, the scalability functional is a waiting-time statistic: it measures the expected depth of the first successful rollout among tasks that eventually succeed. This explains why both immediate successes and permanent failures contribute little to scalability.

Second, in a stylized model where post-training moves a fraction $K$7 of tasks to success probability zero or one, raw scalability decreases proportionally with the collapsed fraction. The result does not depend on whether a task is moved to the successful or unsuccessful extreme: both eliminate the value of retries. A policy can therefore improve pass@1 while simultaneously reducing calibrated scalability, provided that some tasks are pushed to the never-solved state.

Third, PTGS improves the composition of RL rollout groups. Heating a hard prompt makes at-least-one-success groups more likely for any success probability increase and makes mixed groups more likely while the success probability remains below one-half. The latter is directly relevant to GRPO’s relative-advantage signal. These claims depend on conditional independence of rollouts given the prompt and on the empirical assumption that temperature increases success probability for the hard prompts it heats. The paper explicitly notes that higher temperature is not guaranteed to improve every hard prompt.

Limitations and open questions

The main checkpoint analysis is observational. The exact mixtures of SFT, preference optimization, RL, and other post-training procedures used to produce the public instruct checkpoints are not disclosed. Consequently, the study identifies a robust association between post-training and reduced test-time scalability but does not isolate the causal contribution of RL, data composition, optimization duration, or reward design.

The comparison also uses asymmetric scaffolding: base models receive a universal harness, whereas post-trained models receive native templates and parsers. This is defensible as a deployment-oriented comparison, and the authors report harness ablations, but alternative harness designs could change the crossover points and tax estimates. Similarly, the budget is measured in rollouts rather than tokens; although thinking-mode ablations preserve the qualitative conclusion, different accounting conventions could alter comparisons involving long internal reasoning.

The benchmark scope is limited to text-based agents and binary success signals. It excludes broader multimodal interaction, open-ended generation, and tasks where quality and diversity are continuous rather than binary. The theoretical model assumes conditionally independent repeated rollouts, while real agent trajectories can exhibit correlations caused by deterministic interfaces, shared context, parser failures, or systematic environmental bottlenecks. Finally, PTGS is evaluated on two relatively small RL environments with Qwen2.5-7B-Instruct and five runs per condition. Whether its gains persist under large-scale agentic RL, different reward structures, multimodal policies, or post-training recipes with explicitly diversity-preserving objectives remains unresolved.

Conclusion

“Sharpening Tax in Post-Training” (2610.01509) presents a coherent empirical and theoretical account of the accuracy-coverage tension in post-trained agent policies. Across 42 model-benchmark combinations, post-training usually improves pass@1 and consistency by concentrating task success probabilities near zero or one, while reducing the value of repeated sampling. The effect is strongest for larger models, where base models already contain a broad set of partially successful agentic behaviors. Sharpening Tax summarizes this loss, predicts it from small rollout budgets, and supports task-level routing between base and post-trained policies. PTGS provides a compatible training-time intervention that improves both single-shot accuracy and repeated-sampling coverage in the reported PPO and GRPO experiments. The paper’s principal methodological recommendation is that post-trained agents should be evaluated not only by pass@1, but also by the amount of test-time scalability they retain relative to their base checkpoints.