Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay

Abstract: Audited against causal ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals used to train LLM agents -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- identifies which steps causally matter better than chance. Existing evaluations grade these signals against annotated step correctness; we audit them against step contribution -- what re-sampling the policy's own alternatives at each decision point and rolling forward actually changes about the outcome -- and the two come apart. The ground truth itself is structured: causal contribution is sparse (30.5% of decision points where ground truth is defined carry measurable effect), and measurability is model-dependent -- the fraction of points with no policy-supported counterfactual differs by a factor of two (13.1% vs. 26.8%) between two similar-scale policies. The failure mode is identifiable: implicit credit echoes the policy's fluency (median rank correlation +0.75, replicating at +0.70 in a second family under a corrected instrument), while conditioning on the outcome adds no causal information (partial correlation -0.004, Qwen). A confidence-only router recovers pivotal steps at chance level, but cuts judge cost by 13.1% per turn (14.0% per trajectory). In a seven-arm pre-registered training experiment, no arm reliably outperforms the untrained policy, and the checkpoints' apparent instrument signature is fully explained by training dose -- sparser credit retains fewer examples, an order-of-magnitude spread in optimizer steps -- not credit content. Comparisons of credit rules must therefore match effective sample size, or they measure dose, not credit.

• The paper builds executed replay as causal ground truth and finds that implicit, judge-based, and policy-confidence signals identify pivotal steps no better than shuffled controls.
• Implicit credit primarily tracks action fluency rather than causal contribution, correlating strongly with policy probability while its partial correlation with causal impact is effectively zero.
• Training comparisons remain inconclusive because credit sparsity changes optimizer-step dose, and confidence-based routing reduces evaluation cost but detects pivotal steps only at chance levels.

Motivation and the audit question

Step-level credit signals — LLM-judge scores, outcome-conditioned log-probability ratios, and policy confidence — are moving from evaluation harnesses into agentic RL training loops, yet the assumption underlying all of them has not previously been tested: that the credit assigned to a step tracks what that step causally contributed to the outcome. This paper builds an external ground truth for contribution via executed replay in ALFWorld with Qwen2.5-7B-Instruct (50 trajectories) and replicates in a second family, Llama-3.1-8B-Instruct (27–28 trajectories). At every action turn, the environment is re-executed under the factual action and under $K{=}4$ distinct admissible alternatives sampled from the same policy snapshot, each rolled to terminal at least three times; the replay advantage $A_{\mathrm{replay}}(t)$ is the difference between the factual and alternative outcome means. Crucially, nothing in this construction consults any of the audited signals.

The paper distinguishes its object from existing evaluations: step-level benchmarks grade credit against annotated step correctness, whereas a training loop pays for contribution — a correct step can be causally inert, and an incorrect one pivotal. Concurrent work (CAR) validates a replay-based estimator on planted effects; C3 audits multi-agent credit against its own replay advantages; this paper instead grades signals already in training use against executed causal truth they do not participate in computing.

The structure of causal ground truth

The first substantive result concerns the ground truth itself. Under Qwen2.5-7B, only 30.5% of complete turns are pivotal ($n{=}1{,}768$), and dynamics are strongly absorbing ($\sigma_{\mathrm{floor}} = 0$ at 86.0% of turns); every zero carries an explicit resolution bound ($|\Delta p| \leq 0.632$ excluded at one-sided 95%), so zeros are "indistinguishable at achieved sampling resolution," never claims of no effect.

Measurability is model-dependent in both directions. The policy-supported counterfactual is undefined at 13.1% of intervened turns for Qwen versus 26.8% for Llama (Wilson intervals non-overlapping; factor 2.05), driven by probability-mass concentration rather than capability. Yet where counterfactuals exist, Llama carries more signal: 38.3% of its complete turns are pivotal against Qwen's 30.5%. This bidirectional dependence is a scope condition on replay methodology itself: neither rate is an environment constant.

Exclusions are not benign: turns lacking a counterfactual are systematically low-entropy, with included-minus-excluded mean policy log-probability of $-0.70$ for Qwen, so analysis sets are right-truncated on the fluency axis — a range restriction measured for Qwen only (the Llama counterpart cannot be recomputed after the instrument correction).

The fidelity audit: both families are placebo-level

Under the frozen verdict order, whose controls gate fires first, both audited families earn verdict H3: rank fidelity indistinguishable from their own marginal-matched shuffled control.

Family	$\hat\rho$ median [95% CI]	$n_{\mathrm{eff}}$	Own shuffle [95% CI]
Implicit (HCAPO $\rho_t$, Qwen)	0.0193 $[-0.109, 0.081]$	37/50	$A_{\mathrm{replay}}(t)$0
Judge (Qwen2.5-72B, Qwen)	0.1142 $A_{\mathrm{replay}}(t)$1	37 (32 comp.)	$A_{\mathrm{replay}}(t)$2
Implicit (Llama, corrected instrument)	$A_{\mathrm{replay}}(t)$3 $A_{\mathrm{replay}}(t)$4	21/28	$A_{\mathrm{replay}}(t)$5

Three estimators sharing no statistic or failure mode converge on the null: rank fidelity versus shuffled control, per-step sign agreement versus chance, and partial correlation of credit with the causal increment after conditioning out fluency. One judge sub-result does clear chance — per-step sign agreement of 60.4% [52.1, 68.2] — but it buys no concentration on pivotal turns: the implicit family's precision-at-pivotal lift is 0.940 $A_{\mathrm{replay}}(t)$6, entirely below the chance line of 1.0, while the judge's 1.000 interval contains it.

Two exclusions strengthen the reading: the null is not an artifact of unmeasurable ground truth (all quantities computed only over defined turns), and the scorers' outputs are strongly structured — just not by causal contribution. The Llama replication was run twice, once before and once after a chat-template defect was found and repaired; no verdict class moved under either instrument generation. A pre-registered cross-family transfer criterion held for the fluency correlation, though by essentially zero margin (lower edge 0.6482 against registered bound 0.6474).

Mechanism: credit echoes fluency

Implicit credit's structure is explained by a single pre-registered prediction that survived Holm correction: credit rises with the policy's own action probability (median rank correlation +0.75, replicated across families at +0.70). In a joint regression on fluency and causal increment, fluency carries roughly 2.3× the weight (standardized 0.955 vs. 0.402), and the credit–increment partial correlation is −0.004 ($A_{\mathrm{replay}}(t)$7): nothing causal survives conditioning. The fluency-dominance ratio is robust across templates and families (2.29–2.37). Outcome conditioning does not rescue the signal — the hindsight increment is uncorrelated with the causal increment.

A trajectory-level illustration makes the mechanism concrete: a looping policy repeating take cloth 1 four times shows scorer confidence exceeding 0.9999 on turns replay judges pivotal. A fluency echo therefore concentrates certainty exactly where trajectories stop progressing.

The training layer: dose, not doctrine

Seven arms (outcome-only, implicit, judge, shuffled/inverted controls, masked, reduced-resolution replay-truth), trained under common random numbers and evaluated on 128 held-out tasks, produced no arm reliably beating the untrained base (0.422), with all six pre-registered confirmatory comparisons inconclusive under the frozen ±3 pp band — which the design could not resolve anyway, since its minimum detectable effect was ≈11.8 pp due to an inverted frozen power proxy. The paper correctly reports this as inconclusive, not equivalence.

Checkpoints do separate distributionally: between-arm Jensen–Shannon divergence exceeds within-arm by 2.6× ($A_{\mathrm{replay}}(t)$8), and a pre-registered partition comparison showed the instrument partition separates checkpoints ($A_{\mathrm{replay}}(t)$9) while the information partition does not. But even this signature reduces to training dose: sparser credit drops more examples, buying fewer optimizer steps (112 to 8 per round under identical budgets), and controlling for realized parameter change removes the correspondence entirely (partial Mantel $n{=}1{,}768$0, $n{=}1{,}768$1) while credit sparsity itself certifies at +0.912. Comparisons of credit rules must match effective sample size, or they measure dose.

A substrate-repair rescue experiment also failed: with diagnosed defects corrected, outcome-only training diverged in every configuration tried, insensitive to learning rate and gradient clipping — indicating directional drift below the clip, not oversized steps. The null cannot be laundered into a substrate complaint.

Finally, a frozen confidence-only router recovers pivotal steps at chance recall (11.9% [9.4, 14.9], never exceeding matched random routing at any threshold) but cuts judge calls by 13.1% per turn / 14.0% per trajectory — granularities that move oppositely and must always be reported together. It works as a cost rule precisely because it fails as a detector: stuck, pivotal steps are high-confidence ones.

Integrity taxonomy

A secondary contribution is a four-dimension integrity taxonomy — identity, creation-time validity, persistence, diagnosability — derived from incidents in the authors' own pipeline, with the recurring failure being one check mistaken for another's answer. The taxonomy made a falsifiable prediction (the next incident would land on identity or diagnosability, the two untested defences) that was confirmed by the subsequent packaging-boundary failure; diagnosability remains a live open prediction. The most consequential incident was a semantic instrument defect — two byte-identical files constituting different instruments — caught by human notice, quantified by A/B trace, and resolved by full re-replay; verdicts survived, measurability rates moved materially (12.3 pp vs. 12.0 pp predicted). Notably, a pre-registered completeness gate fired and was overridden at the adjudication layer, disclosed here as evidence that thresholds reinterpreted when binding are not thresholds.

Limitations

Every claim is scoped to ALFWorld with binary outcome; the training experiments are single-family, single-scale (7B, LoRA, offline), with KS2 lacking a validated positive control — hence the inconclusive framing. Coverage on the corrected Llama instrument is 88.3%, missingness concentrated exclusively in long trajectories, so length-sensitive quantities inherit that caveat. Fidelity estimates run at small effective $n{=}1{,}768$2. The intended family-by-length interaction is unmeasurable in this data. Whether fluency echoes damage trained policies remains unresolved given the training study's power floor.

Conclusion

Audited against executed-replay causal ground truth, none of the step-level credit signals currently entering LLM-agent training loops identifies causally pivotal steps better than chance; implicit credit is largely a fluency echo, apparent differences among credit rules reduce to optimizer-step dose, and a confidence router is useful only as a cost mechanism. The paper leaves open whether these failures transfer beyond replayable single-agent tool environments, whether any credit rule can survive dose matching, and whether the corrected substrate can be stably trained at all.