Where Do CoT Training Gains Land in LLM based Agents?

Abstract: Chain-of-thought (CoT) reasoning is widely used in language-model agents, but prior work has shown that verbalized CoT is not always faithful and may instead reflect post-hoc reasoning, which means the model already knows the answer before reasoning. We therefore ask what CoT training is actually improving: is the model getting better at changing its action through generated reasoning, or is it getting better at predicting the action directly from the prompt? We study this question by comparing \emph{prompt actions} (predicting action without CoT) with CoT actions (predicting action with CoT). Across checkpoints, prompt-action quality improves substantially. While interacting with the environment, the relative advantage of CoT actions over prompt actions remains similar, showing that CoT training does not widen the advantage of CoT reasoning, and it helps to improve the quality of prompt actions. We further find that later checkpoints are less likely to revise the action in response to CoT, suggesting greater reliance on the prompt. Motivated by these patterns, we selectively mask action-token supervision on a fraction of training examples. This intervention improves out-of-domain generalization.

• The paper demonstrates that both CoT-conditioned and prompt actions improve in parallel during supervised training.
• It reveals that LLM agents predominantly rely on prompt tokens for action selection, diminishing the causal role of explicit reasoning traces.
• A reduced action-token supervision approach boosts the utilization of reasoning traces, enhancing out-of-domain generalization.

Diagnosing the Impact of Chain-of-Thought Supervised Training in LLM-Based Agents

Overview

This paper provides a rigorous analysis of how chain-of-thought (CoT) reasoning-based supervision shapes next-action prediction in LLM agents, especially in long-context environments. The central inquiry is whether CoT training enhances the agent’s ability to revise actions via generated reasoning traces, or whether it simultaneously reinforces direct action prediction from the prompt itself—potentially establishing a “shortcut” that weakens the intended role of explicit reasoning.

Experimental Framework

The diagnostic framework centers on a dual-mode action decoding paradigm: prompt actions are actions generated directly from the prompt (without generating reasoning), while CoT actions are predicted following the generation of a reasoning trace. The tested agent settings include complex sequential environments—ALFWorld, ScienceWorld, and BFCL—requiring multifaceted decision making with task instructions, environmental feedback, and interaction histories embedded in lengthy prompts.

Prompt and CoT actions are contrasted throughout training, and additional probes include conflicting-trace substitution (introducing reasoning traces incongruent with the prompt), as well as interpretability analyses via attention and gradient flow.

Parallelism of Gains: Prompt vs. CoT Actions

A strong empirical finding is the parallel improvement of both CoT-action and prompt-action accuracies during supervised fine-tuning. As the model's CoT-conditioned final actions improve, so does its capacity to predict the correct action directly from the prompt, and the success gap between CoT and prompt actions does not significantly widen.

Figure 1: Alignment between prompt/CoT actions and the reference action. Training gains are increasingly recoverable from the prompt alone, with the CoT-versus-prompt gap remaining flat.

Prompt/CoT-action consistency is increasing during training: superior checkpoints exhibit higher agreement between the two decoding modes, signaling convergence on the same action regardless of reasoning trace generation.

Figure 2: Prompt/CoT-action consistency during training. Higher-performing checkpoints show greater consistency across environments and model scales.

During online evaluation on unseen tasks, the CoT-versus-prompt action gap remains notably stable, both in trajectory-level success and local action quality as judged by GPT-5.4, indicating that improvements are largely prompt-driven even out-of-domain.

Figure 3: Trajectory success gap between CoT-action and prompt-action execution on ALFWorld remains flat across training.

Probing Causal Directionality: Shortcut Dominance

By substituting the model's generated reasoning trace with a conflicting trace (truncated at various lengths), the study directly tests which information source dominates action selection. Later checkpoints increasingly preserve the prompt-implied action, resisting revision by disagreeing reasoning traces. This anchoring effect is consistent across all tested environments and training regimes (SFT and RL), and is more pronounced for shorter, substituted reasoning traces.

Figure 4: Checkpoint-wise consistency under perturbed reasoning traces in ALFWorld. Later checkpoints display increased prompt-action preservation, indicating stronger shortcut reliance.

This behavior indicates that, rather than enhancing the agent’s ability to revise actions via reasoning, training makes the final action more anchored to the prompt and less susceptible to alteration by explicit reasoning.

Mechanism: Structural Biases in Long-Context Agents

Analysis of attention maps reveals that, during action prediction in long-context agents, the majority of attention mass is allocated to prompt tokens, with prompt tokens consistently dominating even among the most attended positions.

Figure 5: Prompt versus CoT attention during action prediction. Prompt tokens overwhelmingly attract attention mass at action time.

Gradient flow analysis corroborates this, showing the majority of the action-token supervision gradient is also focused on prompt tokens across all architectures and environments. This structural asymmetry is mainly due to the length and informativeness of prompts in agentic contexts—prompt tokens not only outnumber reasoning tokens, but often carry sufficient cues for action selection. Such prompt-dominance biases optimization, leading models to rely on cues from the prompt rather than rationales when under strong action-token supervision.

Training Intervention: Reducing Shortcut Reliance

To mitigate over-reliance on prompt-based shortcuts, the authors introduce reduced action supervision: masking the action-token loss for a subset of training samples, thereby weakening the direct optimization signal on final actions without reducing rationale supervision. This intervention leaves more “room” for CoT-based action revision.

Empirical results demonstrate significant improvements in out-of-domain generalization for ALFWorld and ScienceWorld, with the action-gap between CoT and prompt actions increasing under the intervention. This suggests greater utilization of reasoning traces, rather than direct prompt matching, when finalizing actions.

Figure 6: Mean evaluation score on OOD tasks under vanilla CoT supervision and reduced action supervision—showing marked generalization gains, notably for ALFWorld and ScienceWorld.

Figure 7: CoT-minus-prompt action gap under reduced action supervision becomes larger in several settings, indicating increased sensitivity to reasoning.

The masking-based intervention makes subsequent checkpoints less prompt-anchored under conflicting trace attacks, further supporting a reduction in shortcut dependence.

Figure 8: Consistency with the prompt action under perturbed reasoning traces (truncation ratio 0.1) remains flat, indicating reduced shortcut anchoring under the intervention.

Implications and Future Directions

Practical Implications:

CoT-supervision in long-context agents does not guarantee that deployed models rely on explicit reasoning traces when selecting actions. Instead, strong gradient and attention flow to prompt tokens can create robust prompt-to-action shortcuts, reducing the functional role of generated rationales. Deployments relying on explicit reasoning for interpretability, faithfulness, or generalization should not equate gains under CoT-training with increased causal efficacy of reasoning, especially in agentic tasks with information-rich prompts.

Theoretical Implications:

The findings underscore the importance of structural and optimization biases in sequential LLM environments. Shortcut learning, previously discussed in vision and generative modeling, is also prevalent in agentic LLMs, significantly driven by context length and the allocation of training loss.

Speculation:

Future research may focus on designing supervision regimes that explicitly encourage causal mediation of action by reasoning traces—possibly by counterfactual data augmentation, trace-sensitive losses, or architectural modifications that restrict shortcut pathways. There is also a need for more nuanced benchmarks that distinguish between action faithfulness to reasoning traces versus prompt-only cues.

Conclusion

The study systematically demonstrates that, in LLM-based agentic settings, conventional CoT training amplifies direct action predictability from the prompt rather than the causal use of explicit reasoning traces. This effect is structurally supported by prompt-dominated attention and gradient flow at action time. A targeted reduction of action-token supervision can mitigate shortcut reliance and improve generalization. These results highlight the necessity of finer-grained diagnostic and intervention strategies in the supervision of reasoning-augmented language agents to ensure the intended role of explicit step-by-step reasoning in decision making.