Persistence of Token-Level Attribution for Distributed Behaviors

Determine whether the token-level attribution signal observed for a single tied-embedding trigger survives for behaviors distributed across an entire response, such as sycophancy.

Background

The paper finds that the gradient of the specific trigger token aligns with an in-context behavior target across all three experimental seeds, whereas the gradient of the whole response does not reliably localize the behavior. The authors therefore identify token- or position-level attribution as a potentially more promising approach for tracing behavior-relevant spans in online reinforcement learning.

The reported positive result uses a single behavior token with tied input and output embeddings. It remains unresolved whether the same attribution signal will persist when the behavior is distributed across many response tokens and lacks a single surface-markable trigger, with sycophancy given as an example.

References

The caveat is that our trigger is a single tied-embedding token; whether the signal survives for behaviors spread across a response, such as sycophancy, is open, and is exactly where a cause that does not look like the behavior would make attribution worth its cost.

— Which Rollout Taught It That? BehaviorTrace and the Limits of Training-Data Attribution in Online RL  (2610.10422 - Nautiyal, 7 Oct 2026) in Discussion, paragraph “Future work: the token-level lead”