Determine the origin of fixed-sentence activation control

Determine how the control observed in Experiment 2 arises, including how scores presented in token space causally influence the representations of the fixed sentence in activation space.

Background

Experiment 2 fixes the sentence that the model processes at every turn, thereby ruling out explanations based on selecting different text in response to feedback. Three of the ten tested models nonetheless exhibit small but statistically significant control effects on activation readouts. The paper does not establish the mechanism producing these effects.

The authors propose causal interventions as a way to reveal how feedback scores, which are presented to the model as tokens, influence the sentence’s internal representations. Understanding this mechanism could clarify whether the observed behavior reflects a particular form of activation control and could support the design of stronger activation monitors.

References

Our work leaves a number of questions unanswered. First, where does the control we observe in Experiment~2 come from? Causal interventions could reveal how the scores, which the model sees in the token space, influence the representations of the sentence in the activation space.

— Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs  (2610.20179 - Baldelli et al., 14 Oct 2026) in Section 5, paragraph “Future work” (Discussion and conclusion)