Localizing RL-Induced Tool Use to a Single Crosscoder Feature

Abstract: Fine-tuning through RL reshapes the internal representations of LLMs to enable agentic behaviors such as tool use, yet the mechanistic basis of these changes remains poorly understood. While RL substantially improves structured tool-call generation, it is unclear which features emerge, which are preserved, and whether identified features can be leveraged for retraining-free behavioral control. In this work, we show that $\textit{Dedicated Feature Crosscoders (DFC)}$ isolate a compact set of RL-specific features that mediate tool-calling capability in $\texttt{Qwen2.5-3B}$. Across a $48$-crosscoder hyperparameter sweep, encode-decode reconstruction improves the RL model's tool correctness by $+31.1 \pm {9.7}$ pp and passively transfers tool-calling ability to the frozen base model by $+6.8 \pm 5.0$ pp which we call a $\textit{capability spillover}$. Our findings show that DFC partitioning concentrates RL-introduced capability into a minimal, steerable feature set that enables runtime behavioral control of agentic LLMs.

• The paper demonstrates that RL-induced tool use is localized to a minimal set of A-exclusive features using DFC partitioning.
• It introduces Dedicated Feature Crosscoders that partition shared and exclusive subspaces, validated through geometric and metric analyses.
• Single-feature steering achieves up to +65.0 percentage points improvement, highlighting practical avenues for training-free runtime control.

Isolating RL-Induced Tool Use Through Crosscoder Feature Partitioning

Introduction

This paper systematically investigates the mechanistic substrate underlying reinforcement learning (RL)-induced tool use in LLMs, focusing on Qwen2.5-3B fine-tuned with ToolRL. The authors leverage Dedicated Feature Crosscoders (DFCs) to partition the features of jointly sparse-encoded representations into model-exclusive and shared subspaces in order to localize the capability differences introduced by RL. Central to this analysis is the demonstration that RL-induced behavioral changes—specifically, structured tool-call generation—can be attributed to a sharply delimited set of features, enabling precise, retraining-free behavioral control. This work advances the agenda of mechanistic interpretability and runtime steering for agentic LLMs.

Methodology and Experimental Setup

The study compares two models: the base Qwen2.5-3B and an RL-fine-tuned Qwen2.5-3B (ToolRL-Qwen2.5-3B). A suite of 48 crosscoder configurations, spanning conventional Crosscoders and DFCs with systematic variations in sparsity, partition allocation, and regularization, is evaluated. DFCs enforce feature exclusivity via partitioned dictionaries and gradient masking—distinguishing between A-exclusive (RL-induced), B-exclusive, and shared subsets—while conventional Crosscoders lack such guarantees, permitting entangled feature allocations.

Behavioral metrics include format accuracy, tool correctness, and a composite overall score. Targeted neuron steering is performed based on features ranked by task-specific selectivity (Cohen's $d$), with performance modulated through direct manipulations in the DFC-induced activation space.

Architecture and Geometric Analysis

The DFC architecture is designed to disambiguate the representational loci of RL-induced capability. Feature partitioning, enforced by gradient-masking, is shown—through UMAP projections of decoder directions—to yield geometrically distinct partitions in the DFC case, as opposed to the mixing observed in conventional Crosscoders. This architectural constraint thus reifies the claim that RL-specific signal can be sequestered into an interpretable, steerable subspace.

Figure 1: Schematic of crosscoder architectures, contrasting unpartitioned Crosscoders with partitioned DFCs that explicitly allocate A-exclusive, B-exclusive, and shared feature sets for joint sparse decomposition.

Figure 2: The decoder UMAP comparison between DFC (sharp partitioning) and CrossCoder (no separation) at matched hyperparameters, confirming that architecture, not label imbalance, enforces separability.

The empirical geometry—quantified via metrics such as silhouette coefficients, $k$-NN purity, and adjusted Rand index—underscores that the DFC's explicit partitioning imposes nontrivial structural separations on encoded features, elevating the interpretability and precision of downstream mechanistic attributions.

Reconstruction and Capability Spillover

DFC-based encode-decode reconstruction substantially increases tool-calling ability in the RL model, with tool correctness gains averaging $+31.1\pm9.7$ percentage points. Notably, the same reconstruction, when applied to the frozen base model (Model B), results in a nontrivial $+6.8\pm5.0$ percentage point improvement in tool correctness, without any fine-tuning—a phenomenon termed "capability spillover." This spillover is statistically robust (exact-binomial sign test $p \ll 10^{-10}$) and indicates that joint training delocalizes behavioral capacity, contradicting the DFC ideal of perfectly isolating capability in exclusive partitions. Notably, surface-form fidelity (format accuracy) does not spill over, indicating a separation between semantic intent and surface structural implementations in the DFC representation.

Steering and Single-Feature Saturation

Targeted sparse-feature steering clearly demonstrates that RL-induced tool-use capability condenses into a minimal set of A-exclusive features. In DFCs, manipulating a single A-exclusive neuron suffices to saturate behavioral gains, with $\Delta$ tool correctness reaching $+65.0$ percentage points. This is not the case for shared or CrossCoder cases, where many features must be steered to achieve similar performance enhancement.

(Figure 3)

Figure 3: Example before/after steering output for ToolRL, showing the effect of a single A-exclusive feature intervention—transforming an ambiguous generic response into a correctly formatted and semantically accurate tool call.

Steering B-exclusive features has no effect on RL-induced tool use in the RL-tuned model, and concurrent A-exclusive plus shared steering exhibits destructive interference—consistent with subspace orthogonality and nontrivial decoder orientation structures in the DFC.

Layerwise, this effect generalizes: the single-feature steering effect is present in the majority of layers probed, not just at a single layer, with mean best-cell improvement of $\Delta=+43.3$ percentage points.

Implications and Theoretical Impact

The results validate several key implications for mechanistic interpretability and practical control:

• Minimal Feature Sets: RL-induced behavioral capabilities condense to extremely sparse, interpretable feature sets under DFC partitioning, as evidenced by both geometric (UMAP, HDBSCAN) and behavioral ablation (saturation, interference) analyses.
• Delocalization and Side Channel: While DFCs concentrate the strongest behaviorally-relevant features, capability spillover indicates that underlying subspaces structure cannot be perfectly isolated; information relevant for tool calling persists in the shared partition, creating inference-time "side channels".
• Inference-Time and Gradient-Free Steering: The ability to robustly enhance or suppress agentic behaviors via activation-space interventions—without retraining—suggests DFC-based steering as a lightweight tool for runtime behavioral control, with obvious ramifications for LLM safety, alignment, and capability supervision.

Challenges remain, particularly in achieving perfect partitioning (as instead of acting as a "sink," the exclusive partition acts as a "filter") and extending these findings to other tasks or architectures. The results highlight the need for more sophisticated diffing tools, such as Delta-Crosscoders, for precise asymmetric capability localization.

Limitations and Future Directions

The study is restricted to a single model pair and task. The evidence for architecture-level differences in spillover is suggestive but statistically underpowered. Prompt diversity and scaling to larger architectures remain open challenges. The current scope for automated interpretability and autointerp is incomplete, mostly covering A-exclusive features, suggesting further investigations aimed at the shared partition will elucidate the structural basis for capability leakages.

Future work will extend DFC-based methods to knowledge-boundary discovery—the identification of features controlling commit/abstain decisions in LLM agents—and pursue architectures explicitly modeled to capture asymmetric capability deltas between models.

Conclusion

This work establishes that RL-induced tool-use capability can be attributed to, and controlled via, a compact set of interpretable features under DFC-based sparse decomposition; a single A-exclusive feature suffices to saturate behavioral transfer in most tested layers. However, spillover into shared partitions reveals structural limitations, underscoring that joint sparse model diffing identifies but does not perfectly sequester capability differences. These findings deepen mechanistic understanding of RL-induced behaviors in LLMs while opening avenues for runtime, training-free behavioral modulation and enhanced interpretability.