---
title: 'GateProbe: Probing Techniques in Diverse Domains'
url: https://www.emergentmind.com/topics/gateprobe
type: topic
---

# GateProbe: Probing Techniques in Diverse Domains

Searching arXiv for recent papers relevant to "GateProbe" and the cited works.
Searching arXiv for "GateProbe" and related probing papers.
GateProbe is used in several distinct research settings to denote a gate-mediated probe that perturbs or interrogates a system. In Vision-Language-Action model analysis, GateProbe is a one-shot, gradient-based sensitivity metric that inserts a virtual scalar gate on each residual branch and ranks transformer blocks by contribution to downstream action loss within the Drop-Then-Recovery protocol [2606.27755]. In scanning gate microscopy, GateProbe is the use of a conductive, biased tip as a local electrostatic gate that perturbs electron transport in a buried two-dimensional electron gas and enables conductance imaging across strongly and weakly invasive regimes [1709.08559]. In silicon electronic devices, GateProbe is a localized single-electron quantum dot induced directly inside the device by the biased tip of a low-temperature scanning tunneling microscope, simultaneously short-range tunnel-coupled to a source reservoir and energy-controlled by a distant gate reservoir [2001.10225]. A related transformer-interpretability literature does not use the name GateProbe, but “Geometric Evolution Maps” provides methodology directly relevant to any gating-based probing approach that operates on concept probes in the residual stream, especially for layer selection, intervention placement, and direction specificity [2605.25848].

## 1. Terminological scope

The term spans at least three operationally different probe classes and one closely related methodology. What unifies them is the use of a gate or gate-like intervention to expose latent structure; what differs is the substrate, the observable, and the perturbative model.

| Usage | Gate or probe mechanism | Primary observable |
|---|---|---|
| VLA model analysis | Virtual scalar gate on a residual branch | Action-loss sensitivity and post-removal recoverability |
| Scanning gate microscopy | Conductive biased tip creating $U_\mathrm{tip}$ in a 2DEG | $G(x,y)$, $l_\mathrm{c}$, and $\sigma_G$ |
| Silicon STM device probing | Tip-induced single-electron quantum dot inside the device | Resonant tunneling, lever arms, field mapping, defect sensing |
| Residual-stream concept intervention | Settled concept direction chosen after rotation ceases | Separation suppression under directional ablation |

The resulting literature should not be collapsed into a single definition. In the VLA setting, GateProbe is an activation-space importance metric for structured pruning and recoverability analysis. In mesoscopic transport, it is a local electrostatic perturbation whose weak-limit response can be tied to properties of the unperturbed system. In atomic-scale silicon devices, it is a movable, gate-controllable single-electron spectroscopic element embedded within the device. The residual-stream work suggests a further extension: gate-based interventions can depend critically on where a representation has geometrically settled, rather than merely where a scalar separation score peaks [2606.27755] [1709.08559] [2001.10225] [2605.25848].

## 2. GateProbe as a virtual-gate sensitivity metric in VLA models

In "Drop-Then-Recovery: How Redundant Are Vision-Language-Action Models?" GateProbe is defined as a one-shot, gradient-based sensitivity metric that operates in activation space to quantify each transformer block’s contribution to the downstream action loss. Each droppable transformer block $B_i$ is written in residual form,
$$
h_i = h_{i-1} + F_i(h_{i-1};\, \theta_i),
$$
and GateProbe inserts a virtual scalar gate on the residual branch,
$$
\tilde{h}_i = h_{i-1} + \alpha_i \cdot F_i(h_{i-1};\, \theta_i).
$$
Setting $\alpha_i = 0$ is equivalent to dropping the block, whereas $\alpha_i = 1$ recovers the original model.

The importance score is the expected absolute sensitivity of the downstream task loss to that gate,
$$
I_{\text{gate}}(B_i) = \mathbb{E}_{x \sim \mathcal{D}} \left[ \left| \frac{\partial \mathcal{L}(x)}{\partial \alpha_i} \right|_{\alpha_i=1} \right].
$$
By the chain rule,
$$
\frac{\partial \mathcal{L}}{\partial \alpha_i}\bigg|_{\alpha_i=1}
=
\left\langle
\frac{\partial \mathcal{L}}{\partial h_i},\;
F_i(h_{i-1})
\right\rangle,
$$
with $F_i(h_{i-1}) = h_i - h_{i-1}$. The appendix also writes the layerwise score as
$$
S_l = \mathbb{E}\!\left[\left|\left\langle \frac{\partial \mathcal{L}}{\partial \mathbf{h}_l},\; F_l(\mathbf{h}_{l-1})\right\rangle\right|\right].
$$
The paper interprets this as a first-order Taylor approximation of the loss change when scaling a residual contribution to zero:
$$
\mathcal{L}|_{\alpha_i=0}
\approx
\mathcal{L}|_{\alpha_i=1}
-
\frac{\partial \mathcal{L}}{\partial \alpha_i}\bigg|_{\alpha_i=1}.
$$
Larger $|I_{\text{gate}}(B_i)|$ indicates a larger instantaneous loss increase under removal, so the block is deemed more important and is preferentially kept [2606.27755].

The implementation is explicitly hook-based rather than architectural. No actual parameters are added. Forward pre-hooks at each block’s input normalization layer cache $h_{i-1}$ and $h_i$, and $h_i$ is marked with `retain_grad()`. One forward pass computes $\mathcal{L}(o,p,a)$; one backward pass yields $g_i = \partial \mathcal{L}/\partial h_i$; the algorithm then accumulates
$$
S_i \leftarrow S_i + |\langle g_i,\; h_i - h_{i-1}\rangle|
$$
over calibration batches, and sets the final score by averaging over $M$ batches. The ranking criterion is descending importance, with lower-scored blocks removed first. This is presented as Algorithm 1, “Importance Profiling,” and its measured cost on $\pi0.5$ with $64$ calibration batches of batch size $8$ on one H200 GPU is $24.9\,\mathrm{s}$, compared with $25.4\,\mathrm{s}$ for IGIA, $26.1\,\mathrm{s}$ for Fisher, $77.2\,\mathrm{s}$ for Hessian trace, $\sim 9.3\,\mathrm{s}$ for CosSim, $160.8\,\mathrm{s}$ for PPL, $0.2\,\mathrm{s}$ for Magnitude, and $471.7\,\mathrm{s}$ for Taylor because of CPU accumulation overhead [2606.27755].

## 3. Drop-Then-Recovery, recoverability, and empirical asymmetry

Within Drop-Then-Recovery, GateProbe is used in Stage 1 to select which blocks to remove. Given a target drop count $K$, the method removes the $K$ least important blocks, physically short-circuiting the residual connection $h_i = h_{i-1}$ and discarding parameters $\theta_i$. Stage 2 then fine-tunes the reduced model against the downstream action objective,
$$
\theta^* = \arg\min_{\theta}\; \mathcal{L}_{\text{action}}\!\left(\pi_{\theta}(a \mid o, p),\; a^{\text{gt}}\right),
$$
with recoverability measured by post-recovery Success Rate on evaluation suites such as LIBERO, LIBERO-Plus, and RoboTwin 2.0.

The central empirical result is a strong asymmetry across pathways. Language backbones are described as highly redundant for standard robotic manipulation tasks, whereas vision and action pathways are substantially less tolerant to removal. On LIBERO, removing half of the language blocks improves OpenVLA-OFT from $95.0\%$ to $98.3\%$ under the same downstream fine-tuning budget, and retaining only two language blocks still recovers baseline-level performance at $95.1\%$. For $\pi0.5$, the corresponding values are $91.7\% \rightarrow 93.3\%$ when half the language layers are removed, and $91.0\%$ when only two language blocks are kept. By contrast, vision and action compression degrade performance much more severely: on $\pi0.5$, Vision Drop Half yields $80.8\%$, Vision Keep 2 yields $62.4\%$, and Action Keep 2 yields $26.2\%$. Under FLOPs-matched training on $\pi0.5$, Drop-12, which keeps $6$ of $18$ language blocks, reaches $93.7\%$, exceeding the $91.7\%$ baseline; even Drop-17, which keeps $1$ block, recovers to $91.0\%$ [2606.27755].

Against alternative importance metrics, GateProbe is reported as best or second-best, with its advantage growing under aggressive compression. On $\pi0.5$ across four drop levels, GateProbe averages $94.0$ at Drop-9, $92.1$ at Drop-12, $92.2$ at Drop-16, and $88.7$ at Drop-17. The paper notes that at Drop-16 it exceeds weaker baselines by as much as $+3.9$, and at Drop-17 by as much as $+4.3$. The comparison set includes parameter-space Taylor, IGIA, Fisher, Hessian trace, activation-space CosSim, PPL, and Magnitude. The stated rationale for preferring GateProbe is that it couples each block’s residual contribution directly to downstream sensitivity, requires only standard backpropagation, and is explicitly aligned with the action-learning loss rather than with static similarity or parameter magnitude [2606.27755].

The real-robot industrial results qualify the apparent redundancy of language depth. On an xArm 850 warehouse parcel-sorting setup, large language drops preserve near-baseline performance in near-training conditions: in Env 1, Baseline $63.3\%$, Drop-9 $65.0\%$, Drop-16 $55.0\%$; in Env 2, Baseline $75.0\%$, Drop-9 $71.7\%$, Drop-16 $66.7\%$. Under distribution shifts such as lighting changes, novel objects, and container changes, heavily dropped models degrade more than the full model. A common misconception would be to read high recoverability on LIBERO as evidence that language capacity is generally unnecessary; the reported out-of-distribution results instead indicate that redundancy is benchmark- and distribution-dependent. The paper’s limitations also emphasize that GateProbe is a first-order approximation, may miss higher-order block interactions, and can show increased variance with extremely small calibration sets [2606.27755].

## 4. GateProbe in scanning gate microscopy

In "Scanning gate experiments: from strongly to weakly invasive probes," GateProbe refers to the use of a conductive, biased tip as a local electrostatic gate that perturbs electron transport in a buried two-dimensional electron gas. The experiment uses an open resonator fabricated in a high-mobility GaAs/AlGaAs 2DEG located $90\,\mathrm{nm}$ below the surface, with density $n_s = 1.5 \times 10^{11}\,\mathrm{cm}^{-2}$, mobility $\mu = 3.35 \times 10^6\,\mathrm{cm}^2/\mathrm{Vs}$, and temperature $T = 270\,\mathrm{mK}$. A quantum point contact with lithographic gap $300\,\mathrm{nm}$ lies at the bottom of the scan area. A semicircular cavity gate of radius $2\,\mu\mathrm{m}$ and opening angle $90^\circ$, centered at the QPC, depletes the 2DEG below it for $V_\mathrm{cav} \le V_{\mathrm{depl,cav}} \approx -0.25\,\mathrm{V}$. The working point is $V_\mathrm{qpc} = -0.5\,\mathrm{V}$, placing the QPC on the third spin-degenerate plateau, $G \approx 3 \times 2e^2/h$. The tip is scanned $65\,\mathrm{nm}$ above the surface and the two-terminal linear conductance map
$$
G(x,y) = I_\mathrm{SD}(x,y)/V_\mathrm{SD}
$$
is recorded with $V_\mathrm{SD} = 100\,\mu\mathrm{V}_\mathrm{rms}$ and lock-in detection.

The tip voltage is calibrated through a dimensionless tip strength,
$$
u_\mathrm{t} = \frac{V_\mathrm{tip}-V_\mathrm{li}}{V_\mathrm{depl,tip}-V_\mathrm{li}},
$$
where the least-invasive voltage is $V_\mathrm{li} = +0.4\,\mathrm{V}$ and the tip begins to deplete the 2DEG at $V_{\mathrm{depl,tip}} = -5\,\mathrm{V}$, so that $U_\mathrm{t} = E_\mathrm{F}$ there. The sample has $E_\mathrm{F} = 5.3\,\mathrm{meV}$, and the tip potential is written
$$
U_\mathrm{tip}(\mathbf{r},\mathbf{r}_\mathrm{tip}) = E_\mathrm{F}\, u_\mathrm{t}\, v(\mathbf{r}-\mathbf{r}_\mathrm{tip}).
$$
For $U_\mathrm{t} \le E_\mathrm{F}$, COMSOL indicates an approximately Lorentzian profile; for $U_\mathrm{t} > E_\mathrm{F}$, a Gaussian tail better describes the potential. The appendix uses the Lorentzian example
$$
v(\mathbf{r}) = \frac{w^2}{r^2+w^2},
$$
whereas the numerical transport simulations adopt a Gaussian tip with width parameter $\sigma = 177\,\mathrm{nm}$ to suppress long-range tails.

The paper’s central distinction is between strongly invasive and weakly invasive GateProbe operation. In the strongly invasive regime, $U_\mathrm{t} > E_\mathrm{F}$, the tip induces a depletion disk and acts as a movable hard wall. With $V_\mathrm{cav}=0$, branched electron flow appears as strong thin features in $dG/dx$ maps; once the cavity is formed, branch patterns disappear and conductance modulations spread over the cavity area with characteristic $\lambda_\mathrm{F}/2$ scale. In the weakly invasive regime, $U_\mathrm{t} < E_\mathrm{F}$, the tip cannot efficiently backscatter through the QPC when $V_\mathrm{cav}=0$, and once the cavity is formed the map develops rich spatial structures inside the cavity region through gentle lensing rather than hard reflections. Near $V_\mathrm{li}$, the spatial pattern becomes insensitive to tip strength while the amplitude scales linearly with $u_\mathrm{t}$; the paper presents this as the hallmark of the perturbative regime.

The perturbative theory uses the first-order conductance correction
$$
G^{(1)} = - \frac{8 \pi e^2}{h} \, \mathrm{Im}\!\left[\mathrm{Tr}\!\left\{ r^\dagger\, t'\, \mathcal{V}^{2,1} \right\}\right],
$$
with
$$
\mathcal{V}^{2,1}_{a,a'}(\mathbf{r}_\mathrm{tip})
=
\int \mathrm{d}^2r \,
\psi^{(2)*}_{a}(\mathbf{r})\,
U_\mathrm{tip}(\mathbf{r},\mathbf{r}_\mathrm{tip})\,
\psi^{(1)}_{a'}(\mathbf{r}),
$$
and the standard Landauer expression
$$
G = \frac{2 e^2}{h}\, \mathrm{Tr}\!\left[ t^\dagger t \right].
$$
Within the weak regime, the amplitude of the SGM response scales linearly with $|u_\mathrm{t}|$, and a correlation analysis in simulation shows that the full response remains accurately predicted by first order up to $u_\mathrm{t} \approx 0.025$, corresponding through calibration to $|V_\mathrm{tip}-V_\mathrm{li}| \approx 0.135\,\mathrm{V}$. Inside this window, changing $V_\mathrm{tip}$ rescales the amplitude without modifying the spatial pattern, so $\delta G$ is determined by the unperturbed reflection and transmission matrices and the unperturbed scattering states.

The paper quantifies invasiveness using a characteristic length scale $l_\mathrm{c}$ and a modulation amplitude $\sigma_G$. From the one-dimensional power spectral density
$$
P(k) = |\widetilde{\delta G}(k)|^2,
$$
it fits
$$
f(k)= a e^{-b k} + c,
$$
defines
$$
k_\mathrm{c} = b^{-1}\ln(a/c), \qquad l_\mathrm{c} = k_\mathrm{c}^{-1},
$$
and finds that $l_\mathrm{c}$ peaks near $V_\mathrm{li}$ at approximately $120\,\mathrm{nm}$ before decreasing toward $\lambda_\mathrm{F}/2$, with measured $\lambda_\mathrm{F}/2 \approx 32\,\mathrm{nm}$. The perturbative regime is thus characterized by maximal $l_\mathrm{c}$, minimal $\sigma_G$, and linear amplitude scaling. The maximum $l_\mathrm{c}$ is disorder-limited and comparable to the disorder correlation length $\ell_\mathrm{dis} \sim 100\,\mathrm{nm}$. The practical guidance is correspondingly explicit: identify $V_\mathrm{li}$ from minimal $\sigma_G$ and maximal $l_\mathrm{c}$, operate within $|V_\mathrm{tip}-V_\mathrm{li}| \lesssim 0.135\,\mathrm{V}$, avoid $U_\mathrm{t} > E_\mathrm{F}$ if an interpretation in terms of the unperturbed system is desired, and treat large positive $V_\mathrm{tip}$ with caution because screening charges in the doping plane complicate the response [1709.08559].

## 5. GateProbe as a tip-induced single-electron probe in silicon

In "Scanned single-electron probe inside a silicon electronic device," GateProbe is realized by inducing a localized quantum dot directly inside a working multi-terminal silicon device with the biased tip of a low-temperature STM. A negative sample bias $V_s$ bends the silicon bands downward locally under the tip and forms a confinement potential that traps an electron beneath the hydrogen-terminated silicon surface. The dot is centered under the tip and therefore follows the tip laterally with sub-nanometer precision. The source reservoir is an Sb-doped $n^{++}$ region that supplies electrons to the dot at rate $\Gamma_\mathrm{in}$ when the dot level aligns with the source chemical potential; the dot then empties to the tip at rate $\Gamma_\mathrm{out}$. A separate Sb-doped $n^{++}$ gate reservoir, approximately $900\,\mathrm{nm}$ away across an undoped/p-type channel, shifts the dot level electrostatically via $V_g$.

The transport signature is resonant single-electron tunneling. A resonant step in $I(V_s)$, or peak in $\mathrm{d}I/\mathrm{d}V_s$, appears when the dot level aligns with the source reservoir, and the current through a single discrete level is
$$
I = e \left(\Gamma_\mathrm{in}^{-1} + \Gamma_\mathrm{out}^{-1}\right)^{-1}.
$$
The experiment is operated in the regime $\Gamma_\mathrm{in} \ll \Gamma_\mathrm{out}$, so the measured peak height primarily reflects $\Gamma_\mathrm{in}$. This was verified by varying the tip height by $\pm 20$–$25\,\mathrm{pm}$ at fixed lateral position, which left the peak height nearly unchanged. As the tip is moved toward the source by only a few nanometers, the QD–source separation decreases and $\Gamma_\mathrm{in}$ increases exponentially. The measured peak-current increase between $x=4.0$ and $13.0\,\mathrm{nm}$ follows
$$
\Delta I \propto \exp(\kappa x),
$$
with best-fit $\kappa = 0.2288\,\mathrm{nm}^{-1}$. Using the overlap model described in the supplementary material gives a QD wavefunction decay length
$$
\lambda = 2/\kappa \approx 9\,\mathrm{nm}.
$$
The WKB expression
$$
\Gamma \propto \exp(-2\kappa d), \qquad
\kappa = \sqrt{2m(\Phi-E)}/\hbar,
$$
is used as the tunneling picture behind the effective position-dependent barrier [2001.10225].

The electrostatics are expressed through capacitances $C_t$, $C_s$, $C_g$, and $C_0$, with total
$$
C_\Sigma = C_t + C_s + C_g + C_0,
$$
and lever arms $\alpha_i = C_i/C_\Sigma$. The dot level shifts as
$$
\Delta E = e(\alpha_g \Delta V_g + \alpha_t \Delta V_t + \alpha_s \Delta V_s + \alpha_0 \Delta V_0).
$$
In the STM biasing configuration, the effective bias lever arm is
$$
\alpha_0 = (C_\Sigma - C_s)/C_\Sigma.
$$
From the spatially resolved spectroscopy, the dot resonance in $V_s$ shifts by approximately $30\,\mathrm{mV}$ as $x$ increases by $9.2\,\mathrm{nm}$, and field-based calibration yields $\alpha_0 \approx 0.1$. Independent fits to a single-level tunneling model give $\alpha_0 \approx 0.14 \rightarrow 0.08$ as $x$ increases from $11.8$ to $17.8\,\mathrm{nm}$. In the stability diagram, the resonance slope is $\partial V_s/\partial V_g \approx 0.77$ for $V_g \lesssim -1.6\,\mathrm{V}$, which with $\alpha_0 \approx 0.1$ gives a gate lever arm
$$
\alpha_g = \alpha_0 \left(\partial V_s/\partial V_g\right) \approx 0.08.
$$
The capacitance ratios extracted in the supplementary material are
$$
C_s/C_\Sigma \approx 0.9,\quad
C_g/C_t \approx 3.4,\quad
C_s/C_g \approx 12,\quad
C_s/C_t \approx 39.
$$
These values show that the source dominates the dot’s self-capacitance, but the lithographic gate still provides substantial capacitive control even though the dot is only approximately $1\,\mathrm{nm}$ from the metallic tip.

The GateProbe is also used as a local electrometer. During spatial scans with $V_g=-1.6\,\mathrm{V}$ and $V_s$ near resonance, the planar electric field between source and gate is estimated as
$$
E_x \approx (V_g - V_s)/900\,\mathrm{nm},
$$
giving approximately $0.4\,\mathrm{MV/m}$. More generally, once $\alpha_0$ is calibrated, local electric fields can be mapped from resonance shifts through
$$
E(\mathbf{r}) \approx -\nabla[\alpha_0 \Delta V_s(\mathbf{r})].
$$
When the moving dot passes near localized charges, its level shifts by Coulomb interaction, producing spatially localized distortions of the resonance. The reported examples include a step edge and a doubly charged dangling bond; approaching a negative charge requires more negative $V_s$ to maintain resonance, allowing both defect polarity and location to be identified with atomic-scale precision.

The paper positions this GateProbe against scanning single-electron transistor microscopy and scanning gate microscopy. Unlike those approaches, where interaction is purely capacitive and long-ranged, the silicon GateProbe achieves short-range tunnel coupling between the probe dot and the device source reservoir while retaining sizable gate control, with $\alpha_g \approx 0.08$ and $C_g/C_t \approx 3.4$. The limitations are equally explicit: the probe necessarily perturbs the local potential through tip-induced band bending, can hybridize with localized states in heavily doped regions, and requires cryogenic temperature and UHV. Within those constraints, it provides atomic-scale spatial resolution, quantitative tunnel-coupling control, and in-device electrometry [2001.10225].

## 6. Relation to residual-stream concept probing: Geometric Evolution Maps

"Geometric Evolution Maps: Extracting Stable Concept Probes from Transformer Residual Streams" does not mention GateProbe by name. The paper is nevertheless directly relevant to any gating-based probing approach that operates on concept probes in the residual stream, including layer selection, ablation or intervention placement, and direction specificity. Its starting point is that a concept probe is a direction vector in residual-stream activation space that discriminates between sentences expressing a concept versus matched sentences not expressing that concept. The probe at layer $l$ is the L2-normalized difference between class centroids,
$$
u^{(l)} =
\frac{\bar{h}_+^{(l)} - \bar{h}_-^{(l)}}{\|\bar{h}_+^{(l)} - \bar{h}_-^{(l)}\|},
$$
using last-token activations from the post-MLP residual output. The Fisher-normalized separation score is
$$
S(l) =
\frac{\|\bar{h}_+^{(l)} - \bar{h}_-^{(l)}\|_2}{
\sqrt{\tfrac{1}{2}\left(\operatorname{tr}(\Sigma_+^{(l)})+\operatorname{tr}(\Sigma_-^{(l)})\right)}
}.
$$
The paper defines a Concept Allocation Zone as the contiguous layer interval containing the primary separation event, from above-noise onset through the peak and back toward a floor.

The key geometric claim is that concept directions rotate substantially during assembly and do not settle into a stable direction until a characteristic handoff layer after the CAZ. Rotation is quantified via adjacent-layer directional similarity
$$
DS(l) = |u^{(l)} \cdot u^{(l-1)}|,
$$
and angular velocity
$$
\omega(l) = 1 - |u^{(l)} \cdot u^{(l-1)}|.
$$
Entry-to-exit cosine within the CAZ is
$$
EEC = u^{(L_\mathrm{entry})} \cdot u^{(L_\mathrm{exit})}.
$$
Across $23$ architectures and $17$ concepts, mean EEC is $0.233$ and median EEC is $0.216$; $93.9\%$ of pairs have EEC $< 0.5$, and $23.8\%$ have EEC $< 0.1$. The paper interprets this as showing that directions at CAZ entry are poor predictors of settled directions at CAZ exit. The handoff layer is defined as the first post-CAZ layer after rotation ceases under the threshold $\epsilon = 0.05$:
$$
L_H = \min(L_{\mathrm{CAZ\_end}}+1,\; N-1).
$$
The handoff cosine $u^{(L_H)} \cdot u^{(N-1)}$ has mean $0.942$ and median $1.000$, which the paper presents as evidence that the settled probe direction is typically stable through to the final layer [2605.25848].

For intervention, the paper uses directional ablation at the extraction layer,
$$
h' = h - (h \cdot u)\,u,
$$
and measures retained percentage of separation. GEM compares the probe extracted at the handoff layer against the probe extracted at the peak layer. Across $391$ concept-model pairs, GEM-extracted probes are at least as precise as peak-layer probes in $268/391$ trials $(68.5\%)$ and strictly outperform in $259/391$ $(66.2\%)$. Trial-level Wilcoxon on $382$ non-ties gives $p \approx 3.21 \times 10^{-17}$, and model-level Wilcoxon signed-rank gives $W=214$, $N=23$, $p=0.010$ one-sided; excluding gpt2 yields $N=22$, $p=0.0036$. The architecture split is pronounced: MHA models favor the handoff in $173/221$ trials $(78.3\%)$, whereas GQA models favor the handoff in $56/119$ trials $(47.1\%)$.

The paper also gives practical guidance that maps directly onto GateProbe-style intervention design. The default ablation width is $w=3$ consecutive layers starting at $L_H$, but if $L_H/N > 0.85$ the near-final rule sets $w=1$ to avoid contamination by unembedding-preparation layers. This rule is triggered in $79/391$ cases $(20.2\%)$ and improves probe quality in $60/79$ triggered cases $(75.9\%)$ with mean gain $+7.44$ percentage points; the overall mean across all $391$ pairs is $+0.93$ points. A direction-specificity control compares concept-direction ablation against $10$ random unit vectors per pair, giving mean concept-direction reduction $45.6\%$ versus mean random-direction reduction $0.24\%$, median specificity ratio $377\times$, median z-score $252.7$, and $110/111$ pairs $(99.1\%)$ in which the concept direction beats all $10$ random seeds. The practical implication is narrow but important: gate-based probing in residual streams should target a settled direction at the handoff layer rather than a rotating peak-layer direction or an arbitrarily late layer. The stated caveats are that $\epsilon = 0.05$ is empirically set without a formal sensitivity analysis, late handoffs are common at $N=250$, shallow models can fail near the unembedding, some concepts may never settle, and sliding-window attention variants can violate depth-matched expectations [2605.25848].

Source: https://www.emergentmind.com/topics/gateprobe