Cross-Variation Patching in Transformers

Updated 6 July 2026

Cross-variation patching is a method that treats a transformer's residual stream as a continuous field and injects localized differences to transfer prompt behavior.
It leverages sensitivity fields and empirical Green functions to predict the impact of interventions via first-order approximations.
The framework supports optimal patch-site selection and checkpoint recombination, enhancing the interpretability of model responses.

Searching arXiv for the primary paper and closely related work on patching diagnostics and cross-patching. Cross-variation patching, in the continuous-depth field-theoretic language of Olivieri & Pérez Rodríguez, treats a Transformer’s residual stream as a field over depth and token position and transfers the residual-field difference between two prompt variants by inserting that difference as a localized source at a chosen site. In this formulation, patching is not only an intervention but also a prediction problem: the downstream effect of the injected variation can be estimated from a first-order sensitivity field or from an empirical Green-function slice, and the same framework supports patch-site inference and cross-scale transfer (Olivieri et al., 24 May 2026). Related work uses cross-patching to separate upstream state from late readout across pretrained and instruction-tuned checkpoints, showing that a late-layer effect need not be self-contained (Zhou, 8 May 2026).

1. Continuous-depth residual-field formulation

The starting point is a continuous “depth” coordinate $t \in [0,T]$ with $t_\ell=\ell\,T/L$ and a token-space coordinate $x$ . The residual stream of a pre-norm Transformer is represented as a field

$r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$

In the absence of interventions, the field evolves by the residual ODE

$\partial_t\,r(t,x)=F_t[r](x),$

where $F_t$ encodes attention $+$ MLP updates at depth $t$ (Olivieri et al., 24 May 2026).

A discrete activation patch at layer $t_0$ and token $x_0$ is modeled as an impulsive source term $t_\ell=\ell\,T/L$ 0:

$t_\ell=\ell\,T/L$ 1

with

$t_\ell=\ell\,T/L$ 2

This makes patching a localized source insertion rather than an ad hoc replacement rule. Within this representation, a patch is specified by its support in depth-token space and by the vector it injects.

The significance of this reformulation is organizational as well as mathematical. The abstract explicitly states that the framework treats “patching as localized source insertion,” “patch effects as sensitivity-field predictions,” and “downstream propagation as empirical Green-function response,” thereby providing a common language for activation patching, causal tracing, path patching, and steering directions (Olivieri et al., 24 May 2026).

2. Sensitivity fields, adjoints, and empirical Green functions

For a scalar output $t_\ell=\ell\,T/L$ 3, such as a logit difference at the final token, a small source $t_\ell=\ell\,T/L$ 4 induces the first-order change

$t_\ell=\ell\,T/L$ 5

where the adjoint or “sensitivity field” is

$t_\ell=\ell\,T/L$ 6

For a small source inserted at a single site $t_\ell=\ell\,T/L$ 7,

$t_\ell=\ell\,T/L$ 8

The quantity $t_\ell=\ell\,T/L$ 9 is therefore the local first-order predictor of patch efficacy (Olivieri et al., 24 May 2026).

The sensitivity field is obtained from an adjoint construction. Introducing an adjoint field $x$ 0 and the action

$x$ 1

stationarity $x$ 2 gives the backward equation

$x$ 3

whose solution is $x$ 4. In this sense, the backward pass defines the same response object that first-order patch prediction uses.

Linearizing around the unpatched trajectory $x$ 5 by writing $x$ 6 yields

$x$ 7

with $x$ 8. The corresponding fundamental solution is the Green’s function

$x$ 9

satisfying

$r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 0

Once $r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 1 is known, any localized patch propagates by

$r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 2

The paper’s abstract reports empirical measurement of “structured anisotropic propagation across depth and token position” and construction of “response descriptions from high-sensitivity sites and sliced Green operators,” which situates these objects as experimentally measurable rather than purely formal (Olivieri et al., 24 May 2026).

3. Transfer between prompt variants

The canonical cross-variation setting considers two prompt variants $r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 3 and $r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 4 with residual fields $r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 5 and $r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 6. At a fixed site $r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 7, the patch-direction is

$r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 8

Its first-order scalar effect on run $r(t,x)\in\mathbb R^{d_{\rm model}},\qquad t=0,\dots,T,\;x=1,\dots,n.$ 9 is predicted by

$\partial_t\,r(t,x)=F_t[r](x),$ 0

Operationally, one computes $\partial_t\,r(t,x)=F_t[r](x),$ 1 by backpass on the $\partial_t\,r(t,x)=F_t[r](x),$ 2 run and measures the inner product with $\partial_t\,r(t,x)=F_t[r](x),$ 3. If it is large and positive, one expects the patched run to exhibit behavior closer to $\partial_t\,r(t,x)=F_t[r](x),$ 4 (Olivieri et al., 24 May 2026).

A more precise transfer uses a Green slice. For a downstream readout at $\partial_t\,r(t,x)=F_t[r](x),$ 5,

$\partial_t\,r(t,x)=F_t[r](x),$ 6

This supports site choice by maximizing $\partial_t\,r(t,x)=F_t[r](x),$ 7 over candidate locations. The field-theoretic description therefore distinguishes two predictive modes: a scalar first-order criterion using the sensitivity field and a downstream-state criterion using the empirical Green operator.

The worked toy example uses $\partial_t\,r(t,x)=F_t[r](x),$ 8 “The capital of Spain is” and $\partial_t\,r(t,x)=F_t[r](x),$ 9 “The capital of Italy is,” with the model predicting “ Madrid” and “ Rome,” respectively. Both runs are taken to depth $F_t$ 0 at final token $F_t$ 1, the patch-direction

$F_t$ 2

is recorded, and backprop on $F_t$ 3 yields

$F_t$ 4

The predicted effect is

$F_t$ 5

If $F_t$ 6, the patch is expected to favor “Rome” over “Madrid”; after applying

$F_t$ 7

the “Rome” logit rises, often overtaking “Madrid,” thereby effecting a cross-variation transfer of the capital-fact behavior (Olivieri et al., 24 May 2026).

4. Patch-site inference and the local linear regime

The same formalism yields an adjoint variational problem for optimal patch-site selection. To maximize a desired $F_t$ 8-behavior $F_t$ 9, one may solve

$+$ 0

In the linear regime, where $+$ 1, and with the energy penalty

$+$ 2

the Lagrange-multiplier condition gives

$+$ 3

Thus the optimal source aligns with the sensitivity field, while a sparsity or site-support constraint restricts the support of $+$ 4 to a small set of $+$ 5 and thereby selects a handful of patch sites (Olivieri et al., 24 May 2026).

This formulation places site selection on the same footing as forward-response prediction. Instead of searching over interventions solely by brute-force patching, one first computes response objects and then uses them to propose high-value sites. The abstract reports that the paper identifies “a bounded local linear regime” and predicts “patch effects from first-order sensitivities across residual sites,” which is the empirical condition under which the proportionality $+$ 6 is useful in practice (Olivieri et al., 24 May 2026).

A common misconception is to treat any successful patch as intrinsically local and self-explanatory. The field-theoretic treatment suggests a stricter interpretation: a local intervention is only one element of a distributed response, and its effect depends on both its first-order sensitivity and its downstream propagation. This suggests that causal claims from patching are strongest when accompanied by response objects—sensitivities, propagated fields, or Green-operator slices—rather than by endpoint behavior alone.

5. Checkpoint recombination and first-divergence cross-patching

A distinct but closely related diagnostic is “first-divergence cross-patching,” introduced to study cooperation between earlier computation and the late stack in pretrained base (PT) and instruction-tuned (IT) checkpoints. Let $+$ 7 be the depth at which the “late-stack boundary” is drawn, typically $+$ 8 of total depth. At the first token where PT and IT disagree under greedy sampling, the shared history is $+$ 9 and the divergent tokens are

$t$ 0

The protocol constructs four hybrid forward passes by mixing upstream state from one checkpoint with the late stack from either checkpoint:

$t$ 1

and studies the divergent-token margin

$t$ 2

This yields

$t$ 3

and the interaction

$t$ 4

Across the Core-5 dense families (4B–32B), the reported point estimates are $t$ 5, $t$ 6, and $t$ 7 logits; the interaction is positive in every family (Zhou, 8 May 2026).

The interpretation given in the paper is that the IT late stack has a real PT-upstream effect, but its larger effect in the IT checkpoint appears only when it reads its own post-trained upstream state. The reported “Portable (PT-upstream) share” is $t$ 8 on average, with family range $t$ 9– $t_0$ 0, leaving the remaining $t_0$ 1 of the IT late-stack effect dependent on IT upstream state. Sparse final-MLP features partially mediate this interaction: ablating the top-200 features in the IT late stack reduces the interaction by $t_0$ 2– $t_0$ 3, patching those features back into the weak hybrid rescues $t_0$ 4 logits, and an earlier-layer patch into the final stack recovers $t_0$ 5 logits, with $t_0$ 6 logit mediated by those same final-layer features (Zhou, 8 May 2026).

The paper also reports a structured boundary-state closure result for Llama: injecting a rank-256 approximation of the descendant-minus-base $t_0$ 7-level residual shift into the weak hybrid recovers $t_0$ 8 of the missing margin, while the full-delta recovers $t_0$ 9; random or sign-flipped directions have near-zero or negative effect. Forced-token scoring further shows that the local token choice can change later exact-answer success: on CONTENT-REASON exact-answer prompts, forcing $x_0$ 0 yields a $x_0$ 1 gain in suffix-only exact-match success, with smaller positive effects on safety ( $x_0$ 2) and format ( $x_0$ 3) validators (Zhou, 8 May 2026).

The main caution is explicit: when a behavior is localized to late layers, the late-stack effect should be tested under the other checkpoint’s upstream state before being treated as self-contained. In the paper’s phrasing, first-divergence cross-patching separates a direct late-stack component from an upstream-dependent component, and in the reported experiments most of the IT late-stack effect depends on the model’s own upstream computations (Zhou, 8 May 2026).

Outside mechanistic interpretability, cross-variation comparison appears in hardware patch-design analysis. “Theoretical Patchability Quantification for IP-Level Hardware Patching Designs” defines patchability as a combination of controllability and observability and uses this to compare IP variations or patching-logic choices at RTL. For a signal $x_0$ 4, the overall metric is

$x_0$ 5

with the paper taking all weights to be $x_0$ 6, so $x_0$ 7. A cross-variation comparison parses each RTL variant, marks directly patched nets, propagates $x_0$ 8 and $x_0$ 9 through the dataflow graph, computes $t_\ell=\ell\,T/L$ 00 or $t_\ell=\ell\,T/L$ 01, and compares patchability under equal investment budgets. In the reported case study on “reglk_wrapper,” configuration V3 invests $t_\ell=\ell\,T/L$ 02 bits and yields $t_\ell=\ell\,T/L$ 03, whereas V4 invests more ( $t_\ell=\ell\,T/L$ 04 bits) but gets only $t_\ell=\ell\,T/L$ 05, so V3 is strictly better (Liu et al., 2023).

A different software-repair use of cross-variation operates over candidate patches rather than activations. “A Single Patch Is Not Enough: Deterministic Fusion of Repair Candidates” defines a pool $t_\ell=\ell\,T/L$ 06 of candidate patches, decomposes each patch into edit atoms $t_\ell=\ell\,T/L$ 07, measures pairwise similarity by Jaccard over edit-atom sets,

$t_\ell=\ell\,T/L$ 08

builds repair neighborhoods from agreement graphs, selects a representative by medoid-style agreement, and applies evidence-constrained fusion (ECF) to retain repeated edit atoms and prune unsupported parts. On PatchFuseBench, PatchFusion solves $t_\ell=\ell\,T/L$ 09 bugs on SWE-bench Verified, $t_\ell=\ell\,T/L$ 10 on SWE-bench Multilingual, and reaches $t_\ell=\ell\,T/L$ 11 plausible patches on Defects4J; ECF alone adds $t_\ell=\ell\,T/L$ 12 solved with zero regressions (Yang et al., 2 Jul 2026). This is not activation patching, but it is a related instance of using structured variation across candidates to construct a stronger patch.

Multi-hunk repair studies formalize variation within a single patch. “Characterizing Multi-Hunk Patches: Divergence, Proximity, and LLM Repair Challenges” defines overall hunk divergence

$t_\ell=\ell\,T/L$ 13

together with a five-class spatial proximity taxonomy: Nucleus, Cluster, Orbit, Sprawl, and Fragment. On Hunk4J, which consists of $t_\ell=\ell\,T/L$ 14 real-world multi-hunk bugs mined from Defects4J, model success rates decline with increased divergence and spatial dispersion; with vanilla prompts, Plausible@1 rates are $t_\ell=\ell\,T/L$ 15– $t_\ell=\ell\,T/L$ 16, and no model succeeds in the most dispersed Fragment class (Nashid et al., 4 Jun 2025). A plausible implication is that cross-variation patching becomes more difficult as the variation to be transferred or coordinated becomes more heterogeneous and more spatially dispersed.

Taken together, these adjacent literatures reinforce a common methodological point. Whether the object being patched is a residual stream, a checkpoint boundary state, an RTL design, or a candidate repair pool, cross-variation methods are most informative when they explicitly model what varies, where it is inserted or compared, and how the induced effect propagates.