Residual Replacement Model

• Residual Replacement Model is a method that substitutes standard residual stream updates with learned or fitted surrogates while preserving key network behaviors.
• It offers a versatile framework applied to vision transformers for interpretability as well as to architectural compression for efficiency improvements.
• Empirical evaluations demonstrate high faithfulness, effective debiasing, and the ability to maintain target-logit performance despite significant model compression.

Residual replacement model denotes a class of methods that alter the canonical residual update $x_{\ell+1}=x_\ell+F(x_\ell)$ by substituting some part of the residual mechanism with a learned, fitted, or interpretable surrogate. In the most specific usage, the Residual Replacement Model (RRM) is a vision-transformer interpretability framework that replaces layerwise residual-stream activations with sparse autoencoder features plus an explicit reconstruction error term, yielding a graph of interpretable nodes and attributed edges (Kim et al., 22 Sep 2025). In related architectural and compression work, residual replacement instead refers to replacing the fixed residual addition, the costly residual branch, or selected residual submodules while preserving the residual-stream semantics that make deep networks trainable and compressible (Menghani et al., 2024, Cai et al., 10 Jun 2025, Cunegatti et al., 1 Jun 2026).

1. Residual replacement as a generalization of the skip connection

The standard residual connection used in modern deep nets, including ResNets and Transformers, is written for block $\ell$ as

$x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$

where $x_\ell\in\mathbb R^D$ is the residual stream, $F(\cdot;W_\ell)$ is a non-linear transform, and the addition is fixed and unweighted (Menghani et al., 2024). Residual replacement methods intervene on this structure rather than discarding it. The intervention can target the addition operator itself, the residual branch $F(x_\ell)$, a submodule contribution inside a Transformer layer, or the representation inhabiting the residual stream.

This distinction matters because recent work uses the same residual scaffold for different ends. LAuReL replaces the fixed “$+$” with a small learned module on the residual stream; FastFLUX replaces selected structurally complex residual branches in FLUX ResBlocks with affine maps while preserving the identity shortcut; SubFit removes selected Attention and FeedForward residual submodules non-contiguously and fits a lightweight residual bypass for each; and the ViT Residual Replacement Model replaces residual-stream activations with sparse, feature-level surrogates to obtain faithful and parsimonious circuits (Menghani et al., 2024, Cai et al., 10 Jun 2025, Cunegatti et al., 1 Jun 2026, Kim et al., 22 Sep 2025).

A common misconception is that residual replacement is equivalent to deleting layers or shortcuts. The cited works do not support that identification. FastFLUX explicitly preserves the shortcut $x$ and replaces only $R(x)$, giving $\hat y=x+L(x)$ rather than $\ell$0; LAuReL is framed as an in-situ replacement of the canonical residual connection; SubFit leaves all unselected components unchanged and fits bypasses only for chosen submodules; and the ViT RRM keeps an exact record of the discarded reconstruction error $\ell$1 rather than erasing it (Cai et al., 10 Jun 2025, Menghani et al., 2024, Cunegatti et al., 1 Jun 2026, Kim et al., 22 Sep 2025).

2. Formal definition of the ViT Residual Replacement Model

In the formulation introduced for vision transformers, let $\ell$2 denote the residual-stream activations at layer $\ell$3, with $\ell$4 tokens and hidden dimension $\ell$5, and let $\ell$6 denote the combined MHSA and FFN computation. A vanilla ViT updates the residual stream as

$\ell$7

RRM replaces each $\ell$8 with a sparse, feature-level surrogate extracted by a TopK sparse autoencoder (Kim et al., 22 Sep 2025).

For each layer $\ell$9, the model learns

$x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$0

and computes

$x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$1

$x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$2

The model then averages over tokens,

$x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$3

and treats the network as a directed graph $x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$4 whose nodes are the averaged features $x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$5 together with the error nodes $x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$6. Edges connect every feature or error node in layer $x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$7 to nodes in layer $x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$8 through the implicit action of $x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),$9 (Kim et al., 22 Sep 2025).

The resulting surrogate does not analytically re-implement Attention or FFN. Instead, it replaces the original residual streams with interpretable features and records the discarded error explicitly, while edge strengths are approximated by gradient-based attribution. The paper states that, by construction, $x_\ell\in\mathbb R^D$0 perfectly reconstructs $x_\ell\in\mathbb R^D$1, so no information is lost beyond reconstruction error. This is the basis for the claim that the method is faithful yet parsimonious (Kim et al., 22 Sep 2025).

3. Construction pipeline and circuit extraction

The construction begins by training one TopK SAE per layer on tokens collected from ImageNet images. The reconstruction objective is $x_\ell\in\mathbb R^D$2 plus an auxiliary dead-feature loss, and the sweep over $x_\ell\in\mathbb R^D$3 retains settings with fraction-of-variance-unexplained below $x_\ell\in\mathbb R^D$4. Across all layers and three ViT variants—supervised ViT, CLIP, and DINOv2—this yields roughly $x_\ell\in\mathbb R^D$5K features in total (Kim et al., 22 Sep 2025).

The feature space is then annotated by inspecting, for each layer’s approximately $x_\ell\in\mathbb R^D$6 strongest features, maximally activated patches, maximally activated images, and class logits. Features are categorized as Color, Line, Curve, Texture, Shape, Semantic, Object, Background, Positional, Miscellaneous, Polysemantic, or Uninterpretable. Equivalent categories are cross-checked and merged across annotators (Kim et al., 22 Sep 2025).

Edge importance is estimated with attribution patching, described as a first-order Taylor approximation to an interventional causal effect. For an edge $x_\ell\in\mathbb R^D$7, where $x_\ell\in\mathbb R^D$8 is an upstream node at layer $x_\ell\in\mathbb R^D$9 and $F(\cdot;W_\ell)$0 is a node at $F(\cdot;W_\ell)$1,

$F(\cdot;W_\ell)$2

where $F(\cdot;W_\ell)$3 is the normalized logit of the target class and $F(\cdot;W_\ell)$4 is the node’s median activation over the dataset. A Jacobian-vector trick reduces the naive $F(\cdot;W_\ell)$5 backpropagations to $F(\cdot;W_\ell)$6 (Kim et al., 22 Sep 2025).

Circuit extraction is recursive. At the final layer, the top-$F(\cdot;W_\ell)$7 nodes are chosen by importance into the output. For preceding layers, node selection uses an edge-based rule,

$F(\cdot;W_\ell)$8

so that features are kept for their specific effect on already selected downstream nodes rather than for global activity alone. Because ViTs exhibit noisy gradients, the framework applies LibraGrad’s gradient pruning and scaling to intermediate SAE features, restoring FullGrad-style completeness,

$F(\cdot;W_\ell)$9

This enforces an exact decomposition in which feature and bias contributions sum to the output (Kim et al., 22 Sep 2025).

4. Faithfulness, completeness, and empirical validation

The RRM paper evaluates circuits using three criteria. Faithfulness measures recovered logit mass,

$F(x_\ell)$0

Completeness is defined as

$F(x_\ell)$1

and causality is tested by ablating nodes in layer $F(x_\ell)$2 and measuring the downstream activation drop (Kim et al., 22 Sep 2025).

Table 1 in the paper reports AUC-over-$F(x_\ell)$3 for these metrics, averaged on $F(x_\ell)$4 ImageNet images.

Strategy	Faithfulness (%)	Causality (%)
Random circuit (features)	30.2	35.6
Naïve circuit (features)	64.9	46.9
+ Edge-based discovery	74.2	48.6
+ Gradient correction (full RRM)	94.1	54.5

The same table reports $F(x_\ell)$5Completeness values of $F(x_\ell)$6, $F(x_\ell)$7, $F(x_\ell)$8, and 99.6 for the four strategies, respectively (Kim et al., 22 Sep 2025). By comparison, circuits built from raw neurons achieve $F(x_\ell)$9 faithfulness at best. Appendix experiments further report that alternative selection rules such as top-$+$0 and threshold are near-identical, that Cross-Layer Attribution yields faithfulness of approximately $+$1 versus $+$2 for the full RRM, and that the Jacobian-vector trick provides a $+$3 speed-up in edge scoring (Kim et al., 22 Sep 2025).

These results establish the central empirical claim of the framework: the residual stream can be replaced by a sparse, interpretable feature graph without discarding most of the model’s target-logit behavior. The paper’s own phrasing is that RRM “scalably produces a faithful yet parsimonious circuit for human-scale interpretability” (Kim et al., 22 Sep 2025).

5. Mechanistic findings in vision transformers

The feature inventory derived through RRM reveals a progression from low-level to high-level content. Early layers contain localized patch-level detectors such as single-color, short-line, and small-curve features. Middle layers compose these into curves, shapes such as circles and L-corners, and low-frequency textures. Late layers contain object parts and full semantics, including animal foreheads, Granny Smith apples, and text on signs (Kim et al., 22 Sep 2025).

The paper emphasizes two feature families that are especially informative for ViT internals. First, curve detectors appear in layer 2 as orientation-specific responses to curves, and radial tuning curves verify angular coverage. RRM curve circuits, such as L3#801, are reported to match the classical Contrast$+$4Line$+$5Curve progression found in CNNs. Second, many early-layer features are position detectors with high mutual information $+$6; their averaged activation heatmaps tile the image grid, and deeper circuits show sharp detectors combining into diffuse ones while preserving spatial layout information (Kim et al., 22 Sep 2025).

The framework also supports cross-image circuit comparison. When analyzing the top-100 circuits for image pairs, same-class images share increasingly similar circuits in deeper layers, as measured by rising Adjusted Dice Score, whereas different-class circuits remain dissimilar (Kim et al., 22 Sep 2025). This suggests that the residual feature graph captures not only local detector semantics but also class-conditional convergence in the later residual stream.

A concrete intervention case study concerns the spurious correlation “Freight car $+$7 graffiti.” The top-3 feature circuit for a freight-car image with graffiti reveals, in layer 10, a freight-car detector $+$8 branching to layer-9 features for graffiti, shipping container, and wheel. Identifying $+$9 as the spurious feature, the intervention replaces its activation with its dataset median. On seven known spurious classes, aggregate freight-car numbers show the original ViT at accuracy $x$0 and mAUC $x$1; the intervened RRM-guided ViT at accuracy $x$2 and mAUC $x$3; and SpuFix at accuracy $x$4 and mAUC $x$5. The paper states that a single-feature ablation in one circuit recovers nearly the same debiasing power as a full dataset retraining approach (Kim et al., 22 Sep 2025).

6. Architectural and compression-oriented variants of residual replacement

Outside interpretability, residual replacement is also used to improve model quality, reduce compute, or compress pretrained networks. LAuReL replaces the fixed addition $x$6 with

$x$7

where $x$8 is a learned scalar or vector and $x$9 is a learned linear function on one or more past activations. Menghani et al. study three lightweight instantiations: a Residual-Weights version with $R(x)$0 parameters per layer, a Low-Rank version with $R(x)$1 parameters per layer, and a Previous-Activations version with approximately $R(x)$2 added parameters. On ResNet-50/ImageNet-1K, the Residual-Weights version reaches $R(x)$3 top-1 versus a $R(x)$4 baseline while adding $R(x)$5 parameters, and the Low-Rank $R(x)$6 version reaches $R(x)$7, matching the gain from adding one extra block with $R(x)$8 fewer parameters (Menghani et al., 2024).

FastFLUX uses Block-wise Replacement with Linear Layers (BRLL) in FLUX ResBlocks. A heavy residual branch $R(x)$9 is replaced by a fitted affine map $\hat y=x+L(x)$0, obtained from collected pairs $\hat y=x+L(x)$1 through a least-squares objective, while the shortcut is preserved so that $\hat y=x+L(x)$2. A localized fine-tuning procedure, Sandwich Training, then trains LoRA adapters only in the nearest unpruned neighbors together with the replacement block. On FLUX.1-dev, the reported latency on a 3090 GPU drops from $\hat y=x+L(x)$3 ms to $\hat y=x+L(x)$4 ms at $\hat y=x+L(x)$5 pruning, FLOPs drop from $\hat y=x+L(x)$6 T to $\hat y=x+L(x)$7 T, and HPS v2 changes from $\hat y=x+L(x)$8 to $\hat y=x+L(x)$9 (Cai et al., 10 Jun 2025).

SubFit extends replacement-based compression to submodule granularity. Rather than replacing contiguous full layers, it independently selects non-contiguous Attention and FFN submodules and assigns each a fitted residual bypass,

$\ell$00

with $\ell$01 low-rank. The fitting is closed form and uses only calibration data. Across ten LLMs and sparsities from $\ell$02 to $\ell$03, SubFit reports the best aggregate perplexity-accuracy trade-off among four replacement-based baselines. At $\ell$04 sparsity, it retains $\ell$05 of dense downstream accuracy with $\ell$06 perplexity degradation, versus $\ell$07 and $\ell$08 for the strongest baselines; pruning only, without replacement, collapses perplexity to thousands (Cunegatti et al., 1 Jun 2026).

Taken together, these systems show that residual replacement is not tied to a single granularity. It can operate at the level of the addition operator, residual branch, Transformer submodule, or residual-stream representation itself.

7. Conceptual distinctions, limitations, and scope

The interpretability RRM and the architectural/compression methods share a residual-centered perspective, but their objectives differ. The ViT RRM seeks a faithful surrogate graph for mechanistic analysis; LAuReL aims to improve convergence and final accuracy with modest overhead; FastFLUX targets inference efficiency and memory reduction in diffusion transformers; SubFit targets post-training LLM compression with measurable inference speedup and KV-cache savings (Kim et al., 22 Sep 2025, Menghani et al., 2024, Cai et al., 10 Jun 2025, Cunegatti et al., 1 Jun 2026).

The limitations reported in the literature are correspondingly different. LAuReL introduces extra hyperparameters such as rank $\ell$09 and normalization of $\ell$10, and too large a rank or unconstrained $\ell$11 can slightly degrade performance without proper tuning; the previous-activations version also requires more complex residual bookkeeping (Menghani et al., 2024). FastFLUX acknowledges a small but consistent approximation error $\ell$12, which motivates Sandwich Training, and it frames shortcut preservation as necessary to avoid train-time instabilities associated with total block removal (Cai et al., 10 Jun 2025). SubFit requires calibration data, uses a sequential two-stage selection process, and reports that masked removal scores disagree with replacement-aware scores in $\ell$13 cases, so removal impact is not a reliable proxy for surrogate approximability (Cunegatti et al., 1 Jun 2026). In the ViT RRM, faithfulness is explicitly bounded by reconstruction quality, and the method applies gradient correction because ViTs suffer from noisy gradients (Kim et al., 22 Sep 2025).

A plausible implication is that “residual replacement model” now names a broader methodological family rather than a single recipe. Within that family, the ViT RRM is the most explicit graph-theoretic formulation, because it turns the residual stream into a sparse feature graph with error accounting and attributed causal edges. The other approaches support the same broader thesis from a different direction: residual pathways can be replaced, fitted, or augmented in ways that preserve much of the behavior associated with the original network while exposing new trade-offs in faithfulness, efficiency, compression, or interpretability (Kim et al., 22 Sep 2025, Menghani et al., 2024, Cai et al., 10 Jun 2025, Cunegatti et al., 1 Jun 2026).