---
title: Residual Replacement Model
url: https://www.emergentmind.com/topics/residual-replacement-model
type: topic
---

# Residual Replacement Model

Residual replacement model denotes a class of methods that alter the canonical residual update \(x_{\ell+1}=x_\ell+F(x_\ell)\) by substituting some part of the residual mechanism with a learned, fitted, or interpretable surrogate. In the most specific usage, the Residual Replacement Model (RRM) is a vision-transformer interpretability framework that replaces layerwise residual-stream activations with sparse autoencoder features plus an explicit reconstruction error term, yielding a graph of interpretable nodes and attributed edges [2509.17401]. In related architectural and compression work, residual replacement instead refers to replacing the fixed residual addition, the costly residual branch, or selected residual submodules while preserving the residual-stream semantics that make deep networks trainable and compressible [2411.07501] [2506.10035] [2606.02559].

## 1. Residual replacement as a generalization of the skip connection

The standard residual connection used in modern deep nets, including ResNets and Transformers, is written for block \(\ell\) as
\[
x_{\ell+1} = x_\ell + F(x_\ell;W_\ell),
\]
where \(x_\ell\in\mathbb R^D\) is the residual stream, \(F(\cdot;W_\ell)\) is a non-linear transform, and the addition is fixed and unweighted [2411.07501]. Residual replacement methods intervene on this structure rather than discarding it. The intervention can target the addition operator itself, the residual branch \(F(x_\ell)\), a submodule contribution inside a Transformer layer, or the representation inhabiting the residual stream.

This distinction matters because recent work uses the same residual scaffold for different ends. LAuReL replaces the fixed “\(+\)” with a small learned module on the residual stream; FastFLUX replaces selected structurally complex residual branches in FLUX ResBlocks with affine maps while preserving the identity shortcut; SubFit removes selected Attention and FeedForward residual submodules non-contiguously and fits a lightweight residual bypass for each; and the ViT Residual Replacement Model replaces residual-stream activations with sparse, feature-level surrogates to obtain faithful and parsimonious circuits [2411.07501] [2506.10035] [2606.02559] [2509.17401].

A common misconception is that residual replacement is equivalent to deleting layers or shortcuts. The cited works do not support that identification. FastFLUX explicitly preserves the shortcut \(x\) and replaces only \(R(x)\), giving \(\hat y=x+L(x)\) rather than \(\hat y=L(x)\); LAuReL is framed as an in-situ replacement of the canonical residual connection; SubFit leaves all unselected components unchanged and fits bypasses only for chosen submodules; and the ViT RRM keeps an exact record of the discarded reconstruction error \(\epsilon_\ell\) rather than erasing it [2506.10035] [2411.07501] [2606.02559] [2509.17401].

## 2. Formal definition of the ViT Residual Replacement Model

In the formulation introduced for vision transformers, let \(x_\ell\in\mathbb R^{T\times d}\) denote the residual-stream activations at layer \(\ell\), with \(T\) tokens and hidden dimension \(d\), and let \(\mathrm{Block}_\ell(\cdot)\) denote the combined MHSA and FFN computation. A vanilla ViT updates the residual stream as
\[
x_{\ell+1}=x_\ell+\mathrm{Block}_\ell(x_\ell).
\]
RRM replaces each \(x_\ell\) with a sparse, feature-level surrogate extracted by a TopK sparse autoencoder [2509.17401].

For each layer \(\ell\), the model learns
\[
W^{(\ell)}_{\mathrm{enc}}\in\mathbb R^{f_\ell\times d},\quad
W^{(\ell)}_{\mathrm{dec}}\in\mathbb R^{d\times f_\ell},\quad
b^{(\ell)}_{\mathrm{pre}}\in\mathbb R^d,
\]
and computes
\[
z_\ell=\mathrm{TopK}\!\bigl(W^{(\ell)}_{\mathrm{enc}}(x_\ell-b^{(\ell)}_{\mathrm{pre}})\bigr)\in\mathbb R^{T\times f_\ell},
\]
\[
\hat x_\ell=W^{(\ell)}_{\mathrm{dec}}z_\ell+b^{(\ell)}_{\mathrm{pre}},
\qquad
\epsilon_\ell=x_\ell-\hat x_\ell.
\]
The model then averages over tokens,
\[
\bar z_\ell=\tfrac1T\sum_{t=1}^T z_{\ell,t},
\qquad
\bar\epsilon_\ell=\tfrac1T\sum_{t=1}^T \epsilon_{\ell,t},
\]
and treats the network as a directed graph \(\mathcal G\) whose nodes are the averaged features \(\{\bar z_\ell^{(i)}\}\) together with the error nodes \(\{\bar\epsilon_\ell\}\). Edges connect every feature or error node in layer \(\ell\) to nodes in layer \(\ell+1\) through the implicit action of \(\mathrm{Block}_\ell\) [2509.17401].

The resulting surrogate does not analytically re-implement Attention or FFN. Instead, it replaces the original residual streams with interpretable features and records the discarded error explicitly, while edge strengths are approximated by gradient-based attribution. The paper states that, by construction, \(\{z_\ell,\epsilon_\ell\}\) perfectly reconstructs \(x_\ell\), so no information is lost beyond reconstruction error. This is the basis for the claim that the method is faithful yet parsimonious [2509.17401].

## 3. Construction pipeline and circuit extraction

The construction begins by training one TopK SAE per layer on tokens collected from ImageNet images. The reconstruction objective is \(\mathcal L_{\mathrm{recon}}=\|x_\ell-\hat x_\ell\|_2^2\) plus an auxiliary dead-feature loss, and the sweep over \((f_\ell,k)\) retains settings with fraction-of-variance-unexplained below \(0.15\). Across all layers and three ViT variants—supervised ViT, CLIP, and DINOv2—this yields roughly \(6.6\)K features in total [2509.17401].

The feature space is then annotated by inspecting, for each layer’s approximately \(200\) strongest features, maximally activated patches, maximally activated images, and class logits. Features are categorized as Color, Line, Curve, Texture, Shape, Semantic, Object, Background, Positional, Miscellaneous, Polysemantic, or Uninterpretable. Equivalent categories are cross-checked and merged across annotators [2509.17401].

Edge importance is estimated with attribution patching, described as a first-order Taylor approximation to an interventional causal effect. For an edge \((u\to d)\), where \(u\) is an upstream node at layer \(\ell\) and \(d\) is a node at \(\ell+1\),
\[
\mathcal I(u\!\to\!d)
=
\nabla_d m
\times
\nabla_u d
\times
(u-u'),
\]
where \(m\) is the normalized logit of the target class and \(u'\) is the node’s median activation over the dataset. A Jacobian-vector trick reduces the naive \(\mathcal O(Tf)\) backpropagations to \(\mathcal O(f)\) [2509.17401].

Circuit extraction is recursive. At the final layer, the top-\(k\) nodes are chosen by importance into the output. For preceding layers, node selection uses an edge-based rule,
\[
\mathcal V_\ell
=
\operatorname{TopK}_{u\in \text{layer }\ell}
\sum_{d\in \mathcal V_{\ell+1}}
\mathcal I(u\!\to\!d),
\]
so that features are kept for their specific effect on already selected downstream nodes rather than for global activity alone. Because ViTs exhibit noisy gradients, the framework applies LibraGrad’s gradient pruning and scaling to intermediate SAE features, restoring FullGrad-style completeness,
\[
m(a;b)=\nabla_a m\cdot a+\nabla_b m\cdot b.
\]
This enforces an exact decomposition in which feature and bias contributions sum to the output [2509.17401].

## 4. Faithfulness, completeness, and empirical validation

The RRM paper evaluates circuits using three criteria. Faithfulness measures recovered logit mass,
\[
\mathrm{faith}(\mathcal C)=
\frac{m(\mathcal C)-m(\emptyset)}{m(\mathcal G)-m(\emptyset)}\in[0,1].
\]
Completeness is defined as
\[
\mathrm{compl}(\mathcal C)=
1-\frac{m(\mathcal G\setminus \mathcal C)-m(\emptyset)}{m(\mathcal G)-m(\emptyset)},
\]
and causality is tested by ablating nodes in layer \(\ell\) and measuring the downstream activation drop [2509.17401].

Table 1 in the paper reports AUC-over-\(k\) for these metrics, averaged on \(1{,}500\) ImageNet images.

| Strategy | Faithfulness (%) | Causality (%) |
|---|---:|---:|
| Random circuit (features) | 30.2 | 35.6 |
| Naïve circuit (features) | 64.9 | 46.9 |
| + Edge-based discovery | 74.2 | 48.6 |
| + Gradient correction (full RRM) | **94.1** | 54.5 |

The same table reports \(1-\)Completeness values of \(78.1\), \(94.2\), \(93.4\), and **99.6** for the four strategies, respectively [2509.17401]. By comparison, circuits built from raw neurons achieve \(61.4\%\) faithfulness at best. Appendix experiments further report that alternative selection rules such as top-\(p\) and threshold are near-identical, that Cross-Layer Attribution yields faithfulness of approximately \(56\%\) versus \(94\%\) for the full RRM, and that the Jacobian-vector trick provides a \(200\times\) speed-up in edge scoring [2509.17401].

These results establish the central empirical claim of the framework: the residual stream can be replaced by a sparse, interpretable feature graph without discarding most of the model’s target-logit behavior. The paper’s own phrasing is that RRM “scalably produces a faithful yet parsimonious circuit for human-scale interpretability” [2509.17401].

## 5. Mechanistic findings in vision transformers

The feature inventory derived through RRM reveals a progression from low-level to high-level content. Early layers contain localized patch-level detectors such as single-color, short-line, and small-curve features. Middle layers compose these into curves, shapes such as circles and L-corners, and low-frequency textures. Late layers contain object parts and full semantics, including animal foreheads, Granny Smith apples, and text on signs [2509.17401].

The paper emphasizes two feature families that are especially informative for ViT internals. First, curve detectors appear in layer 2 as orientation-specific responses to curves, and radial tuning curves verify angular coverage. RRM curve circuits, such as L3#801, are reported to match the classical Contrast\(\to\)Line\(\to\)Curve progression found in CNNs. Second, many early-layer features are position detectors with high mutual information \(I(\mathrm{act},pos)\); their averaged activation heatmaps tile the image grid, and deeper circuits show sharp detectors combining into diffuse ones while preserving spatial layout information [2509.17401].

The framework also supports cross-image circuit comparison. When analyzing the top-100 circuits for image pairs, same-class images share increasingly similar circuits in deeper layers, as measured by rising Adjusted Dice Score, whereas different-class circuits remain dissimilar [2509.17401]. This suggests that the residual feature graph captures not only local detector semantics but also class-conditional convergence in the later residual stream.

A concrete intervention case study concerns the spurious correlation “Freight car \(\leftrightarrow\) graffiti.” The top-3 feature circuit for a freight-car image with graffiti reveals, in layer 10, a freight-car detector \(\mathrm{L10\#1534}\) branching to layer-9 features for graffiti, shipping container, and wheel. Identifying \(\mathrm{L9\#2371}\) as the spurious feature, the intervention replaces its activation with its dataset median. On seven known spurious classes, aggregate freight-car numbers show the original ViT at accuracy \(0.849\) and mAUC \(0.854\); the intervened RRM-guided ViT at accuracy \(0.848\) and mAUC \(0.904\); and SpuFix at accuracy \(0.848\) and mAUC \(0.917\). The paper states that a single-feature ablation in one circuit recovers nearly the same debiasing power as a full dataset retraining approach [2509.17401].

## 6. Architectural and compression-oriented variants of residual replacement

Outside interpretability, residual replacement is also used to improve model quality, reduce compute, or compress pretrained networks. LAuReL replaces the fixed addition \(x_\ell+F(x_\ell)\) with
\[
x_{\ell+1}=\alpha[F(x_\ell)] + g(x_\ell,x_{\ell-1},\dots,x_0),
\]
where \(\alpha\) is a learned scalar or vector and \(g\) is a learned linear function on one or more past activations. Menghani et al. study three lightweight instantiations: a Residual-Weights version with \(2\) parameters per layer, a Low-Rank version with \(2rD\) parameters per layer, and a Previous-Activations version with approximately \(2rD+(\ell+1)\) added parameters. On ResNet-50/ImageNet-1K, the Residual-Weights version reaches \(75.10\%\) top-1 versus a \(74.95\%\) baseline while adding \(0.003\%\) parameters, and the Low-Rank \(r=16\) version reaches \(75.20\%\), matching the gain from adding one extra block with \(2.6\times\) fewer parameters [2411.07501].

FastFLUX uses Block-wise Replacement with Linear Layers (BRLL) in FLUX ResBlocks. A heavy residual branch \(R(x)\) is replaced by a fitted affine map \(L(x)=Wx+b\), obtained from collected pairs \(\{(x_i,f(x_i))\}_{i=1}^N\) through a least-squares objective, while the shortcut is preserved so that \(\hat y=x+L(x)\). A localized fine-tuning procedure, Sandwich Training, then trains LoRA adapters only in the nearest unpruned neighbors together with the replacement block. On FLUX.1-dev, the reported latency on a 3090 GPU drops from \(98.26\) ms to \(80.83\) ms at \(20\%\) pruning, FLOPs drop from \(38.19\) T to \(32.46\) T, and HPS v2 changes from \(28.32\) to \(27.76\) [2506.10035].

SubFit extends replacement-based compression to submodule granularity. Rather than replacing contiguous full layers, it independently selects non-contiguous Attention and FFN submodules and assigns each a fitted residual bypass,
\[
\widetilde F_\ell(x;\theta_\ell)=g_\ell\odot x+b_\ell+(x-\mu_\ell)U_\ell,
\]
with \(U_\ell=V_\ell^\top W_\ell\) low-rank. The fitting is closed form and uses only calibration data. Across ten LLMs and sparsities from \(12.5\%\) to \(37.5\%\), SubFit reports the best aggregate perplexity-accuracy trade-off among four replacement-based baselines. At \(25\%\) sparsity, it retains \(84.6\%\) of dense downstream accuracy with \(2.42\times\) perplexity degradation, versus \(81.6\%\) and \(4.34\times\) for the strongest baselines; pruning only, without replacement, collapses perplexity to thousands [2606.02559].

Taken together, these systems show that residual replacement is not tied to a single granularity. It can operate at the level of the addition operator, residual branch, Transformer submodule, or residual-stream representation itself.

## 7. Conceptual distinctions, limitations, and scope

The interpretability RRM and the architectural/compression methods share a residual-centered perspective, but their objectives differ. The ViT RRM seeks a faithful surrogate graph for mechanistic analysis; LAuReL aims to improve convergence and final accuracy with modest overhead; FastFLUX targets inference efficiency and memory reduction in diffusion transformers; SubFit targets post-training LLM compression with measurable inference speedup and KV-cache savings [2509.17401] [2411.07501] [2506.10035] [2606.02559].

The limitations reported in the literature are correspondingly different. LAuReL introduces extra hyperparameters such as rank \(r\) and normalization of \(\alpha,\beta\), and too large a rank or unconstrained \(\alpha,\beta\) can slightly degrade performance without proper tuning; the previous-activations version also requires more complex residual bookkeeping [2411.07501]. FastFLUX acknowledges a small but consistent approximation error \(\Delta(x)=R(x)-L(x)\), which motivates Sandwich Training, and it frames shortcut preservation as necessary to avoid train-time instabilities associated with total block removal [2506.10035]. SubFit requires calibration data, uses a sequential two-stage selection process, and reports that masked removal scores disagree with replacement-aware scores in \(5/8\) cases, so removal impact is not a reliable proxy for surrogate approximability [2606.02559]. In the ViT RRM, faithfulness is explicitly bounded by reconstruction quality, and the method applies gradient correction because ViTs suffer from noisy gradients [2509.17401].

A plausible implication is that “residual replacement model” now names a broader methodological family rather than a single recipe. Within that family, the ViT RRM is the most explicit graph-theoretic formulation, because it turns the residual stream into a sparse feature graph with error accounting and attributed causal edges. The other approaches support the same broader thesis from a different direction: residual pathways can be replaced, fitted, or augmented in ways that preserve much of the behavior associated with the original network while exposing new trade-offs in faithfulness, efficiency, compression, or interpretability [2509.17401] [2411.07501] [2506.10035] [2606.02559].

Source: https://www.emergentmind.com/topics/residual-replacement-model