Skipless Transformers: Residual-Free Design

• Skipless Transformers are architectures that remove additive residual (and sometimes normalization) connections, relying on pure compositions of transformations.
• Exact algebraic reformulations, such as Q+P elimination, reduce model parameters by up to 15% while maintaining the exact end-to-end function.
• Stabilization strategies like tailored initialization and orthogonal self-attention mitigate optimization challenges due to the absence of identity-based regularization.

Skipless Transformers are Transformer architectures in which the usual additive residual or skip connections are removed, so depth is realized as a pure composition of transformations rather than as perturbations around an identity path. In the strictest formulations, skiplessness is paired with the removal of normalization layers as well; in other formulations, the standard Transformer internals are preserved and only the residual additions are deleted. The topic has developed along three closely related axes: exact algebraic reformulations of skipless blocks that eliminate redundant linear maps, dynamical analyses that explain why residual-free optimization is difficult, and architectural alternatives that attempt to recover stable signal propagation without reinstating generic identity shortcuts (Graef, 2024, Zhang et al., 5 Feb 2026, Ji et al., 30 Sep 2025).

1. Definitions and architectural scope

In the literature summarized here, “skipless” has a precise architectural meaning. A standard Transformer block contains two additive shortcuts, one around self-attention and one around the feed-forward network. A skipless or residual-free Transformer removes these additive identity pathways from both sublayers, so the network no longer benefits from the usual activation and gradient transport through the residual stream (Ji et al., 30 Sep 2025).

A stricter subcase is the skipless, no-normalization setting studied in algebraic simplification work. There, Transformer blocks are defined as having neither skip or residual connections nor normalization. That assumption is central because residual additions create multiple signal paths, and normalization is nonlinear with respect to affine basis changes; both break the exact compositional linear algebra used to remove or absorb weights (Graef, 2024).

This distinction matters because not every architecture that weakens or structures residual transport is skipless in the ordinary sense. “Three-Phase Transformer” retains the standard pre-norm additive residual form around attention and FFN, and is therefore not a skipless Transformer. Its relevance is indirect: it constrains the geometry of the residual stream and inserts a non-residual orthogonal transformation between attention and FFN, but it does not abolish the skip backbone (Ayyash, 15 Apr 2026).

A common misconception is that any result labeled “skipless” transfers immediately to ordinary residual pre-norm LLMs. The exact equivalence results in skipless simplification papers do not support that reading: they are exact only when skip connections and normalization are absent, and when the eliminated matrix is square and invertible (Graef, 2024).

2. Exact skipless reformulations and weight elimination

A distinctive line of work studies skipless Transformers as algebraic objects whose adjacent linear maps can be merged or eliminated exactly. The two basic operations are the merging of adjacent linear layers and the insertion of an invertible matrix with its inverse, followed by absorption into neighboring weights. Because there are no residual additions and no normalization layers, these manipulations preserve the exact end-to-end function (Graef, 2024).

The most basic identity merges the post-attention output projection $P$ into the next FFN input matrix $M$: $M^* = PM.$ This removes $d^2$ parameters per block. A more consequential transformation eliminates the query projection $Q$ by pushing it into the preceding FFN output matrix $O$ and compensating in the key and value projections: $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$ This removes $Q$ and $P$, a total of $2d^2$ weights per block, and is exact when $M$0 is invertible (Graef, 2024).

The importance of this $M$1 elimination is architectural generality. Earlier skipless simplifications showed that $M$2 and $M$3 could be removed in standard multi-head attention, but that construction depends on the equality $M$4, so it is restricted to MHA. The $M$5 reformulation also works for multi-query attention and grouped-query attention, where the key-value channel dimension is reduced and $M$6 and $M$7 are rectangular rather than square. This makes the construction relevant to many current LLM families, including Llama 2, Mistral, Mixtral, PaLM, and Gemma (Graef, 2024).

For MHA, two additional exact variants exist: $M$8 and

$M$9

These are MHA-only because they require $M^* = PM.$0 or $M^* = PM.$1 to be square and invertible. In MQA and GQA, only the $M^* = PM.$2 route is supported by the exact algebra presented in the paper (Graef, 2024).

3. Why skipless Transformers are hard to train

A second research strand reframes skipless Transformers as a conditioning problem. The central claim is that residual connections stabilize optimization primarily by improving the conditioning of the network Jacobian, and that removing them exposes severe pathologies in self-attention (Ji et al., 30 Sep 2025).

In a residual Transformer, the derivative of the network output with respect to layer parameters contains factors of the form $M^* = PM.$3 and $M^* = PM.$4, where $M^* = PM.$5 and $M^* = PM.$6 are the input Jacobians of the self-attention and MLP sublayers. Without skips, those products become $M^* = PM.$7 and $M^* = PM.$8 alone. The interpretation is spectral: adding the identity shifts singular values away from zero and reduces condition numbers, whereas removing the identity exposes the raw, often poorly conditioned self-attention Jacobian (Ji et al., 30 Sep 2025).

The conditioning analysis is tied to the attention matrix itself. When attention logits are diffuse at initialization, the row-wise softmax produces a matrix close to $M^* = PM.$9, which is rank-1 and very poorly conditioned. By contrast, if the logits are diagonally dominant, the softmax output is close to identity and correspondingly better conditioned. This yields a precise mechanism for why naive skipless Transformers are unstable: default initialization often produces diffuse attention rather than diagonal dominance (Ji et al., 30 Sep 2025).

The same concern appears in work on Orthogonal Self-Attention, but there it is stated more strongly. That paper argues that standard Softmax Self-Attention is intrinsically poorly suited to skipless architectures because repeated softmax attention can drive token representations toward low rank, even rank-1, and because the input-output Jacobian becomes poorly conditioned when neither residuals nor normalization layers are present (Zhang et al., 5 Feb 2026).

A plausible synthesis is that residual-free training difficulties arise from two coupled effects: the loss of identity-based spectral regularization and the contraction or homogenization induced by conventional softmax token mixing. That synthesis is consistent with both the Jacobian analysis of standard skipless ViTs and the orthogonality-based redesign of attention for skipless non-causal Transformers (Ji et al., 30 Sep 2025, Zhang et al., 5 Feb 2026).

4. Stabilization strategies without generic skips

One response to skipless instability is to keep the standard Transformer architecture and intervene only at initialization. “Cutting the Skip” proposes exactly that. The self-attention projections are initialized so that $d^2$0 is scaled orthonormal and

$d^2$1

with $d^2$2. The $d^2$3 term biases the logits toward diagonal dominance, producing a better conditioned attention matrix at initialization, while the orthonormal $d^2$4 product makes the dominant Kronecker term in the self-attention Jacobian well conditioned. The method changes only initialization and does not alter the standard self-attention computation, block structure, normalization placement, training loss, or parameterization during training (Ji et al., 30 Sep 2025).

A second response is to replace softmax attention itself. Orthogonal Self-Attention defines

$d^2$5

Because $d^2$6, $d^2$7 is orthogonal. In the simplified skipless OSA-only setting with orthogonal $d^2$8, the layerwise kernel matrix satisfies

$d^2$9

so the rank and eigenvalues of $Q$0 are preserved. The paper also derives an initialization-time Jacobian conditioning bound,

$Q$1

under the proposed initialization and an orthogonality error assumption on $Q$2 (Zhang et al., 5 Feb 2026).

OSA also addresses implementability. Although a naive matrix exponential over $Q$3 token interactions would be cubic in sequence length, the skew-symmetric score matrix is low rank, with rank $Q$4. Using an orthonormal basis $Q$5 for the span of $Q$6 and $Q$7, the exponential can be written as

$Q$8

reducing the computation to an $Q$9 exponential. Under fixed hidden and head dimensions, the claimed complexity is $O$0 with memory $O$1, compared with softmax self-attention’s $O$2 complexity and $O$3 memory (Zhang et al., 5 Feb 2026).

5. Empirical evidence and practical consequences

The empirical record is heterogeneous because the evaluated settings differ substantially. The algebraic skipless simplification paper is not a training paper; its practical headline is parameter elimination. For a skipless version of Mistral-7B using GQA, removing $O$4 and $O$5 would reduce total weights from $O$6B to $O$7B, a $O$8 saving, with a projected $O$9 speedup for a batch-1 system limited by memory bandwidth during autoregressive next-token generation. For Pythia-6.9B, the table reports $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$0 weight savings after removing $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$1 and $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$2, with a possible $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$3 speedup (Graef, 2024).

The residual-free initialization paper evaluates standard-architecture skipless ViTs on ImageNet-1k and self-supervised dense prediction tasks. In supervised ImageNet-1k training with ViT-Base, the reported validation accuracies are $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$4 for residual ViT with AdamW, $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$5 for naive skipless ViT with AdamW, and $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$6 for skipless ViT with the proposed initialization and AdamW; with SOAP, the corresponding figures are $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$7, $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$8, and $$O^* = OQ,\qquad Q^* = 1,\qquad K^* = Q^{-1}K,\qquad V^* = Q^{-1}V,\qquad M^* = PM.$$9. In DINO ViT-Small self-supervision, skipless models outperform residual baselines on several dense tasks, including substantial gains in object discovery: on VOC2012 with SOAP, $Q$0 for the residual model versus $Q$1 for skipless at 300 epochs and $Q$2 for the skipless 200-epoch checkpoint; on COCO20k, $Q$3 versus $Q$4 and $Q$5 respectively (Ji et al., 30 Sep 2025).

Orthogonal Self-Attention provides smaller-scale but direct evidence for residual-free and normalization-free training. On MNIST, a standard ViT reaches $Q$6 test accuracy; a ViT without skip connections reaches $Q$7; a ViT without skip connections and without LayerNorm drops to $Q$8; and the OSA-Transformer reaches $Q$9 with QR basis computation and $P$0 with Newton–Schulz. The paper presents this as an initial validation rather than a large-scale benchmark (Zhang et al., 5 Feb 2026).

These results support different claims. The Mistral and Pythia examples show that skiplessness can expose exact algebraic redundancy in attention and FFN stacks. The ViT and OSA results show that residual-free optimization is not impossible, but its success depends strongly on stabilization strategy, problem setting, and scale (Graef, 2024, Zhang et al., 5 Feb 2026, Ji et al., 30 Sep 2025).

6. Limitations, controversies, and adjacent directions

Several limitations recur across the literature. First, exact skipless simplifications are not theorems about ordinary residual Transformers. Their exactness requires no skip connections, no normalization, suitable layer adjacency, and invertibility of the eliminated matrix. The papers explicitly leave it open whether related ideas remain beneficial once normalization and skip connections are reintroduced (Graef, 2024).

Second, current stabilization results are domain- and architecture-specific. “Cutting the Skip” is demonstrated on ViTs, not on language Transformers or large autoregressive pretraining. Orthogonal Self-Attention is explicitly restricted to non-causal settings such as ViTs and DiTs; it does not provide a causal or masked analogue for autoregressive language modeling (Ji et al., 30 Sep 2025, Zhang et al., 5 Feb 2026).

Third, claims about representational superiority remain context-dependent. The residual-free ViT paper argues that skipless models learn richer hierarchical and semantically coherent features, especially for dense vision tasks, because shallow features are not continually reintroduced into deeper layers. That interpretation is supported by dense prediction and object discovery results, but it is not established as a universal property of skipless Transformers across modalities or scales (Ji et al., 30 Sep 2025).

A related but distinct direction is structured residual transport rather than skip removal. Three-Phase Transformer retains residuals, RMSNorm, RoPE, and GQA, but partitions the residual stream into cyclic channel groups, inserts a non-residual orthogonal rotation between attention and FFN, and injects a fixed Gabriel’s horn profile $P$1 into a one-dimensional DC subspace. At $P$2M parameters on WikiText-103, it reports $P$3 perplexity and $P$4 step-count convergence speedup over a matched RoPE-only baseline at a $P$5-parameter overhead, but its authors explicitly characterize it as a residual-stream structural prior rather than a skipless architecture (Ayyash, 15 Apr 2026).

The broader implication is not that the field has converged on a single replacement for residual connections. Rather, the current literature identifies several non-equivalent routes: exact algebraic elimination in strict skipless blocks, Jacobian-aware initialization for residual-free standard attention, orthogonal token-mixing operators designed for skipless stability, and structured-residual designs that constrain rather than remove the skip pathway. This suggests that “skipless Transformers” now names both a concrete architectural class and a broader research program on how Transformer depth can remain expressive and trainable when generic identity transport is reduced, removed, or tightly structured (Graef, 2024, Zhang et al., 5 Feb 2026, Ji et al., 30 Sep 2025, Ayyash, 15 Apr 2026).