Doubly Stochastic Residual Connections

• Doubly stochastic residual connections are defined as neural network architectures that use learnable, mean-preserving mixing matrices to control multi-stream feature flow and maintain stability.
• Various parameterization methods—including Sinkhorn-Knopp, permutation-based, and Kronecker product approaches—offer trade-offs between exact constraint enforcement and computational efficiency.
• Alternative spectral-sphere constraints overcome expressivity limitations by allowing negative interactions, thus enhancing feature mixing while preserving overall network stability.

Doubly stochastic residual connections are architectural mechanisms that generalize traditional residual connections in neural networks by incorporating learnable mixing matrices constrained to be doubly stochastic. These connections fundamentally enable controlled information flow across multiple parallel streams or tokens, ensuring training stability, mean preservation, and useful regularization effects. The most notable instantiations arise in the context of Hyper-Connections (HC), advanced forms of skip connections designed for vectorized (multi-stream) layers, as well as in attention mechanisms within deep sequence models.

1. Doubly Stochastic Matrices and the Birkhoff Polytope

A matrix $H \in \mathbb{R}^{n \times n}$ is doubly stochastic if all entries are non-negative and each row and column sums to $1$. The set of all such matrices forms the Birkhoff polytope $B_n$: $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$ Doubly stochastic matrices preserve the mean of the input vector under multiplication and satisfy a spectral norm upper bound of $1$. The Birkhoff–von Neumann theorem establishes that every $H \in B_n$ can be decomposed as a convex combination of $n!$ permutation matrices, introducing a useful parameterization for enforcing this constraint in neural network settings (Yang et al., 9 Jan 2026, Zhou et al., 29 Jan 2026).

2. Doubly Stochastic Residual Hyper-Connections: Formulation

Hyper-Connections (HC) extend standard residual connections—from $x_{l+1} = x_l + F(x_l; W_l)$—to $n$ parallel streams of features. The layer update generalizes as: $$X_{l+1} = H_l^{\mathrm{res}} X_l + (H_l^{\mathrm{post}})^T F(H_l^{\mathrm{pre}} X_l; W_l)$$ where $1$0 is the matrix of $1$1 streams. $1$2 mixes streams residually, $1$3 gathers, and $1$4 redistributes features across streams (Liu et al., 21 Mar 2026, Yang et al., 9 Jan 2026).

To guarantee the identity-mapping property and prevent exploding or vanishing signals in deep networks, $1$5 is constrained to be doubly stochastic. This stabilizes forward and backward dynamics, maintains the spectral norm at $1$6, and, by the closure of $1$7 under matrix multiplication, avoids cumulative drift away from stability over many layers (Yang et al., 9 Jan 2026, Zhou et al., 29 Jan 2026).

The following table summarizes key options for parameterizing doubly stochastic residual matrices:

Approach	Exactness	Parameter Complexity	Notable Constraints
Sinkhorn-Knopp (SK)	Approximate	$1$8	Iterations $1$9, no exact DS
Permutation-based	Exact	$B_n$0	Factorial blowup
KromHC (Kronecker)	Exact	$B_n$1	Requires factorization of $B_n$2

3. Practical Parameterizations and Variants

Sinkhorn–Knopp (mHC)

The SK algorithm enforces approximate double stochasticity. Given an unconstrained matrix $B_n$3, iterative row and column normalization is performed:

• $B_n$4
• Alternating row/column normalization $B_n$5 times yields $B_n$6

However, finite $B_n$7 results in only approximate constraint satisfaction. Accumulatively, drift in marginal sums and spectral norm can occur at depth, causing loss of training stability (Yang et al., 9 Jan 2026, Liu et al., 21 Mar 2026).

Permutation-based (mHC-lite)

The Birkhoff–von Neumann theorem enables exact construction: $B_n$8 where the weights $B_n$9 are learned via softmax parameterizations. This approach guarantees $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$0 exactly, eliminating error accumulation, but is only tractable at small $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$1 due to the $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$2 basis size (Yang et al., 9 Jan 2026).

Kronecker Product Parameterization (KromHC)

KromHC reduces parameter growth by decomposing $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$3 as a Kronecker product of $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$4 small doubly stochastic matrices $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$5: $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$6 Each $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$7 is parameterized as a convex combination of $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$8 permutation matrices, with $$B_n = \{H \in \mathbb{R}^{n \times n} : H \mathbf{1}_n = \mathbf{1}_n,~ \mathbf{1}_n^T H = \mathbf{1}_n^T,~ H_{ij} \ge 0~\forall i,j\}$$9 small (ideally $1$0 or $1$1). This construction is closed under the DS constraint and scales as $1$2, making it feasible for large $1$3 (Zhou et al., 29 Jan 2026).

4. Limitations and Expressivity Bottlenecks

Three core limitations of doubly stochastic residual connections have been identified (Liu et al., 21 Mar 2026):

• Identity degeneration: Empirically, the learned $1$4 for mHC/mHC-lite tend to concentrate weight on the diagonal, collapsing to near-identity and thereby suppressing cross-stream mixing.
• Expressivity bottleneck: Non-negativity enforces convex combinations of streams, disallowing subtractive (negative) interactions, which drives up the similarity between streams and limits feature diversification.
• Parameterization inefficiency: Sinkhorn-based methods require expensive, custom CUDA implementation and may fail to reach true double stochasticity. Permutation-based methods incur factorial parameter and computational cost, scaling poorly as $1$5 increases.

These factors collectively motivate the search for alternative geometric constraints for the residual mixing matrices to balance stability and expressivity.

5. Spectral-Sphere-Constrained Hyper-Connections as an Alternative

Spectral-Sphere-Constrained Hyper-Connections (sHC) replace the Birkhoff polytope constraint with a spectral sphere: $1$6 where $1$7, i.e., the mean-preserving affine subspace, and $1$8 is the spectral norm.

Crucially, sHC allows $1$9 to be negative, removing the expressivity bottleneck and allowing subtractive feature disentanglement, while still guaranteeing stability via spectral-norm and mean-preservation. The feasible set $H \in B_n$0 is strictly contained in $H \in B_n$1. Parameterization leverages a spectral-decoupling trick with SVDs in the zero-marginal subspace, together with dynamically generated orthogonal factors (Liu et al., 21 Mar 2026).

Empirical results consistently show that sHC outperforms prior doubly stochastic approaches on cross-entropy, perplexity, gradient stability, and inter-stream similarity metrics, while requiring no Sinkhorn iteration or permutation enumeration (Liu et al., 21 Mar 2026).

6. Applications and Empirical Observations in Attention and HC Architectures

Doubly stochastic constraints have also been adopted in Transformer attention, where enforcing doubly stochasticity via Sinkhorn normalization improves entropy regularization and slows the otherwise doubly exponential rank collapse of feature representations across layers (Lapenna et al., 9 Apr 2026). However, the addition of residual connections (skips) to such architectures further moderates this collapse, transforming it to an exponential rate and keeping nontrivial signal diversity even in very deep models:

• Row-stochastic Softmax attention: rapid rank and entropy collapse without skips.
• Sinkhorn-normalized (doubly stochastic) attention: better preservation of attention entropy and rank, further improved when integrated with skip connections (Lapenna et al., 9 Apr 2026).

7. Scalability, Practical Recommendations, and Open Challenges

Doubly stochastic residual parameterizations offer precise regularization and stability advantages, but scalability and implementation cost remain nontrivial for large $H \in B_n$2:

• Sinkhorn-based methods: best used for moderate $H \in B_n$3, require careful engineering and may accumulate constraint violations.
• Permutation-based methods: suitable only for small $H \in B_n$4.
• Kronecker (KromHC): scalable to larger $H \in B_n$5, especially when $H \in B_n$6 is composite and can be factorized.
• Spectral-sphere constraints (sHC): provide expressivity without the bottlenecks of DS constraints, with parameter counts $H \in B_n$7 comparable to mHC but no projection/permutation overhead (Liu et al., 21 Mar 2026).

For practical adoption, choice of parameterization is driven by the required expressivity, anticipated model depth, and resource constraints. While exact doubly stochasticity is feasible and desirable for some domains, more flexible spectral-norm constraints such as in sHC may offer an improved tradeoff between stability and functionality.

---

Key References:

• "Beyond the Birkhoff Polytope: Spectral-Sphere-Constrained Hyper-Connections" (Liu et al., 21 Mar 2026)
• "mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations" (Yang et al., 9 Jan 2026)
• "KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices" (Zhou et al., 29 Jan 2026)
• "Sinkhorn doubly stochastic attention rank decay analysis" (Lapenna et al., 9 Apr 2026)