Hyper-Connections in Neural Networks

• Hyper-Connections (HC) are neural architectures that replace single residual connections with multiple parallel streams and learned mixing operators to aggregate and redistribute signals.
• They enable dynamic routing across network depth and width, addressing issues like gradient vanishing and representation collapse while retaining the identity mapping property.
• HC implementations, including static and dynamic variants, offer minimal computational overhead while boosting performance across language, vision, graph, and medical imaging tasks.

Hyper-Connections (HC) are a family of neural-network connectivity schemes that replace the single residual shortcut by multiple parallel residual streams and learned mixing operators. In the formulation introduced in "Hyper-Connections" (Zhu et al., 2024), a network maintains $n$ parallel hidden states, learns how to aggregate them into the layer transform, and learns how to redistribute both the transformed signal and the bypassed signal across streams. Subsequent work recast the same idea through learned pre-, post-, and residual mixing matrices, and then imposed manifold, Kronecker, transportation-polytope, spectral-sphere, and orthogonal constraints to recover the identity mapping property, improve scalability, and reduce systems overhead (Xie et al., 31 Dec 2025). This suggests that HC is best understood not as a single block replacement, but as a broader multi-stream residual design framework.

1. Core mathematical formulation

A standard residual step is

$x_k = f(W_k x_{k-1}) + x_{k-1}.$

HC generalize this by replicating the input $n$ times: $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$ In the static formulation, one introduces

$$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$

and computes

$$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$

This is the original SHC construction in "Hyper-Connections" (Zhu et al., 2024).

A later, now standard, notation writes the same idea as

$$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$

with $\mathbf{X}_l\in\mathbb R^{n\times C}$, $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$, $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$, and $x_k = f(W_k x_{k-1}) + x_{k-1}.$0 (Xie et al., 31 Dec 2025). In this form, the residual stream width is $x_k = f(W_k x_{k-1}) + x_{k-1}.$1 but the per-token FLOPs of $x_k = f(W_k x_{k-1}) + x_{k-1}.$2 remain unchanged. When $x_k = f(W_k x_{k-1}) + x_{k-1}.$3, the HC update reduces exactly to the standard residual block (Alimaskina et al., 2 Jun 2026).

The formulation separates three operations that are conflated in an ordinary residual connection: reading from multiple streams, transforming a pooled representation, and writing the result back into multiple streams. That separation is the defining structural move of HC.

2. Residual connections, the seesaw effect, and sequential–parallel duality

The original motivation for HC was the “seesaw effect” between gradient vanishing and representation collapse. In the analysis accompanying the original method, Pre-Norm yields no gradient vanishing but deeper hidden states collapse, whereas Post-Norm avoids representation collapse but re-introduces gradient vanishing (Zhu et al., 2024). The same source states that HC allow the network to adjust the strength of connections between features at different depths and dynamically rearrange layers.

Within the HC formalism, ordinary residual schemes appear as special cases with $x_k = f(W_k x_{k-1}) + x_{k-1}.$4 and fixed connection matrices. The paper gives

$x_k = f(W_k x_{k-1}) + x_{k-1}.$5

which makes precise the claim that standard residual connections are non-trainable HC instances (Zhu et al., 2024).

The same work also gives a sequential–parallel duality: HC can learn to arrange blocks either sequentially, like residual stacks, or in parallel, like parallel transformer blocks, or in a soft mixture between the two (Zhu et al., 2024). This is significant because the residual path is no longer a fixed identity route; it becomes a learned routing topology over depth and width.

3. Static and dynamic variants, initialization, and overhead

Two main variants appear in the original formulation: Static Hyper-Connections (SHC), where $x_k = f(W_k x_{k-1}) + x_{k-1}.$6, $x_k = f(W_k x_{k-1}) + x_{k-1}.$7, and $x_k = f(W_k x_{k-1}) + x_{k-1}.$8 are directly learned, and Dynamic Hyper-Connections (DHC), where these coefficients are predicted from normalized hidden states through a small linear layer and a $x_k = f(W_k x_{k-1}) + x_{k-1}.$9 gate with small learnable scales (Zhu et al., 2024). The dynamic form keeps the mixing input-dependent while bounding the dynamic contribution.

Implementation overhead was explicitly quantified in the original paper. Static HC adds $n$0 parameters per module, and Dynamic HC adds approximately $n$1 parameters. For OLMo-1B with $n$2, SHC adds $n$3 parameters $n$4, whereas DHC adds $n$5 $n$6. The HC cost scales as $n$7, which is much smaller than attention+FFN cost $n$8, and extra activation memory is $n$9, reported as $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$0 of a standard transformer for $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$1 (Zhu et al., 2024).

The original empirical guidance was that $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$2 is a sweet spot (Zhu et al., 2024). A later summary of the original HC design similarly states that $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$3 or $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$4 already yields most of the benefit, while larger $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$5 increases memory footprint and activation traffic (Zhu et al., 18 Mar 2025). These statements reflect a recurring trade-off in the HC literature: richer residual topology is cheap in arithmetic but not free in memory movement.

4. Manifold-constrained and spectrum-aware HC

Unconstrained HC destabilize the identity mapping property intrinsic to residual networks. In mHC, this issue is formalized by observing that the composite residual mapping

$$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$6

need not preserve row sums or column sums, and the worst-case forward gain and backward gain can grow by orders of $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$7 in HC (Xie et al., 31 Dec 2025). To restore stability, mHC constrain the residual mixer to the Birkhoff polytope

$$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$8

so that row- and column-sums equal $$H^0 = [h^0; h^0; \dots; h^0]\in\mathbb R^{n\times d}.$$9, the spectral norm is $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$0, and the set is closed under multiplication (Xie et al., 31 Dec 2025).

Recent parameterizations differ mainly in how they realize or relax that constraint (Xie et al., 31 Dec 2025, Zhou et al., 29 Jan 2026, Dandachi et al., 2 Apr 2026, Lyubinin, 20 May 2026, Liu et al., 21 Mar 2026, Sengupta et al., 20 Feb 2026).

Variant	Residual-matrix construction	Stated consequence
mHC	Sinkhorn-Knopp projection to the Birkhoff polytope	row/column sums $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$1; spectral norm $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$2
mHC-lite	convex combination of permutation matrices	exact doubly stochasticity; factorial complexity
KromHC	Kronecker products of smaller doubly stochastic matrices	exact doubly stochasticity; $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$3
go-mHC	generalized orthostochastic parameterization	exact $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$4; hyperparameter $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$5
TBP-mHC / RTBP	transportation-polytope chart	exact doubly stochasticity; $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$6 degrees of freedom
sHC	affine-constrained spectral sphere	negative entries; polynomial scaling
JPmHC	operator-norm-bounded manifolds, including Stiefel	Jacobian-spectrum control

The distinctions among these variants are substantive. KromHC guarantees exact double stochasticity while reducing parameter complexity to $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$7 (Zhou et al., 29 Jan 2026). go-mHC introduces an exact parameterization grounded in generalized orthostochastic matrices, scales as $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$8, and exposes a hyperparameter $$A_m\in\mathbb R^{n\times 1},\qquad A_r\in\mathbb R^{n\times n},\qquad B\in\mathbb R^{1\times n},$$9 that interpolates between a computationally efficient boundary and the fully expressive Birkhoff polytope (Dandachi et al., 2 Apr 2026). TBP-mHC and RTBP construct exactly doubly stochastic matrices with $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$0 degrees of freedom while preserving full expressivity of the Birkhoff polytope (Lyubinin, 20 May 2026). sHC moves from the Birkhoff polytope to a spectral norm sphere, allows negative entries, and is presented as a response to identity degeneration, an expressivity bottleneck, and parameterization inefficiencies (Liu et al., 21 Mar 2026). JPmHC replaces identity skips with a trainable linear mixer acting on $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$1 parallel streams, constrains the mixer on operator-norm-bounded manifolds such as bistochastic, Stiefel, and Grassmann, and adds a free-probability analysis, memory-efficient implicit differentiation, and a Stiefel-constrained mixer via Cayley transforms (Sengupta et al., 20 Feb 2026).

Systems work has become part of the HC literature as well. For the practically important $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$2 Birkhoff projection setting, a dual Newton solver with implicit differentiation and a warp-level CUDA kernel was reported to achieve over $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$3 acceleration at large batch sizes while maintaining orders of magnitude smaller marginal errors than representative open-source baselines (Wang et al., 26 May 2026).

5. Mechanistic analyses, stream collapse, and causal diagnostics

The introduction of multiple residual streams does not by itself explain how information is distributed across them. "Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections" introduced the first open-source mHC LLM and a stream ablation-and-rescue framework that zeroes selected streams, restores one of them from cached activations, and measures token-wise KL shifts (Peng et al., 16 Mar 2026). In that framework,

$$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$4

The paper reports that streams $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$5 and $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$6 show high mutual recoverability $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$7, while streams $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$8 and $$h_0 = A_m^\top H^{k-1},\qquad H' = A_r^\top H^{k-1},\qquad h_0' = \mathcal T_k(h_0),\qquad \hat H^k = B^\top (h_0')^\top + H'.$$9 exhibit asymmetric rescue: rescuing $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$0 after ablating $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$1 recovers $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$2, but the reverse recovers only $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$3, a $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$4 pp gap (Peng et al., 16 Mar 2026). It also states that high CKA did not guarantee functional interchangeability.

A distinct line of work focused on the symmetry of HC under stream permutations and the possibility of dominant-stream behavior. "Analyzing Stream Collapse in Hyper-Connections: From Diagnosis to Mitigation" studied average residual mixers, read/write shares, stream-wise norms, residual curvature, and sparse crosscoders, and found that after an early seeding stage, residual mixing often remains close to identity (Alimaskina et al., 2 Jun 2026). The same study reports that one stream rapidly captures $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$5 of read and write mass, and that after layer $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$6, $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$7 of recovered sparse features are assigned to the same dominant stream (Alimaskina et al., 2 Jun 2026). This is direct evidence that a nominally multi-stream residual connection can behave closer to a single-stream residual pathway.

The mitigation proposed there, Learned Stream Scaling (LSS), replaces exact replication by per-stream diagonal scaling

$$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$8

with $$\mathbf{X}_{l+1} = \mathcal{H}_l^{\mathrm{res}}\mathbf{X}_l + \bigl[\mathcal{H}_l^{\mathrm{post}}\bigr]^\top \mathcal{F}\bigl(\mathcal{H}_l^{\mathrm{pre}}\mathbf{X}_l,\mathcal{W}_l\bigr),$$9 initialized close to the all-ones vector (Alimaskina et al., 2 Jun 2026). Reported perplexity improvements for mHC-lite M were: OWT $\mathbf{X}_l\in\mathbb R^{n\times C}$0, WT$\mathbf{X}_l\in\mathbb R^{n\times C}$1 $\mathbf{X}_l\in\mathbb R^{n\times C}$2, WT$\mathbf{X}_l\in\mathbb R^{n\times C}$3 $\mathbf{X}_l\in\mathbb R^{n\times C}$4, and C4 $\mathbf{X}_l\in\mathbb R^{n\times C}$5 (Alimaskina et al., 2 Jun 2026). These results correct two common misconceptions: representational similarity is not equivalent to causal substitutability, and multi-stream residualization does not automatically imply balanced specialization.

6. Empirical scope across language, vision, graphs, and medical imaging

The original HC paper emphasized large-language-model pre-training and vision. On a $\mathbf{X}_l\in\mathbb R^{n\times C}$6 dense model trained for $\mathbf{X}_l\in\mathbb R^{n\times C}$7 tokens, the baseline reported V2 loss $\mathbf{X}_l\in\mathbb R^{n\times C}$8, PPL $\mathbf{X}_l\in\mathbb R^{n\times C}$9, downstream acc $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$0, whereas DHC$$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$1 reported V2 loss $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$2, PPL $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$3, acc $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$4. On a $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$5 dense model, the baseline reported V2 loss $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$6, PPL $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$7, acc $$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$8, whereas DHC$$\mathcal H_l^{\mathrm{pre}}\in\mathbb R^{1\times n}$$9 reported V2 loss $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$0, PPL $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$1, acc $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$2. On a $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$3 MoE model, ARC-Chall improved from $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$4 to $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$5, and the loss curves were reported to converge $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$6 faster (Zhu et al., 2024). In vision, DiT-XL/2 improved from FID $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$7, sFID $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$8, IS $$\mathcal H_l^{\mathrm{res}}\in\mathbb R^{n\times n}$$9 to SHC$x_k = f(W_k x_{k-1}) + x_{k-1}.$00 with FID $x_k = f(W_k x_{k-1}) + x_{k-1}.$01, sFID $x_k = f(W_k x_{k-1}) + x_{k-1}.$02, IS $x_k = f(W_k x_{k-1}) + x_{k-1}.$03, and ViT/16 Large improved from $x_k = f(W_k x_{k-1}) + x_{k-1}.$04 to DHC$x_k = f(W_k x_{k-1}) + x_{k-1}.$05 at $x_k = f(W_k x_{k-1}) + x_{k-1}.$06 (Zhu et al., 2024).

HC were then adapted beyond transformer LLMs. In graph learning, mHC-GNN expands node representations across $x_k = f(W_k x_{k-1}) + x_{k-1}.$07 parallel streams, constrains stream-mixing matrices to the Birkhoff polytope via Sinkhorn-Knopp normalization, and proves exponentially slower over-smoothing, with rate $x_k = f(W_k x_{k-1}) + x_{k-1}.$08 versus $x_k = f(W_k x_{k-1}) + x_{k-1}.$09, together with expressiveness beyond the $x_k = f(W_k x_{k-1}) + x_{k-1}.$10-Weisfeiler-Leman test (Mishra, 5 Jan 2026). Depth experiments from $x_k = f(W_k x_{k-1}) + x_{k-1}.$11 to $x_k = f(W_k x_{k-1}) + x_{k-1}.$12 layers were reported to show that standard GNNs collapse to near-random performance beyond $x_k = f(W_k x_{k-1}) + x_{k-1}.$13 layers, while mHC-GNN maintains over $x_k = f(W_k x_{k-1}) + x_{k-1}.$14 accuracy even at $x_k = f(W_k x_{k-1}) + x_{k-1}.$15 layers, with improvements exceeding $x_k = f(W_k x_{k-1}) + x_{k-1}.$16 percentage points at extreme depths; removing the manifold constraint causes up to $x_k = f(W_k x_{k-1}) + x_{k-1}.$17 performance degradation (Mishra, 5 Jan 2026).

In volumetric medical imaging, HC were introduced as a drop-in replacement for fixed residual connections across nnU-Net, SwinUNETR, VT-UNet, U-Net, and U-Netpp. On BraTS 2021, dynamic HC consistently improved all $x_k = f(W_k x_{k-1}) + x_{k-1}.$18D models, yielding up to $x_k = f(W_k x_{k-1}) + x_{k-1}.$19 percent mean Dice gain with negligible parameter overhead, and the gains were most pronounced in the Enhancing Tumor sub-region (Kumar et al., 20 Mar 2026). The same study reports that HC-equipped models develop sharper sensitivity toward clinically dominant sequences, specifically T1ce for Tumor Core and Enhancing Tumor, and FLAIR for Whole Tumor, while $x_k = f(W_k x_{k-1}) + x_{k-1}.$20D improvements were smaller and configuration-sensitive (Kumar et al., 20 Mar 2026).

A separate response to HC memory cost is Frac-Connections, which divide hidden states into multiple parts rather than expanding their width. That work states that Frac-Connections retain partial benefits of Hyper-Connections while reducing memory consumption, and reports large-scale language experiments up to a $x_k = f(W_k x_{k-1}) + x_{k-1}.$21B MoE model trained on up to $x_k = f(W_k x_{k-1}) + x_{k-1}.$22T tokens, where Frac-Connections significantly outperform residual connections (Zhu et al., 18 Mar 2025).

7. Conceptual status and open directions

The HC literature has converged on a stable set of technical themes. The first is that widening the residual stream and diversifying connectivity patterns can improve optimization and representation, but unconstrained residual matrices compromise the identity mapping property and therefore training stability (Xie et al., 31 Dec 2025). The second is that exact or spectrum-aware control of the residual mixer has become a central design objective, whether through doubly stochasticity, spectral-sphere constraints, or orthogonal/Stiefel constructions (Sengupta et al., 20 Feb 2026). The third is that routing structure and systems structure are intertwined: Sinkhorn iterations, permutation mixtures, Kronecker structure, transportation-polytope charts, and Newton-style dual solvers are simultaneously algorithmic parameterizations and hardware-relevant implementations (Wang et al., 26 May 2026).

Future directions stated within the literature include exploring other manifolds such as orthonormal, isometric, graph-Laplacian, and low-rank constraints under the same projection framework (Xie et al., 31 Dec 2025). JPmHC further frames the topic through Jacobian-spectrum preservation, free-probability analysis, implicit differentiation for fixed-point projections, and a Stiefel-constrained mixer via Cayley transforms, and reports on ARC-AGI that it achieves faster convergence, higher accuracy, and lower computational cost compared to bistochastic baselines (Sengupta et al., 20 Feb 2026). This suggests that ongoing HC research is moving from the original claim of “multiple residual streams” toward a more precise program: topological architecture design with explicit control of residual geometry, gradient conditioning, and stream utilization.