Dependency Depth Bias in Deep Learning

• Dependency depth bias is a phenomenon where deep learning models select structured solutions based on how depth affects parameter interactions and training dynamics.
• It is characterized by effective depth in classification and implicit low-rank bias in matrix factorization and unconstrained feature models.
• Understanding these depth-induced dependencies helps inform network architecture design and training strategies for better generalization and stability.

Searching arXiv for recent papers on dependency depth bias, neural collapse, and depth-induced low-rank implicit bias.
Dependency depth bias denotes a family of implicit-bias phenomena in deep learning in which optimization does not merely fit with the available nominal depth, but instead selects structured solutions shaped by how depth enters the parameterization and the training dynamics. In classification, one manifestation is a bias toward small effective depth, defined as the first layer at which nearest-class-center separability is achieved under stochastic gradient descent [2202.09028]. In deep unconstrained-feature models and deep linear matrix factorization, a distinct but related manifestation is that multiplicative depth induces an implicit low-rank bias: low-rank matrices propagate norm more efficiently through successive multiplications, and cross-layer coupling becomes endemic once depth is at least three, favoring low-rank alternatives to Neural Collapse or high-rank matrix fits [2605.23087]; [2603.04703]. This suggests that dependency depth bias is best understood as a depth-driven restructuring of the optimization landscape, the induced dependencies among parameters and examples, and the effective complexity of the learned solution.

1. Core formulations and scope

Two technical lines of work make the notion precise in different settings. One line studies effective depth in nonlinear classifiers trained by SGD, using the earliest layer at which a nearest-class-center classifier achieves low error as the operative measure of depth [2202.09028]. A second line studies deep linear or UFM-style parameterizations, where the multiplicative structure itself creates coupled dynamics and favors low-rank solutions [2605.23087]; [2603.04703].

Setting	Depth object	Main depth effect
Classification with SGD	$\epsilon$-effective depth	SGD implicitly selects small effective depth
Deep UFM with multiclass cross-entropy	Multiplicative head depth $D$	Depth induces implicit low-rank bias and favors softmax codes
Deep matrix completion	Factorization depth $L$	Depth $\geq 3$ enforces coupling and promotes rank-1 convergence

In the classification formulation, the relevant question is when class structure becomes recoverable by a simple NCC rule. In the deep UFM and matrix-completion formulations, the relevant question is how multiplicative depth changes norm propagation, singular-value dynamics, and coupling among gradients. The common theme is that depth changes the dependencies that matter during optimization, rather than acting only as a static expressivity parameter.

2. Effective depth as the first layer of NCC separability

For a network of depth $L$ with partial feature maps
[
f_i(x)=g^i\circ g^{i-1}\circ\cdots\circ g^1(x),
]
effective depth is defined through nearest-class-center (NCC) separability. At layer $i$, the induced classifier is
[
\hat h_i(x)=\arg\min_{c\in[C]}\;\bigl|f_i(x)-\mu_{f_i}(S_c)\bigr|,
\quad
\mu_{f_i}(S_c)=\frac1{|S_c|}\sum_{(x,y)\in S_c}f_i(x),
]
and the $\epsilon$-effective depth is
[
d_S^\epsilon(h)=\min\Bigl{i\in{1,\dots,L}\mid \err_S(\hat h_i)\le\epsilon\Bigr},
]
or $L$ if no such layer exists [2202.09028].

This definition operationalizes a minimal-depth hypothesis: on a fixed dataset $S$, there is an integer $L_0$ such that if one trains any network of depth $L\ge L_0$ with SGD and weight decay, then adding extra layers beyond $L_0$ does not push back the first layer at which NCC separability occurs. The reported empirical pattern is that, as $L$ grows large, the NCC train-accuracy at intermediate layers saturates to near $100\%$ at a fixed layer $L_0$, and does so at about the same epoch for all deeper models [2202.09028].

The same work connects this bias to generalization through the $\epsilon$-minimal NCC depth
[
d_{\min}^\epsilon(G,S)
=\min\Bigl{L\mid \exists\text{ a network }f^L\in G^L\text{ with }\err_S(\hat h)\le\epsilon\Bigr}.
]
Under two technical assumptions—uniformly distributed mistakes on a hold-out split and stochastic increase of the minimal depth needed to fit randomly corrupted labels—they prove a bound of the form
[
\E_{S_1,\gamma}\bigl[\err_P(h_{S_1})\bigr]
\le
\Pr\Bigl[E_\gamma[d_{S_1}^\epsilon(h_{S_1})]\ge d_{\min}^\epsilon(G,S_1\cup S_2)\Bigr]
+(1+\alpha)p+\delta_m^1+\delta_{m,p,\alpha}^2.
]
The substantive interpretation is that expected test error is controlled by whether the learned effective depth exceeds the depth required to fit a mixture of clean and random labels, rather than by nominal depth or parameter count alone [2202.09028].

Empirically, the effective depth increases monotonically with label noise. On CIFAR10 with a depth-10 CONV-400 model, perfect NCC separability occurs at layer $i=5$ for $p=0\%$, at $i=6$ for $p=10\%$ and $p=25\%$, at $i=7$ for $p=50\%$, and at $i=8$ for $p=75\%$ [2202.09028]. This supports the interpretation that effective depth measures a nontrivial complexity axis that tracks the distinction between fitting signal and fitting noise.

3. Depth-induced low-rank bias in deep UFMs

In the deep unconstrained-feature model, a $K$-class classifier is represented by a learned feature matrix $F\in\mathbb R^{d\times (Kn)}$ and a depth-$D$ linear head with logits
[
z_{c,i}=W_DW_{D-1}\cdots W_1f_{c,i},
\qquad
Z=W_D\cdots W_1F.
]
Training minimizes the unregularized multiclass cross-entropy
[
L(F,W)=\sum_{c=1}^K\sum_{i=1}^n
\Bigl[
-\log \frac{\exp((z_{c,i})c)}{\sum{c'=1}^K\exp((z_{c,i})_{c'})}
\Bigr].
]
Gradient flow drives $L\to 0$ only at infinite parameter norm; among all diverging solutions the dynamics select particular geometric configurations, yielding an implicit bias [2605.23087].

The central mechanism is that, under balancedness, the factors $W_D,\dots,W_1,F$ grow in Frobenius norm in lockstep, so asymptotically the geometric degree of freedom is the direction of the combined logit matrix $Z$ subject to $|Z|_F\to\infty$. For normalized factors,
[
\widehat Z=\widehat W_D\cdots \widehat W_1\widehat F,
]
maximizing the logits’ magnitudes boils down to maximizing $|\widehat Z|_F$ at fixed factor norms [2605.23087].

The key linear-algebra fact is that, for fixed Frobenius norms, low-rank matrices have larger singular values and therefore suffer less decay under repeated multiplications. If all normalized layers share the same nonzero singular values $\sigma_1\ge\cdots\ge\sigma_r>0$, then
[
|\widehat Z|F
=\sqrt{\sum{i=1}^r(\sigma_i)^{2D}}
\ge
\sqrt{r\Bigl(\frac1r\sum_i \sigma_i\Bigr)^{2D}}.
]
As $D$ grows, concentrating singular-value mass on fewer modes increases $|\widehat Z|_F$ exponentially in $D$. Comparing the full-rank Neural Collapse solution with a rank-$r$ solution gives
[

\frac{|\widehat Z_{\rm NC}|F}{|\widehat Z{\rm low-rank}|_F}

\Bigl(\frac{K-1}{r}\Bigr)^{D/2}.
]
Thus low-rank structures achieve much larger logits for the same factor norms and hence smaller cross-entropy loss [2605.23087].

Within this model, the asymptotic max-margin landscape changes with depth: the global optimum is not Neural Collapse but a rank-2 softmax code in the $D\to\infty$ limit. This directly contradicts the idea that Neural Collapse is always the asymptotically preferred geometry in deep classifiers [2605.23087].

4. Training dynamics, singular-value repulsion, and the Neural Collapse basin

To analyze finite-time behavior, the deep UFM study imposes a spectral or Hadamard initialization that fixes the singular vectors of all layers and the feature matrix to those of a Sylvester-Hadamard matrix, so only the singular values $a_i(t)$ evolve. The resulting ODE is
[
\frac{da_i}{dt}=D(t)\,[b_i(t)a_i(t)],
]
with
[
b_i(t)=\sum_{j=1}^{K-1}V_{ij}\exp\Bigl(-\sum_{\ell=0}^D a_j(t)\Bigr),
\qquad
D(t)=1+\sum_{j=1}^{K-1}\exp\Bigl(-\sum_{\ell=0}^D a_j(t)\Bigr),
]
where $V$ is the core ETF Gram-difference matrix [2605.23087].

In the linearized regime $|a|_1\ll 1$, the flow satisfies
[

\frac{da_i}{dt}

\frac1{2^D}\bigl[a_i(t)\,|a(t)|_1\bigr]
+
O(|a|_1^{D+2}),
]
and therefore
[
\frac{d}{dt}\Bigl(\frac{a_i}{a_j}\Bigr)\approx \frac{1}{2^D}(a_i-a_j).
]
Larger modes thus grow relatively faster than smaller ones. The paper describes this as a rich-get-richer effect or singular-value repulsion, with low-rank structure emerging before significant norm growth [2605.23087].

This dynamic mechanism reshapes the basin of attraction for Neural Collapse. In the shallow UFM with $D=1$, the KL divergence to the ETF direction is a Lyapunov function and Neural Collapse is the unique attractor from any positive initialization. For $D>1$, by contrast, the Neural Collapse direction becomes unstable for small $|a|_1$: there is a threshold
[
|a|_1<(K-1)2^{-D+1}
]
below which NC is repulsive under the linearized ODE. Alternative low-rank directions, including rank-2 softmax codes, acquire their own basins of attraction above a norm threshold, and the KL divergence to NC need not decrease monotonically and can diverge along certain gradient-flow trajectories [2605.23087].

A further complication is that not all depth-adjacent effects point toward lower rank. For randomly initialized networks, increasing head width $d$ biases the initial logit derivative toward the ETF structure,
[
\frac{dZ}{dt}\Bigl|_{t=0}\xrightarrow[p]{d\to\infty} 2^D\,S\otimes \mathbf1_n^T,
]
where $S=I_K-(1/K)11^T$. By continuity, sufficiently large width causes the trajectory to spend an $O(1)$ interval close to the NC direction before the depth-driven rich-get-richer effect can act [2605.23087]. A common misconception is therefore that depth alone completely determines the outcome; the reported result is instead a competition between depth-induced low-rank bias and width-induced stabilization of higher-rank NC-like geometry at early times.

5. Coupled dynamics and rank-1 bias in deep matrix completion

A second formalization of dependency depth bias appears in deep matrix factorization for matrix completion. Here the predictor is
[
\hat X=W_LW_{L-1}\cdots W_1,
]
and the loss is
[
\ell(W_1,\dots,W_L)=\frac12\sum_{(i,j)\in\Omega}(\hat X_{ij}-X^*_{ij})^2.
]
Under gradient flow,
[
\frac{dW_\ell}{dt}=-\frac{\partial \ell}{\partial W_\ell}
=-A_\ell(\hat X-X^*)B_\ell,
]
with $A_\ell=W_L\cdots W_{\ell+1}$ and $B_\ell=W_{\ell-1}\cdots W_1$ [2603.04703].

The distinction between shallow and deep models is formulated through coupled versus decoupled dynamics. Training is decoupled with respect to a partition $\Omega=\cup_{k=1}^K\Omega_k$ if, for any entries from different parts,
[
\bigl\langle \nabla_\theta \hat X_{ij}(t),\nabla_\theta \hat X_{pq}(t)\bigr\rangle =0
\quad \forall\, t\ge 0.
]
Otherwise the dynamics are coupled. For depth $L=2$, disconnected observations can evolve independently. For depth $L\ge 3$, however, intermediate factors appear in every observed entry’s gradient, so unless a layer is initialized in a special diagonal form, gradients share parameters and therefore couple [2603.04703].

The depth-induced coupling theorem states that for any depth $L\ge 3$, if the layers are initialized by an absolutely continuous distribution such as i.i.d. Gaussian, then with probability $1$ the gradients for any two distinct observed entries have nonzero inner product at initialization; hence the dynamics are coupled almost surely, regardless of $\Omega$ [2603.04703].

For block-diagonal observations and the initialization family
[
W_\ell(0)=\alpha I_d+\beta J_d,
]
the singular-value dynamics can be characterized explicitly. When $L=2$, the solution retains multiple comparable singular values and need not drop rank. When $L\ge 3$ and initialization is generic, the nonzero singular values satisfy the implicit system
[
\sigma_1+(n-1)\sigma_2=s\,w^*,
\qquad
\sigma_1^{-2/L}-\sigma_2^{-2/L}=C_{m,d,L},
]
with $C_{m,d,L}\to 0$ as $\alpha\to 0$, implying $\sigma_2/\sigma_1\to 0$ and stable rank $\to 1$ as $\alpha\to 0$ [2603.04703]. In this block-diagonal setting, the paper concludes that
converges to rank-1 if and only if the dynamics are coupled.

This is the paper’s strongest formulation of dependency depth bias: depth yields endemic coupling that biases toward simpler, rank-1 solutions. A plausible implication is that multiplicative depth can act as an inductive-bias control even when the nominal architecture remains heavily overparameterized.

6. Plasticity, generalization, and conceptual synthesis

The matrix-completion results also address loss of plasticity. In depth-2 models, pre-training on sparse or disconnected observations can fit the data while remaining high-rank, and a subsequent warm start on an augmented observation set can fail to reduce the rank, even though a cold start on the full set would produce a low-rank solution. A detailed $2\times 2$ analysis shows exponential decay of the new-data loss while stable rank stays bounded away from $1$ [2603.04703]. By contrast, for $L\ge 3$, inherent coupling forces realignment at every stage, and numerical experiments show that deep models avoid plasticity loss and quickly reconverge to near rank $1$.

Across the three formulations, several objective clarifications emerge. First, depth and effective depth are not interchangeable. A nominally deep model may behave as though it were shallow in the NCC sense because SGD finds the earliest layer at which classes separate [2202.09028]. Second, Neural Collapse is not universally the asymptotic endpoint of depth. In deep UFMs trained without regularization, the global optimum shifts toward rank-2 softmax codes as $D\to\infty$ [2605.23087]. Third, low-rank bias is not a generic consequence of any overparameterization; in matrix completion it depends on coupled dynamics, and in the block-diagonal case rank-1 convergence is equivalent to coupling [2603.04703].

These works therefore place dependency depth bias at the intersection of optimization geometry, representation collapse, and inductive bias. In one direction, depth can be effectively ignored because SGD attains NCC separability as early as possible; in another, depth fundamentally changes the training dynamics because successive multiplications and shared parameters favor low-rank norm propagation or cross-entry coupling. This suggests that the term is best reserved for situations in which depth changes which dependencies are active during learning—between layers, singular modes, classes, or observed entries—and thereby changes the selected solution even when many solutions interpolate the data.