--- title: Depth-to-Width Ratio (DWR) url: https://www.emergentmind.com/topics/depth-to-width-ratio-dwr type: topic --- # Depth-to-Width Ratio (DWR) Searching arXiv for the cited papers to ground the article in current records. {"query":"id:2202.03841 OR id:2306.17759 OR id:2202.00553 OR id:1512.05180 OR id:2605.27989 OR id:2507.07431 OR id:2106.04013 OR id:2302.00453 OR id:2006.12467 OR id:2411.15267","max_results":10,"sort_by":"submittedDate","sort_order":"descending"} I’m going to look up the supplied arXiv identifiers to verify the records before writing the entry. {"query":"2202.03841 2306.17759 2202.00553 1512.05180 2605.27989 2507.07431 2106.04013 2302.00453 2006.12467 2411.15267","max_results":10} Depth-to-Width Ratio (DWR) is a dimensionless quantity that compares a system’s depth to a corresponding notion of width. In contemporary theory, the term is not attached to a single universal formula: for ReLU networks it is used as $L/n$ [2202.03841], for proportional-limit attention models as $T=d/n$ [2306.17759], for deep fully connected NTK analysis as $\lambda=L/M$ [2202.00553], for decoder-only Transformer shape studies as $R_{D/W}=L/d_{\mathrm{model}}$ [2605.27989], for products of rectangular Ginibre matrices as $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$ [2507.07431], and in true 2D granular collapse as the aspect ratio $a=h_0/d_0$ [1512.05180]. Across these settings, DWR functions as a control parameter: it specifies how accumulated compositional depth, multiplicative layering, or vertical extent scales relative to the available lateral degrees of freedom. ## 1. Definitions and notational variants The principal difficulty in interpreting DWR is terminological rather than conceptual: different literatures use different width variables, different asymptotic regimes, and different operational meanings. | Domain | DWR definition | Width notion | |---|---|---| | ReLU, NTK, ResNet, deep linear theory | $L/n$, $d/n$, $\lambda=L/M$, $\beta=\sum_i 1/n_i$, $\rho=L(N)/N$ | hidden-layer width or common width scale | | Transformers and self-attention | $D/W=L/d_x$, $R_{D/W}=L/d_{\mathrm{model}}$, $T=d/n$ | hidden size or feature dimension | | Rectangular products and granular flows | $\Delta_{M,N}=\sum_{j=0}^{M}\frac{1}{N+v_j}$, $a=h_0/d_0$ | factor width or initial column width | In the ReLU approximation setting, width is defined as $n:=\max\{d,n_1,\dots,n_{L-1}\}$, depth is $L$, and the parameter count is at most $O(n^2\cdot L)$ [2202.03841]. In the proportional-limit NTK setting, the constant-width case takes $n_1=\cdots=n_{L-1}=M$ and identifies the DWR as $\lambda=L/M$ [2202.00553]. In the finite-depth finite-width NTK corrections literature, the relevant quantity is the inverse-temperature parameter $\beta:=\sum_{i=1}^d (1/n_i)$, which reduces to $d/n$ for equal widths [1909.05989]. In residual-network proportional limits, the notation $c=d/n$ is used for fully connected ReLU ResNets [2106.04013], whereas in deep linear networks the proportional limit is expressed as $L(N)/N\to a\in[0,\infty)$ [2411.15267]. Transformer theory uses several non-equivalent conventions. In self-attention scaling laws, depth is the number of stacked self-attention layers $L$, width is the internal representation dimension $d_x$, and DWR is $L/d_x$ [2006.12467]. In the shaped Transformer, width $n$ denotes the feature dimension of token embeddings, depth $d$ is the number of layers, and the proportional-limit time horizon is $T=d/n$ [2306.17759]. In the neural-interaction study, width is fixed as $d_{\mathrm{model}}$, depth is $L$, and the paper uses both $R_{D/W}$ and $\alpha=L/d_{\mathrm{model}}$ [2605.27989]. A common misconception is that DWR is a single architecture-independent invariant. The literature instead treats it as a family of dimensionless shape parameters whose exact form depends on what “depth” and “width” mean in the model under study. This is especially visible when comparing neural-network theory with rectangular random matrix products or granular-flow aspect ratios. ## 2. Expressivity and approximation in ReLU networks For ReLU networks, DWR is most directly tied to expressivity through the asymmetry between depth and width. The central result is that any target ReLU network with inputs in $\mathbb{R}^d$ can be approximated by a width-$O(d)$ network whose number of parameters is essentially larger only by a linear factor, whereas depth-separation theorems imply that an analogous replacement of depth by width cannot hold in general [2202.03841]. The formal approximation theorem considers a target network $N_0:[-A,A]^d\to\mathbb{R}$ of width $n$, depth $L$, and weights bounded in $[-B,B]$, together with a distribution $D$ on $[-A,A]^d$ whose density satisfies $p_D(x)\le \beta$. For any $\epsilon,\delta>0$, there exists a ReLU network $N:[-A,A]^d\to\mathbb{R}$ of width at most $\max\{5d,10\}$ and depth $O(n^2L^2\cdot \log(ABn\cdot \epsilon^{-1}))$ such that, with probability at least $1-\delta$ over $x\sim D$, one has $|N(x)-N_0(x)|\le \epsilon$. The number of parameters in $N$ is $O(n^2L^2\cdot \log(ABn\cdot \epsilon^{-1}))$. Since $N_0$ has $O(n^2L)$ parameters, the overhead is essentially linear in $L$, up to logarithmic factors. The same paper proves a close-to-minimal-width construction. Under the same assumptions, there exists a network of width at most $\max\{d+2,10\}$ and depth $O(n^2L^2\cdot \log(ABn\cdot \epsilon^{-1}))$ with probability at least $1-\delta$ approximation error at most $\epsilon$. Its parameter count is $O(d\cdot n^2L^2\cdot \log(ABn\cdot \epsilon^{-1})^2)$, which becomes $O(n^2L^2\cdot \log(ABn\cdot \epsilon^{-1})^2)$ if $n\ge d^{1.5}$. This is close to the minimal-width threshold because width $1$. In the infinite-depth-and-width limit, the normalized NTK dispersion $$ \frac{\mathbb{E}[\Theta(x,x)^2]}{\mathbb{E}[\Theta(x,x)]^2} $$ converges to $1$ in the ordered phase, but grows exponentially in DWR at the edge of chaos and in the chaotic phase. In the chaotic phase, the limit is $$ \frac{1}{2\lambda} e^{5\lambda}\left(1-\frac{1}{4\lambda}(1-e^{-4\lambda})\right). $$ The same work shows that the NTK may stay constant during training only in the ordered phase. In the chaotic phase, even the first GD step forces a relative NTK change that diverges unless the learning rate decays as $O(a^{-L})$. The finite-depth finite-width correction theory arrives at the same structural conclusion through a different parameterization. There the DWR is $\beta=\sum_{i=1}^d (1/n_i)$, which becomes $d/n$ in the equal-width case, and the on-diagonal NTK satisfies $$ \mathbb{E}[K(x,x)] = d/2 + (d/n_0)\|x\|_2^2. $$ For equal widths, the normalized second moment obeys $$ \frac{\mathbb{E}[K(x,x)^2]}{\mathbb{E}[K(x,x)]^2}\asymp e^{5\beta}(1+O(\beta/n)), $$ so the standard deviation is exponential in DWR. The paper also proves that the mean of the first SGD update scales as $$ \frac{\mathbb{E}[\Delta K(x,x)]}{\mathbb{E}[K(x,x)]}\asymp \lambda \times (d\beta/n_0)\times e^{5\beta}\times (1+O(d/n^2)), $$ again showing exponential sensitivity to DWR [1909.05989]. Taken together, these results establish a sharp limitation of the classical fixed-depth infinite-width regime. When depth remains negligible relative to width, the NTK concentrates and training stays close to lazy dynamics. When DWR stays bounded away from zero, finite-width corrections accumulate across layers instead of averaging out. This suggests that DWR is not merely a geometric descriptor of architecture but a dynamical parameter for kernel stochasticity and feature learning. ## 4. Residual and linear networks beyond the classical Gaussian-process limit Residual architectures change the role of DWR because skip connections alter how layerwise fluctuations accumulate. For fully connected residual networks with branch scaling $1/\sqrt{L}$, width and depth limits commute: the pre-activations converge to Gaussian distributions and the covariance kernel converges to the same deterministic ODE limit regardless of whether width tends to infinity first, depth tends to infinity first, or both diverge jointly with finite DWR [2302.00453]. In that setting, for fixed $t\in[0,1]$ and a fixed neuron coordinate, $Y_{\lfloor tL\rfloor,i}(x)$ converges in distribution to $\mathcal{N}(0,v(t,x))$ with $$ v(t,x)=\frac{\|x\|^2}{d}\exp(t/2), $$ and the convergence rate is bounded by $C\cdot (1/\sqrt{n}+1/\sqrt{L})$. The same analysis argues that $1/\sqrt{L}$ is the only nontrivial residual scaling: if the branch is scaled by $L^{-\alpha}$ with $\alpha<1/2$, covariance explodes; if $\alpha>1/2$, the residual contribution vanishes. A different residual theory studies fully connected ReLU ResNets in the simultaneous infinite-depth-and-width limit with $c=d/n$ held fixed. There the output at initialization is not Gaussian but log-Gaussian: $$ y \stackrel{d}{=} \frac{\|x\|}{\sqrt{n_{\mathrm{in}}}}(\alpha^2+\lambda^2)^{d/2}\exp\!\left(\frac{1}{2}G\right)\vec{Z}, $$ where $\vec{Z}$ has i.i.d. standard normal entries and $G$ converges to a Gaussian random variable whose mean and variance depend on $c=d/n$, the skip/residual strengths, hypoactivation, and interlayer correlations [2106.04013]. The same work shows that vanilla ResNets are hypoactivated at initialization: fewer than half of the ReLUs are activated on average. It also identifies positive interlayer correlations that produce exponential variance growth in the output. The proposed Balanced ResNets randomize the sign of the nonlinearity, eliminate hypoactivation and interlayer activation-pattern correlations at initialization, and retain log-Gaussian behavior with substantially reduced variance growth. Deep linear networks exhibit yet another proportional-limit alternative to the NNGP picture. With equal hidden widths $N$, depth $L(N)$, and $L(N)/N\to \rho$, the case $\rho=0$ recovers the classical Gaussian prior $$ f_{L,N}(X|\theta)\xrightarrow{d} ZN_0^{-1/2}X, $$ where $Z$ has i.i.d. standard normal entries. For $\rho>0$, the limit is instead $$ f_{L,N}(X|\theta)\xrightarrow{d} V^{\infty}ZN_0^{-1/2}X, $$ where $V^{\infty}$ is a random lower-triangular matrix built from Brownian motions and multiple stochastic integrals. Conditional on $Q^{\infty}=V^{\infty}(V^{\infty})^\top$, the prior is Gaussian; unconditionally it is a nontrivial Gaussian mixture. Under Gaussian likelihood, the posterior remains a mixture, and the unconditional posterior covariance depends on the observed labels when $\rho>0$ [2411.15267]. A common simplification in deep-learning folklore is that “infinite width” automatically implies Gaussianity. The residual and deep-linear proportional-limit results show that this is true only when depth does not scale with width in a way that preserves a nonzero DWR, or when the architecture and scaling eliminate the relevant finite-width accumulations. ## 5. Transformer architectures: expressivity, stability, and interaction efficiency In self-attention, DWR governs at least three distinct phenomena: depth-efficiency of expressivity, covariance stability at initialization, and fixed-budget interaction efficiency. For expressivity, self-attention admits a width-dependent transition between depth-efficient and depth-inefficient regimes. With depth $D=L$, width $W=d_x$, and parameter count approximately $P\approx \alpha D W^2$ with $\alpha=2m+4$ and $\alpha=12$ when $m=4$, the separation-rank analysis identifies a threshold $$ L_{\mathrm{th}}(W)\approx \log_3 W. $$ If $L<\log_3 W$, separation rank grows double-exponentially in depth and only polynomially in width. If $L>\log_3 W$, the width caps the benefit of further depth, and the bounds become essentially multiplicative in $WL$. Empirical ablations on decoder-only Transformers of depths $6$ to $48$ fit the crossover width by $$ \log W_*(L)\approx a+bL, $$ with $a=5.039\pm 0.030$ and $b=5.55\times 10^{-2}\pm 1.3\times 10^{-3}$, giving a transition-size curve $$ P_{\mathrm{transition}}(L)=12Le^{2a}e^{2bL}. $$ The same fit yields projected optima such as $L\approx 80$, $W\approx 13500$ for $175$B non-embedding parameters and $L\approx 95$, $W\approx 30100$ for $1$T, leading the authors to describe GPT-3 as too deep for its size [2006.12467]. The shaped Transformer studies DWR from the standpoint of initialization stability. In the proportional limit $d,n\to\infty$ with finite $T=d/n>0$, the neural covariance $V_\ell=(1/n)X_\ell X_\ell^\top$ evolves as an Euler–Maruyama discretization and converges locally to an SDE indexed by $T$. To obtain a stable limit, the attention matrix is modified to $$ A_\ell = I + \mathrm{Softmax}(\tau^{-1}Y_\ell) - \frac{1}{m}\mathbf{1}\mathbf{1}^\top, \qquad \tau=\tau_0\sqrt{nn_k}, $$ with residual parameters satisfying $\lambda^2+\gamma^2=1$. Centering removes the non-infinitesimal zeroth-order drift of unshaped Softmax, the identity bias keeps attention near $I$, and $\gamma$ rescales both drift and diffusion, acting as a time change that reduces effective accumulated depth. Without this shaping, the paper reports exploding covariance and rapid rank collapse at large depth; with shaping, the SDE remains stable and finite models match its predictions well [2306.17759]. The neural-interaction study considers fixed parameter budgets for decoder-only Transformers and defines DWR as $$ R_{D/W}=\frac{L}{d_{\mathrm{model}}}. $$ Interaction is measured through the Average Gradient Outer Product and its off-diagonal energy: AOFE and AOFE-ratio. For Tiny byte-level language models on WikiText-103, the budget-wise best shapes show correlations $$ r(L_{\text{test}},\mathrm{AOFE})=+0.883,\qquad r(L_{\text{test}},\mathrm{AOFE\text{-}Ratio})=-0.967, $$ and the best DWRs for budgets at or above $1.0$M parameters concentrate in an “interaction-efficient interval” $$ 0.023\le \frac{L}{d_{\mathrm{model}}}\le 0.047. $$ The paper also reports that smaller distance to this interval tends to correlate with higher MMLU-Pro among comparable small dense LLMs, while explicitly noting that direct transfer of the numeric interval to billion-parameter systems would be premature [2605.27989]. These Transformer results point in different directions unless their assumptions are kept separate. High depth can be expressively advantageous below the self-attention threshold, but proportional depth can destabilize initialization unless attention is shaped, and under a fixed budget there may be an intermediate interaction-efficient interval rather than a monotone preference for either depth or width. ## 6. Random matrix products and non-neural aspect-ratio usage Outside standard neural-network architecture theory, DWR appears as a sharp asymptotic control parameter in products of rectangular complex Ginibre matrices. For a product $$ Y_M=X_MX_{M-1}\cdots X_1, $$ with $X_j$ of size $(N+v_j)\times (N+v_{j-1})$, the DWR is defined as $$ \Delta_{M,N}:=\sum_{j=0}^{M}\frac{1}{N+v_j}. $$ This quantity reduces to $(M+1)/N$ in the square case $v_j\equiv 0$. The local soft-edge statistics of the log singular values exhibit a three-phase transition determined entirely by the asymptotic value of $\Delta_{M,N}$: if $\Delta_{M,N}\to 0$, the Airy kernel appears at the soft edge; if $\Delta_{M,N}\to \gamma\in(0,\infty)$, the statistics are governed by the $\gamma$-critical kernel; if $\Delta_{M,N}\to\infty$, the edge fluctuations are Gaussian and higher-order local correlations vanish [2507.07431]. In this setting, DWR measures aggregate inverse width across a multiplicative chain rather than layer count divided by hidden size, but it serves the same structural role of separating distinct asymptotic regimes. The term also appears in an entirely different sense in granular mechanics. In true 2D granular column collapse, DWR is simply the initial aspect ratio $$ a=\frac{h_0}{d_0}, $$ with $h_0$ the initial height and $d_0$ the initial width. Experiments with aluminum rods over the range $0.25\le a\le 8$ identify two thresholds. For $a\le 0.65$, the final height is unchanged, $h/h_0=1$, and an undisturbed top zone persists. For $a>0.65$, the top surface becomes conical and the measured relations are $$ \frac{h_0}{h}=1.42a^{2/3},\qquad \frac{d}{d_0}=4.30a^{0.72}, $$ together with two run-out branches, $$ \frac{d-d_0}{d_0}=3.25a^{0.96}\quad\text{for }0.65