Depth-to-Width Ratio (DWR)

• Depth-to-Width Ratio (DWR) is a dimensionless ratio that compares a system’s depth to its width, with different definitions tailored to neural networks, random matrices, and granular flows.
• DWR controls key phenomena such as expressivity in ReLU networks, kernel variability in NTK theory, and stability in Transformer architectures.
• This ratio serves as a crucial control parameter that determines phase transitions and practical trade-offs between depth and width across various theoretical and applied settings.

Searching arXiv for the cited papers to ground the article in current records. {"query":"id:(Vardi et al., 2022) OR id:(Noci et al., 2023) OR id:(Seleznova et al., 2022) OR id:(Nguyen et al., 2015) OR id:(Sun et al., 27 May 2026) OR id:(Gu, 10 Jul 2025) OR id:(Li et al., 2021) OR id:(Hayou et al., 2023) OR id:(Levine et al., 2020) OR id:(Bassetti et al., 2024)","max_results":10,"sort_by":"submittedDate","sort_order":"descending"} I’m going to look up the supplied arXiv identifiers to verify the records before writing the entry. {"query":"(Vardi et al., 2022, Noci et al., 2023, Seleznova et al., 2022, Nguyen et al., 2015, Sun et al., 27 May 2026, Gu, 10 Jul 2025, Li et al., 2021, Hayou et al., 2023, Levine et al., 2020, Bassetti et al., 2024)","max_results":10} Depth-to-Width Ratio (DWR) is a dimensionless quantity that compares a system’s depth to a corresponding notion of width. In contemporary theory, the term is not attached to a single universal formula: for ReLU networks it is used as $L/n$ (Vardi et al., 2022), for proportional-limit attention models as $T=d/n$ (Noci et al., 2023), for deep fully connected NTK analysis as $\lambda=L/M$ (Seleznova et al., 2022), for decoder-only Transformer shape studies as $R_{D/W}=L/d_{\mathrm{model}}$ (Sun et al., 27 May 2026), for products of rectangular Ginibre matrices as $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$ (Gu, 10 Jul 2025), and in true 2D granular collapse as the aspect ratio $a=h_0/d_0$ (Nguyen et al., 2015). Across these settings, DWR functions as a control parameter: it specifies how accumulated compositional depth, multiplicative layering, or vertical extent scales relative to the available lateral degrees of freedom.

1. Definitions and notational variants

The principal difficulty in interpreting DWR is terminological rather than conceptual: different literatures use different width variables, different asymptotic regimes, and different operational meanings.

Domain	DWR definition	Width notion
ReLU, NTK, ResNet, deep linear theory	$L/n$, $d/n$, $\lambda=L/M$, $\beta=\sum_i 1/n_i$, $T=d/n$0	hidden-layer width or common width scale
Transformers and self-attention	$T=d/n$1, $T=d/n$2, $T=d/n$3	hidden size or feature dimension
Rectangular products and granular flows	$T=d/n$4, $T=d/n$5	factor width or initial column width

In the ReLU approximation setting, width is defined as $T=d/n$6, depth is $T=d/n$7, and the parameter count is at most $T=d/n$8 (Vardi et al., 2022). In the proportional-limit NTK setting, the constant-width case takes $T=d/n$9 and identifies the DWR as $\lambda=L/M$0 (Seleznova et al., 2022). In the finite-depth finite-width NTK corrections literature, the relevant quantity is the inverse-temperature parameter $\lambda=L/M$1, which reduces to $\lambda=L/M$2 for equal widths (Hanin et al., 2019). In residual-network proportional limits, the notation $\lambda=L/M$3 is used for fully connected ReLU ResNets (Li et al., 2021), whereas in deep linear networks the proportional limit is expressed as $\lambda=L/M$4 (Bassetti et al., 2024).

Transformer theory uses several non-equivalent conventions. In self-attention scaling laws, depth is the number of stacked self-attention layers $\lambda=L/M$5, width is the internal representation dimension $\lambda=L/M$6, and DWR is $\lambda=L/M$7 (Levine et al., 2020). In the shaped Transformer, width $\lambda=L/M$8 denotes the feature dimension of token embeddings, depth $\lambda=L/M$9 is the number of layers, and the proportional-limit time horizon is $R_{D/W}=L/d_{\mathrm{model}}$0 (Noci et al., 2023). In the neural-interaction study, width is fixed as $R_{D/W}=L/d_{\mathrm{model}}$1, depth is $R_{D/W}=L/d_{\mathrm{model}}$2, and the paper uses both $R_{D/W}=L/d_{\mathrm{model}}$3 and $R_{D/W}=L/d_{\mathrm{model}}$4 (Sun et al., 27 May 2026).

A common misconception is that DWR is a single architecture-independent invariant. The literature instead treats it as a family of dimensionless shape parameters whose exact form depends on what “depth” and “width” mean in the model under study. This is especially visible when comparing neural-network theory with rectangular random matrix products or granular-flow aspect ratios.

2. Expressivity and approximation in ReLU networks

For ReLU networks, DWR is most directly tied to expressivity through the asymmetry between depth and width. The central result is that any target ReLU network with inputs in $R_{D/W}=L/d_{\mathrm{model}}$5 can be approximated by a width-$R_{D/W}=L/d_{\mathrm{model}}$6 network whose number of parameters is essentially larger only by a linear factor, whereas depth-separation theorems imply that an analogous replacement of depth by width cannot hold in general (Vardi et al., 2022).

The formal approximation theorem considers a target network $R_{D/W}=L/d_{\mathrm{model}}$7 of width $R_{D/W}=L/d_{\mathrm{model}}$8, depth $R_{D/W}=L/d_{\mathrm{model}}$9, and weights bounded in $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$0, together with a distribution $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$1 on $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$2 whose density satisfies $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$3. For any $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$4, there exists a ReLU network $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$5 of width at most $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$6 and depth $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$7 such that, with probability at least $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$8 over $\Delta_{M,N}=\sum_{j=0}^{M} \frac{1}{N+v_j}$9, one has $a=h_0/d_0$0. The number of parameters in $a=h_0/d_0$1 is $a=h_0/d_0$2. Since $a=h_0/d_0$3 has $a=h_0/d_0$4 parameters, the overhead is essentially linear in $a=h_0/d_0$5, up to logarithmic factors.

The same paper proves a close-to-minimal-width construction. Under the same assumptions, there exists a network of width at most $a=h_0/d_0$6 and depth $a=h_0/d_0$7 with probability at least $a=h_0/d_0$8 approximation error at most $a=h_0/d_0$9. Its parameter count is $L/n$0, which becomes $L/n$1 if $L/n$2. This is close to the minimal-width threshold because width $L/n$3 is not universal on compact domains, and prior work places minimal universal width on the order of $L/n$4.

The bounded-weight variant shows that width reduction does not intrinsically require unbounded coefficients. There exists an approximant with width at most $L/n$5, weights bounded by $L/n$6, and depth $L/n$7, at the cost of an extra polynomial factor in depth and parameters. The same work also gives an exact representation theorem: any ReLU network of width $L/n$8 and depth $L/n$9 can be represented exactly by a network of width $d/n$0 and depth $d/n$1; for $d/n$2 and $d/n$3 this conversion does not increase the number of parameters.

The constructive mechanism is highly nonstandard. A wide layer is simulated by a sequence of narrow subnetworks that encode activations and weights into a single scalar, manipulate that encoding using bit-extraction primitives built from Telgarsky’s triangle function, decode neuronwise contributions, and repack layer outputs into bounded-length bit blocks. Error control proceeds via Lipschitz bounds, with the retained number of bits scaling as $d/n$4. This makes a high DWR, interpreted as large depth relative to narrow width, theoretically defensible for ReLU approximation.

These results resolve the open question from Lu et al. (2017) in the negative: wide shallow ReLU networks do not define a class that narrow networks can approximate only with exponentially large depth. A plausible implication is that, in this setting, DWR should be understood less as a heuristic architectural ratio and more as a statement about which resource—depth rather than width—controls expressivity most strongly.

3. DWR as a control parameter in NTK theory

In NTK theory beyond the classical infinite-width limit, DWR controls whether the kernel remains deterministic and effectively frozen or becomes stochastic and evolves during training. The fully connected ReLU analysis with proportional scaling takes $d/n$5, $d/n$6, and $d/n$7, with $d/n$8, and shows that NTK dispersion depends qualitatively on the ordered, edge-of-chaos, and chaotic phases (Seleznova et al., 2022).

For $d/n$9, the central slope parameter is $\lambda=L/M$0, which also equals the fixed-point slope $\lambda=L/M$1. The ordered phase is $\lambda=L/M$2, the edge of chaos is $\lambda=L/M$3, and the chaotic phase is $\lambda=L/M$4. In the infinite-depth-and-width limit, the normalized NTK dispersion

$\lambda=L/M$5

converges to $\lambda=L/M$6 in the ordered phase, but grows exponentially in DWR at the edge of chaos and in the chaotic phase. In the chaotic phase, the limit is

$\lambda=L/M$7

The same work shows that the NTK may stay constant during training only in the ordered phase. In the chaotic phase, even the first GD step forces a relative NTK change that diverges unless the learning rate decays as $\lambda=L/M$8.

The finite-depth finite-width correction theory arrives at the same structural conclusion through a different parameterization. There the DWR is $\lambda=L/M$9, which becomes $\beta=\sum_i 1/n_i$0 in the equal-width case, and the on-diagonal NTK satisfies

$\beta=\sum_i 1/n_i$1

For equal widths, the normalized second moment obeys

$\beta=\sum_i 1/n_i$2

so the standard deviation is exponential in DWR. The paper also proves that the mean of the first SGD update scales as

$\beta=\sum_i 1/n_i$3

again showing exponential sensitivity to DWR (Hanin et al., 2019).

Taken together, these results establish a sharp limitation of the classical fixed-depth infinite-width regime. When depth remains negligible relative to width, the NTK concentrates and training stays close to lazy dynamics. When DWR stays bounded away from zero, finite-width corrections accumulate across layers instead of averaging out. This suggests that DWR is not merely a geometric descriptor of architecture but a dynamical parameter for kernel stochasticity and feature learning.

4. Residual and linear networks beyond the classical Gaussian-process limit

Residual architectures change the role of DWR because skip connections alter how layerwise fluctuations accumulate. For fully connected residual networks with branch scaling $\beta=\sum_i 1/n_i$4, width and depth limits commute: the pre-activations converge to Gaussian distributions and the covariance kernel converges to the same deterministic ODE limit regardless of whether width tends to infinity first, depth tends to infinity first, or both diverge jointly with finite DWR (Hayou et al., 2023).

In that setting, for fixed $\beta=\sum_i 1/n_i$5 and a fixed neuron coordinate, $\beta=\sum_i 1/n_i$6 converges in distribution to $\beta=\sum_i 1/n_i$7 with

$\beta=\sum_i 1/n_i$8

and the convergence rate is bounded by $\beta=\sum_i 1/n_i$9. The same analysis argues that $T=d/n$00 is the only nontrivial residual scaling: if the branch is scaled by $T=d/n$01 with $T=d/n$02, covariance explodes; if $T=d/n$03, the residual contribution vanishes.

A different residual theory studies fully connected ReLU ResNets in the simultaneous infinite-depth-and-width limit with $T=d/n$04 held fixed. There the output at initialization is not Gaussian but log-Gaussian:

$T=d/n$05

where $T=d/n$06 has i.i.d. standard normal entries and $T=d/n$07 converges to a Gaussian random variable whose mean and variance depend on $T=d/n$08, the skip/residual strengths, hypoactivation, and interlayer correlations (Li et al., 2021). The same work shows that vanilla ResNets are hypoactivated at initialization: fewer than half of the ReLUs are activated on average. It also identifies positive interlayer correlations that produce exponential variance growth in the output. The proposed Balanced ResNets randomize the sign of the nonlinearity, eliminate hypoactivation and interlayer activation-pattern correlations at initialization, and retain log-Gaussian behavior with substantially reduced variance growth.

Deep linear networks exhibit yet another proportional-limit alternative to the NNGP picture. With equal hidden widths $T=d/n$09, depth $T=d/n$10, and $T=d/n$11, the case $T=d/n$12 recovers the classical Gaussian prior

$T=d/n$13

where $T=d/n$14 has i.i.d. standard normal entries. For $T=d/n$15, the limit is instead

$T=d/n$16

where $T=d/n$17 is a random lower-triangular matrix built from Brownian motions and multiple stochastic integrals. Conditional on $T=d/n$18, the prior is Gaussian; unconditionally it is a nontrivial Gaussian mixture. Under Gaussian likelihood, the posterior remains a mixture, and the unconditional posterior covariance depends on the observed labels when $T=d/n$19 (Bassetti et al., 2024).

A common simplification in deep-learning folklore is that “infinite width” automatically implies Gaussianity. The residual and deep-linear proportional-limit results show that this is true only when depth does not scale with width in a way that preserves a nonzero DWR, or when the architecture and scaling eliminate the relevant finite-width accumulations.

5. Transformer architectures: expressivity, stability, and interaction efficiency

In self-attention, DWR governs at least three distinct phenomena: depth-efficiency of expressivity, covariance stability at initialization, and fixed-budget interaction efficiency.

For expressivity, self-attention admits a width-dependent transition between depth-efficient and depth-inefficient regimes. With depth $T=d/n$20, width $T=d/n$21, and parameter count approximately $T=d/n$22 with $T=d/n$23 and $T=d/n$24 when $T=d/n$25, the separation-rank analysis identifies a threshold

$T=d/n$26

If $T=d/n$27, separation rank grows double-exponentially in depth and only polynomially in width. If $T=d/n$28, the width caps the benefit of further depth, and the bounds become essentially multiplicative in $T=d/n$29. Empirical ablations on decoder-only Transformers of depths $T=d/n$30 to $T=d/n$31 fit the crossover width by

$T=d/n$32

with $T=d/n$33 and $T=d/n$34, giving a transition-size curve

$T=d/n$35

The same fit yields projected optima such as $T=d/n$36, $T=d/n$37 for $T=d/n$38B non-embedding parameters and $T=d/n$39, $T=d/n$40 for $T=d/n$41T, leading the authors to describe GPT-3 as too deep for its size (Levine et al., 2020).

The shaped Transformer studies DWR from the standpoint of initialization stability. In the proportional limit $T=d/n$42 with finite $T=d/n$43, the neural covariance $T=d/n$44 evolves as an Euler–Maruyama discretization and converges locally to an SDE indexed by $T=d/n$45. To obtain a stable limit, the attention matrix is modified to

$T=d/n$46

with residual parameters satisfying $T=d/n$47. Centering removes the non-infinitesimal zeroth-order drift of unshaped Softmax, the identity bias keeps attention near $T=d/n$48, and $T=d/n$49 rescales both drift and diffusion, acting as a time change that reduces effective accumulated depth. Without this shaping, the paper reports exploding covariance and rapid rank collapse at large depth; with shaping, the SDE remains stable and finite models match its predictions well (Noci et al., 2023).

The neural-interaction study considers fixed parameter budgets for decoder-only Transformers and defines DWR as

$T=d/n$50

Interaction is measured through the Average Gradient Outer Product and its off-diagonal energy: AOFE and AOFE-ratio. For Tiny byte-level LLMs on WikiText-103, the budget-wise best shapes show correlations

$T=d/n$51

and the best DWRs for budgets at or above $T=d/n$52M parameters concentrate in an “interaction-efficient interval”

$T=d/n$53

The paper also reports that smaller distance to this interval tends to correlate with higher MMLU-Pro among comparable small dense LLMs, while explicitly noting that direct transfer of the numeric interval to billion-parameter systems would be premature (Sun et al., 27 May 2026).

These Transformer results point in different directions unless their assumptions are kept separate. High depth can be expressively advantageous below the self-attention threshold, but proportional depth can destabilize initialization unless attention is shaped, and under a fixed budget there may be an intermediate interaction-efficient interval rather than a monotone preference for either depth or width.

6. Random matrix products and non-neural aspect-ratio usage

Outside standard neural-network architecture theory, DWR appears as a sharp asymptotic control parameter in products of rectangular complex Ginibre matrices. For a product

$T=d/n$54

with $T=d/n$55 of size $T=d/n$56, the DWR is defined as

$T=d/n$57

This quantity reduces to $T=d/n$58 in the square case $T=d/n$59. The local soft-edge statistics of the log singular values exhibit a three-phase transition determined entirely by the asymptotic value of $T=d/n$60: if $T=d/n$61, the Airy kernel appears at the soft edge; if $T=d/n$62, the statistics are governed by the $T=d/n$63-critical kernel; if $T=d/n$64, the edge fluctuations are Gaussian and higher-order local correlations vanish (Gu, 10 Jul 2025). In this setting, DWR measures aggregate inverse width across a multiplicative chain rather than layer count divided by hidden size, but it serves the same structural role of separating distinct asymptotic regimes.

The term also appears in an entirely different sense in granular mechanics. In true 2D granular column collapse, DWR is simply the initial aspect ratio

$T=d/n$65

with $T=d/n$66 the initial height and $T=d/n$67 the initial width. Experiments with aluminum rods over the range $T=d/n$68 identify two thresholds. For $T=d/n$69, the final height is unchanged, $T=d/n$70, and an undisturbed top zone persists. For $T=d/n$71, the top surface becomes conical and the measured relations are

$T=d/n$72

together with two run-out branches,

$T=d/n$73

The same study reports that, for $T=d/n$74 mm and $T=d/n$75 mm ($T=d/n$76), replacing a hard bed by a $T=d/n$77 mm soft bed reduces the maximum run-out from approximately $T=d/n$78 mm to approximately $T=d/n$79 mm (Nguyen et al., 2015).

This broader usage makes an important editorial point. DWR is not intrinsically a neural-network term. It is a domain-dependent dimensionless ratio that acquires technical meaning only after the underlying notions of depth and width are fixed. In neural-network theory it may index expressivity, kernel variability, covariance SDEs, or budgeted model shape; in random matrix theory it controls edge universality classes; in granular flow it is the aspect ratio that governs collapse regime and run-out.