Adaptive Depth Growing in Neural Models

Updated 3 July 2026

Depth Growing is a method for incrementally increasing the effective depth of systems by appending layers or modules based on performance thresholds and data complexity.
It utilizes strategies such as AutoGrow, gradient-gated growth, and block duplication to optimize computational resources while enhancing theoretical expressivity and practical accuracy.
The approach extends beyond neural networks to include models in statistical physics, quantum circuits, and tree-structured systems, offering adaptable scaling laws and performance improvements.

Depth Growing denotes algorithmic, architectural, or statistical processes for incrementally increasing the effective depth of models, graphs, or physical systems—either during training/optimization, as a function of data or task complexity, or through explicit scaling laws. In neural networks, “depth growing” is contrasted with static, fixed-depth designs: models are dynamically extended by appending new layers, unrolling recurrent computation, constructing deeper channel hierarchies, or instantiating task-specific submodules, with the goal of increasing representation power, adaptivity, or theoretical expressivity. Analogous processes are observed in statistical physics (e.g., growing length scales), tree-structured models, and quantum circuits. Techniques for depth growing span supervised, continual, automated, and resource-constrained learning modalities, with domain-specific algorithmic realizations and theoretical underpinnings.

1. Formal Models of Depth Growing

Architecturally, depth growing is instantiated via explicit layer-wise or block-wise network expansion, layer gating with gradient-based switching, dynamic allocation of loops/recurrences, or constructive extension of computation graphs. In neural networks, several principal frameworks include:

AutoGrow: Maintains an evolving network of “sub-networks,” each incrementally extended by adding sub-modules (residual blocks, conv-BN-ReLU blocks) if validation accuracy improves past a threshold on a given training interval. Growth and stopping policies are data-driven, globally consistent across architectures and datasets, and result in near-optimal depth estimation relative to hand-designed or brute-force depth sweeps (Wen et al., 2019).
Gradient-Gated Growth: Assigns trainable gate parameters $\alpha_l$ to each potential layer, computing a “soft” active depth $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ . Backpropagation and a depth-cost regularizer drive gates to binary open/closed states, balancing the need for deeper computation against resource penalties. Additional layers are released or pruned using hard thresholds (e.g., $\tau_\text{on}$ , $\tau_\text{off}$ ) (Butkus et al., 24 May 2026).
Block/Layer Duplication in Transformers: MIDAS and LIDAS instantiate depth growing by duplicating and inserting b-layer blocks or mid-network layers at the architectural midpoint, with continued optimization. Growth is staged by a progressive allocation schedule, and duplicated blocks begin as exact parameter clones, gradually diverging during optimization (Kapl et al., 9 Dec 2025, Kapl et al., 18 Feb 2026).
Sparse/Recurrent Depth Growing: Sparse Growing Transformer (SGT) incrementally increases effective depth by introducing selective, head-level attention loopbacks, ordered deep-to-shallow by entropy-based functional maturity, inducing structural sparsity in temporal and parameter space (Chen et al., 25 Mar 2026).
Depth-Weighted Trees: In non-network domains, depth growing is studied in the context of dynamic random trees (DWT( $f$ )), where the depth profile is determined by a generative attachment function and admits regimes (bounded, slowly growing, exponential, super-exponential) of typical depth scaling (Lichev et al., 17 Feb 2026).

2. Theoretical Properties and Scaling Regimes

Depth growing is intimately connected to expressivity, resource-optimization, and scaling theory:

Model Type	Depth Scaling Law	Regime/Result
DNNs (e.g., AutoGrow)	Empirical, adaptive	Layer count adapts to task complexity and data scale
Universal Transformer	$d(n)=\Theta(\log n)$	Regular language and graph connectivity in $\mathsf{TC^1}$
DWT( $f$ ) trees	$d(T_n)\asymp\log n$ or $\gamma n$	Scaling law set by $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 0; logarithmic for bounded/slower $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 1
Quantum circuits	Fourier degree $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 2 (exp)	Exponential expressivity by sequential/parallel encoding

Expressivity Transitions: Transformers with fixed depth cannot express tasks outside $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 3; growing the depth as $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 4 in sequence length suffices for regular language recognition and reachability (Merrill et al., 5 Mar 2025).
Resource Trade-Offs: Breadth, depth, and time (recurrent steps) are substitute axes for representational capacity, but, under fixed resource cost, depth is often the preferred channel for hierarchically compositional tasks (Butkus et al., 24 May 2026).
Optimal Allocation: Automated or curvature-based growth (e.g., split-matrix Hessian traces) allocates incremental depth heterogeneously, focusing capacity on “saddle” regions for maximal descent, yielding increased parameter efficiency and smoother optimization landscapes (T et al., 2024).

3. Algorithmic Implementations

Algorithmic recipes for depth growing combine architectural manipulations, online performance metrics, and parameter-initialization schemes.

AutoGrow Loop:

Start with the shallowest possible seed network.
Iteratively train for fixed intervals; at each, check if an improvement threshold $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 5 in validation accuracy is met.
If so, extend a candidate sub-network/module by one layer; otherwise finalize module.
After all modules finalize, fine-tune the full architecture (Wen et al., 2019).

Gradient-Based Depth Allocation:

Assign continuous gates to layers.
Optimize total loss with explicit resource regularization: $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 6.
Gates open or close according to gradients; hard pruning converts soft gates to fixed structure as training progresses (Butkus et al., 24 May 2026).

Progressive Block Duplication:

Initialize with a small architecture (e.g., block size $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 7).
Allocate budgeted training steps to each depth stage via a schedule (e.g., PROP-1).
At each stage, duplicate and insert a parameter block/layer at the network midpoint.
Continue training; with each stage, actualize $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 8.
Finalize with a cooldown period optimizing at maximal depth (Kapl et al., 18 Feb 2026, Kapl et al., 9 Dec 2025).

Error-Driven Growth for PDE Solvers:

Train a randomized one-layer RNN on the PDE with frequency-matched initialization.
Use collocation residuals as error indicators.
If residual does not decline below tolerance, identify new error-localized neurons or add layers, with parameters set constructively.
Repeat until error is controlled, applying domain splitting for discontinuities as needed (Dang et al., 2024).

4. Empirical Evidence and Performance Analysis

Depth growing shows quantifiable gains across modalities:

Deep Vision Networks: AutoGrow consistently matches or marginally exceeds human-chosen depth-architecture trade-offs on datasets from MNIST to ImageNet, achieving $d(\alpha)=\sum_{l=1}^{L_\text{max}} \sigma(\alpha_l)$ 90.1% deviation from oracle accuracies, and efficiently identifies “elbow” points for accuracy-vs-computation (Wen et al., 2019). GUNN achieves superior convergence and expressivity at zero additional parameter/FLOPs cost by reordering channel computation (Qiao et al., 2017).
Transformers: Gradually depth-grown models (MIDAS/LIDAS) overcome the “curse of depth”—i.e., the under-utilization of latter layers seen in classical pre-layernorm Transformers. Empirically, such models exhibit higher “depth scores,” persistently increasing early-exit performance up to the last layers, and form discrete, permutable computational blocks with cyclical roles (Kapl et al., 9 Dec 2025, Kapl et al., 18 Feb 2026).
Sparse Growing Transformer: SGT achieves higher reasoning accuracy (0.6% absolute) over static looping at only 1–3% FLOPs increase, compared to 16–20% for traditional block-level recurrence. Gains result from preferential, non-uniform attention head recurrence, deep-to-shallow scheduling, and progressive head-entropy tracking (Chen et al., 25 Mar 2026).
PDE Solvers: AG-RNN’s layer growth reduces $\tau_\text{on}$ 0 errors by 2–3 orders of magnitude beyond width-only expansion, with layer parameters determined by local error gradients instead of backpropagation. This enables sharply improved resolution of boundary layers and steep gradients (Dang et al., 2024).
Theoretical Scaling: Empirical results for regular-language tasks confirm the prediction that the number of layers required for 95% accuracy scales linearly with $\tau_\text{on}$ 1: for sequential tasks, $\tau_\text{on}$ 2 matches practical performance (Merrill et al., 5 Mar 2025).

Depth growing is relevant beyond conventional DNNs:

Statistical Physics: The shear penetration depth $\tau_\text{on}$ 3 in supercooled liquids (NiZr $\tau_\text{on}$ 4) quantifies the scale over which perturbations travel, with $\tau_\text{on}$ 5 growing rapidly as glass transition temperature is approached, consistent with Ising-critical exponents (e.g., $\tau_\text{on}$ 6), linking depth growing phenomena in optimization and learning to emergent physical length scales (Weingartner et al., 2015).
Quantum Circuits: Exponential expressivity in variational quantum circuits is achieved by exponentially growing the effective degree of Fourier components with depth, even at fixed or minimal qubit number. Sequential or parallel encoding scales the accessible frequency profile as $\tau_\text{on}$ 7 in $\tau_\text{on}$ 8 layers or qubits, significantly enhancing regression fidelity in quantum regression problems and avoiding “barren plateaus” (Kordzanganeh et al., 2022).
Random Trees: In DWT( $\tau_\text{on}$ 9), the maximum depth grows logarithmically for bounded/ $\tau_\text{off}$ 0, linearly for exponential $\tau_\text{off}$ 1, with sharp transitions at super-exponential regimes. This provides an analytic parallel to depth growth in neural computation (Lichev et al., 17 Feb 2026).

6. Comparative and Mechanistic Perspectives

Depth growing is mechanistically and statistically distinct from uniform scaling (width or input expansion):

Iterative Computation and Looping: Depth growing procedures (making copies of a block of layers) and looped architectures (universal transformers with weight tying) both instantiate an implicit iterative computation, as demonstrated by recurrent, block-wise depth-wise signatures, increased late-layer importance, and cyclic sublayer contributions. While depth growing produces untied parameters that diverge over training, both approaches compose an equivalent functional structure (Kapl et al., 18 Feb 2026).
Heterogeneous Non-Uniform Growth: Data-driven and curvature-based strategies selectively grow depth where optimization progress is hardest, regularizing capacity while speeding escape from saddle points. Unlike uniform scaling, this results in smoother loss landscapes and empirically reduces overfitting, especially on small-to-medium datasets (T et al., 2024).
Task and Data Dependence: Optimal depth (and growth schedule) depends on both task complexity and data regime. Adaptive approaches can discover shallower or deeper structures as a function of subsampled training data, leveraging robust policies that generalize across architectures (Wen et al., 2019).
Continual and Life-Long Learning: Dynamic layer addition is effective for continual learning when new tasks arise, as in tree-structured expansion via similarity-based task insertion and forward transfer of knowledge. These methods mitigate catastrophic forgetting without replay buffers and are compatible with standard computer vision backbones (Kozal et al., 2022).

7. Practical Guidelines and Implementation Considerations

Always begin with an under-parameterized or shallow configuration and grow depth according to performance-driven or regularization-tuned policies, rather than overfitting an over-deep initial model.
For vision and Transformer architectures, use growth intervals and thresholds for expansion, constant-large learning rates during growth, and apply standard staircase decay in final fine-tuning (Wen et al., 2019, Kapl et al., 9 Dec 2025).
During progressive depth growth, freeze earlier parameters, initialize new layers/blocks as deep copies or by constructive rules (for randomized nets or PDEs), and interleave training with structural modifications.
Utilize task/complexity-driven regularizers for depth, breadth, and time, and set explicit cost-function weights (e.g., $\tau_\text{off}$ 2) to explore trade-offs (Butkus et al., 24 May 2026).
Empirically, for sequence/graph reasoning on length $\tau_\text{off}$ 3, choose depth $\tau_\text{off}$ 4. For most practical NLP and vision tasks, data-driven policies produce depth allocations that align with theoretical and resource constraints (Merrill et al., 5 Mar 2025).
For sparsity and computational efficiency, prefer localized, sparse-depth recurrence (as in SGT) over global static looping, and enforce deep-to-shallow growth ordering for stability (Chen et al., 25 Mar 2026).

Depth growing encompasses a diverse set of adaptive, theory-motivated, and data-driven mechanisms for increasing network depth, applicable across neural models, structured statistical systems, and even quantum information architectures. By matching model complexity to task difficulty and data availability, depth growing offers both empirical and theoretical improvements in learning capacity, expressivity, and efficiency, revealing a unifying principle in the design and training of modern computational systems.