Layer Runs in Neural Networks

• Layer Runs are a suite of layer-wise operational techniques in deep networks that dynamically select, skip, or reorder layers based on input complexity.
• They enable adaptive inference by balancing computational cost and accuracy through runtime decisions on layer execution and iterative processing.
• Research on Layer Runs spans adaptive scheduling, robustness probing, and functional tracing, linking methods in deep learning with insights from concurrency theory.

Searching arXiv for papers related to “layer runs” across neural networks, transformers, Petri-net runs, and layered image decomposition. “Layer runs” is used in several adjacent but nonidentical ways in current technical literature. In deep-network research, it most often denotes layer-wise execution patterns—selecting, skipping, iterating, reordering, parallelizing, or resetting layers—rather than treating a model as a fixed-depth pipeline for every input. The available literature suggests that the phrase is best understood as a family of layer-wise operational and analytic techniques, spanning adaptive inference, pipeline scheduling, robustness probing, and representation tracing. A distinct formal use of “runs” exists in concurrency theory, where runs are partially ordered event occurrences rather than neural-network layer executions (Mathur et al., 2023, Sun et al., 2024, Zhang et al., 2019, Wu et al., 22 Apr 2026, Fettke et al., 6 Feb 2026).

1. Scope and nomenclature

The literature reviewed here uses “layer runs” across several research programs that share a layer-wise perspective but differ in objective, formalism, and granularity.

Context	Layer-wise operation	Representative paper
Adaptive inference	Select, skip, or iterate layers per input	DynaLay (Mathur et al., 2023)
Frozen-model execution	Skip, reorder, randomize, repeat, or parallelize layers	“Transformer Layers as Painters” (Sun et al., 2024)
Training-system scheduling	Split each layer into forward and backward sub-operations for pipelining	LayerPipe (Unnikrishnan et al., 2021)
Functional probing	Re-initialize or re-randomize one layer at a time	“Are All Layers Created Equal?” (Zhang et al., 2019)
Layer-wise tracing	Localize task-effective and vulnerable layers in LLMs	LayerTracer (Wu et al., 22 Apr 2026)

Two neighboring usages are important for disambiguation. First, in Petri-net theory a run is an acyclic net module built from steps, and the central theorem states that for nets $M$ and $N$, $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$; this is a theory of partially ordered executions, not of neural-network depth scheduling (Fettke et al., 6 Feb 2026). Second, in layered image synthesis and decomposition, “layers” denote foreground, background, and RGBA scene components rather than sequential network blocks; methods such as LayeringDiff and RevealLayer therefore concern layer recovery and editability, not layer execution policies (Kang et al., 2 Jan 2025, Wang et al., 12 May 2026).

2. Adaptive execution and iterative layer selection

A direct formulation of layer runs in deep networks appears in DynaLay, which replaces fixed-depth inference with an agent-guided execution policy. The architecture consists of a main model with layers $L=\{l_1,\dots,l_n\}$ and an auxiliary AgentNet that observes activations and outputs a distribution over layers. The paper gives both a per-layer form,

$$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$

and a combined form,

$$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$

The selected layer may be an FPI layer, so a layer run is potentially iterative rather than single-pass. The fixed-point dynamics are written as

$$z = f(z, x), \qquad z^{(k+1)} = f(z^{(k)}, x), \qquad || z^{(k+1)} - z^{(k)} || < \epsilon,$$

with the appendix inference criterion

$$\frac{\lVert f_0 - z \rVert}{\lVert f_0 \rVert} < \text{tol}, \qquad \text{tol}=10^{-2}.$$

This makes run length input-dependent: easy inputs may terminate quickly, whereas difficult inputs may trigger repeated layer application until convergence (Mathur et al., 2023).

The control objective explicitly trades accuracy against computation. DynaLay defines

$$R = \alpha \times \text{Classification Accuracy} - \beta \times \text{Computational Cost} \times \text{Consecutive Layer Count},$$

and trains with

$$\mathcal{L} = \text{Cross-Entropy Loss} - \gamma \times R.$$

Training is joint over the main model and AgentNet, using Adam, learning rate $N$0, batch size 64, 100 epochs, and tuned hyperparameters $N$1, $N$2, $N$3. The appendix states that the agent uses adaptive softmax, highest-probability layer selection, and epsilon-greedy exploration during training. Reported selection frequencies indicate that “No Layer” was selected most often, Layer1 FPI about $N$4 of the time, Layer2 FPI about $N$5, and Layer3 less often, which the paper interprets as evidence that computational effort is input-dependent. The reported test accuracy is $N$6 on CIFAR-10 and $N$7 on CIFAR-100, with backward pass times of 87 seconds and 90 seconds respectively (Mathur et al., 2023).

In this formulation, layer runs are neither merely pruning nor static architecture search. They are runtime decisions over whether to execute a layer, which layer to execute, and how long to iterate it.

3. Reordering, skipping, parallelizing, and pipelining layers

A complementary line of work studies nonstandard layer runs in frozen pretrained transformers. “Transformer Layers as Painters” evaluates skipping, switching, reversing, randomizing, repeating, and parallelizing layers in Llama2 and BERT. The paper concludes that the first few layers and final layers are special, while middle layers show a surprising degree of uniformity. It states that “with the important exception of the first and last few layers, Llama2-7B's layers are fairly robust to skipping or even switching layers,” and that hidden-state cosine similarity suggests “three distinct representation spaces for the beginning',middle', and `ending' layers.” At the same time, the middle layers are not functionally redundant: replacing many middle layers with repeated copies of the center layer is described as “the most catastrophic” variant, significantly worse than skipping. Reverse order, random order, and parallel execution all exhibit graceful degradation on many tasks, and “Looped Parallel 3X significantly improves on a single iteration.” The paper further reports that step-by-step reasoning tasks are more sensitive to layer order than semantic tasks, with GSM8K especially fragile under parallelization (Sun et al., 2024).

LayerPipe treats layer runs at the systems level. Its basic claim is that a training layer should not be regarded as a monolithic backward-pass task. Instead, for layer $N$8,

$N$9

and

$\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$0

are separated and scheduled independently. This is the paper’s intra-layer optimization. Inter-layer optimization then splits $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$1 into $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$2 and $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$3 across adjacent layers so that consecutive stages have balanced compute time. The resulting notion of a layer run is a finely scheduled pipeline unit consisting of forward computation, weight-gradient computation $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$4, activation-gradient computation $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$5, and, when necessary, a cross-layer split of $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$6. Quantitatively, LayerPipe reports an average speedup of $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$7 and upwards of $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$8 with 7 to 9 processors relative to PipeDream; for VGG16 it reports improvements of $\sruns(M \compose N)=\sruns(M)\compose \sruns(N)$9 at 2 processors, $L=\{l_1,\dots,l_n\}$0 at 3, $L=\{l_1,\dots,l_n\}$1 at 4, $L=\{l_1,\dots,l_n\}$2 at 5, $L=\{l_1,\dots,l_n\}$3 at 6, $L=\{l_1,\dots,l_n\}$4 at 7, $L=\{l_1,\dots,l_n\}$5 at 8, and $L=\{l_1,\dots,l_n\}$6 at 9, with an average $L=\{l_1,\dots,l_n\}$7 improvement and maximum additional communication overhead of $L=\{l_1,\dots,l_n\}$8 KB. In the 4-layer sample network, the paper reports a $L=\{l_1,\dots,l_n\}$9 throughput improvement, $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$0 cycles versus $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$1, with $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$2 KB additional overhead (Unnikrishnan et al., 2021).

Taken together, these papers separate several meanings of layer runs: semantic execution order, representational compatibility under nonstandard ordering, and low-level systems scheduling of layer sub-operations.

4. Reset probes, robustness, and functional criticality

Another major usage studies a trained network by rerunning it after surgically modifying one layer at a time. “Are All Layers Created Equal?” defines a layer-wise reset probe in which a trained model $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$3 is modified by replacing only one layer with its initialization,

$$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$4

or by replacing it with a fresh random draw $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$5. No retraining or fine-tuning is performed. Layers whose reset causes negligible performance drop are termed robust; layers whose reset destroys performance are termed critical. The paper reports strong heterogeneity. In fully connected networks and VGGs, bottom layers tend to be critical and higher layers more robust; wider models tend to have more robust upper layers; harder tasks engage more layers; ResNets distribute criticality more broadly across stages; and transformer-style models show analogous but architecture-specific patterns in which some higher layers are robust while layernorm modules and some MLP sublayers are sensitive (Zhang et al., 2019).

The paper also emphasizes that individual robustness does not imply joint robustness. A set of layers may each be robust in isolation yet fail when reset together. This makes layer runs a collective phenomenon as well as a per-layer one. The paper further argues that simple quantities such as $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$6 and $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$7 are too coarse to explain layer robustness: some layers move substantially yet remain robust, while others move little but are critical. Its broader conclusion is that deep models do not use all layers uniformly in a task-specific sense (Zhang et al., 2019).

This line of work shifts the meaning of layer runs from execution policy to functional dependency analysis. The network is rerun under layer replacement, and the resulting degradation is used to map where learned computation actually resides.

5. Tracing the onset of task execution and the location of vulnerable layers

LayerTracer extends layer-wise analysis from reset probes to architecture-agnostic tracing of hidden-state evolution in LLMs. It introduces the task particle, defined as the layer where the target token probability first rises significantly, and the vulnerable layer, defined as the layer whose perturbation maximizes Jensen–Shannon divergence in the final output distribution. If $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$8 is the probability of the target token at layer $$p_i = \text{AgentNet}(A_i), \qquad \sum_{i=1}^{n} p_i = 1, \qquad l = \text{Sample}(P),$$9, the relative increase ratio is

$$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$0

and the maximizing layer is the task particle. Vulnerable-layer analysis perturbs a layer, compares the perturbed output distribution $$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$1 with the unperturbed distribution $$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$2, and computes

$$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$3

The vulnerable layer is the one with maximum JS divergence (Wu et al., 22 Apr 2026).

The framework is described as architecture-agnostic and compatible with Transformer, GateDeltaNet, and Mamba-like models. Its two-phase pipeline first maps hidden states layer by layer to vocabulary probability distributions and identifies the task particle, then applies masking perturbation to each layer’s core module and identifies the vulnerable layer. On Qwen3 models at 0.6B, 4B, 8B, and 14B parameters evaluated on AntSynNET, the task-particle-related probability surge appears mainly in the deep half of the network, roughly beyond $$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$4. Vulnerability is scale-dependent: Qwen3-0.6B shows a pronounced U-shaped vulnerability profile with shallow and deep layers more vulnerable than mid layers, whereas 8B and 14B models have smoother, more balanced vulnerability profiles. The paper summarizes this trend with Layer-wise Relative Stability,

$$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$5

where lower LRS indicates more uniform vulnerability across layers (Wu et al., 22 Apr 2026).

This analysis reframes layer runs as trajectories of task emergence and robustness sensitivity. The relevant question is not only which layers run, but where useful task-specific computation starts to crystallize and where perturbations most strongly destabilize final predictions.

6. Structural depth reduction and neighboring concepts

Layer runs can also be reduced structurally rather than scheduled dynamically. “Layer Sparsity in Neural Networks” introduces a depth-oriented regularizer that does not prune individual weights or neurons but encourages entire hidden layers to become removable. For positively homogeneous activations, a layer whose weights are all nonnegative can be fused with the next layer. The regularizer is

$$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$6

with $$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$7. If $$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$8, every entry of $$P = \text{Adaptive Softmax} (F(a_1, a_2, \ldots, a_L)).$$9 is nonnegative, and the paper’s merging theorem shows that neighboring layers can be replaced by a single layer under coordinatewise separability and positive homogeneity. This is fundamentally different from runtime layer selection: a layer can remain dense, yet become inactive in the layer-sparsity sense because it is algebraically mergeable (Hebiri et al., 2020).

Two further notions delimit the term’s boundaries. In formal concurrency, a partially ordered run is an acyclic, non-branching Petri-net module built inductively from steps, and the paper “Compositionality of Systems and Partially Ordered Runs” proves that the set of runs of a composed net equals the composition of the runs of its components, $$z = f(z, x), \qquad z^{(k+1)} = f(z^{(k)}, x), \qquad || z^{(k+1)} - z^{(k)} || < \epsilon,$$0 (Fettke et al., 6 Feb 2026). In computer vision, layered image work uses “layers” to denote editable scene components. LayeringDiff is a “generate first, decompose later” pipeline that generates a composite image, estimates a foreground matte, and decomposes the image into foreground $$z = f(z, x), \qquad z^{(k+1)} = f(z^{(k)}, x), \qquad || z^{(k+1)} - z^{(k)} || < \epsilon,$$1, background $$z = f(z, x), \qquad z^{(k+1)} = f(z^{(k)}, x), \qquad || z^{(k+1)} - z^{(k)} || < \epsilon,$$2, and alpha $$z = f(z, x), \qquad z^{(k+1)} = f(z^{(k)}, x), \qquad || z^{(k+1)} - z^{(k)} || < \epsilon,$$3 under

$$z = f(z, x), \qquad z^{(k+1)} = f(z^{(k)}, x), \qquad || z^{(k+1)} - z^{(k)} || < \epsilon,$$4

while RevealLayer decomposes an RGB image plus user-specified bounding boxes into one background image and multiple foreground RGBA layers using a Region-Aware Attention module, an Occlusion-Guided Adapter, and a composite loss (Kang et al., 2 Jan 2025, Wang et al., 12 May 2026).

These adjacent literatures show that “layer” and “run” are both overloaded terms. In the neural-network sense surveyed above, the central problem is how computation is distributed, scheduled, perturbed, or localized across depth. The available literature suggests three orthogonal questions: which layers execute and for how long, which layers are functionally indispensable, and which layers can be removed or merged without changing the effective computation.