Per-Layer Predictive Threshold in Deep Models

• Per-layer predictive threshold is a layer-specific signal that marks when a network layer shifts from being redundant to becoming predictively decisive.
• It encompasses various interpretations including trainable pruning thresholds, batch-size control signals, and geometric phase boundaries, each serving as a predictive marker.
• Practical applications range from optimizing training dynamics and network compression to informing architectural choices in transformer models.

Per-layer predictive threshold denotes a layer-specific criterion that predicts a qualitative change in model behavior at a particular depth or module. In recent work, the expression is sometimes explicit and sometimes only implicit: it may refer to a trainable pruning threshold, a post-training cutoff derived from layer statistics, a control signal for batch-size scheduling, a depth at which intermediate predictions become stable and causally effective, or a feature-norm value that anticipates neural collapse (Kerce, 8 Mar 2026, Gray et al., 2024, Azarian et al., 2020, Rupa, 31 Mar 2026). This suggests a broad but coherent notion: a layer-indexed signal separates a regime in which a layer is diffuse, redundant, or safely compressible from a regime in which it becomes predictively decisive, intervention-sensitive, or resource-critical.

1. Scope and competing definitions

The term is not standardized across the literature. In some papers it is an explicit scalar attached to each layer. Learned Threshold Pruning assigns each prunable layer a trainable threshold $\tau_l$ and optimizes it jointly with the weights, so the threshold itself becomes a learned parameter of the model (Azarian et al., 2020). Channel Pruning via Optimal Thresholding instead computes a layer-dependent threshold $\gamma_{\mathrm{th},l}$ from the distribution of batch-normalization scaling factors in that layer, using a cumulative squared-sum rule with $\delta = 10^{-3}$ (Ye et al., 2020).

In other papers the threshold is not a literal cutoff but a predictive signal. In transformer training, Normalization Layer Per-Example Gradients shows that a LayerNorm-only Gradient Noise Scale can predict total model Gradient Noise Scale, yielding a layer-derived control quantity for batch-size scheduling (Gray et al., 2024). Engineering Verifiable Modularity in Transformers via Per-Layer Supervision does not name a per-layer predictive threshold explicitly, but its details define one operationally as the point where a layer’s outputs stabilize, matter causally under ablation or scaling, and exhibit task-specific modularity (Kerce, 8 Mar 2026).

A common misconception is that all such thresholds are hard binary boundaries. The literature is more heterogeneous. Some are explicit discontinuities, such as $\tau_l$ or $\gamma_{\mathrm{th},l}$; others are smooth regime separators, such as a rise in ablation variance, a crossing of critical batch size, or a phase boundary in representational geometry. The most stable synthesis is therefore functional rather than terminological: a per-layer predictive threshold is a layer-indexed quantity used to predict when local state, intervention, or compression crosses from negligible effect to structured effect.

2. Architectural thresholds for modularity in transformers

In "Engineering Verifiable Modularity in Transformers via Per-Layer Supervision" (Kerce, 8 Mar 2026), the threshold notion arises from architectural control of when layers become independently predictive. The model computes logits at every layer,

$$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$

and optimizes

$$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$

with $\lambda_{L-1}=1$ and $\lambda_\ell = 0.1\cdot \frac{\ell+1}{L}$ for $\ell=0,\dots,L-2$. The same work combines this objective with dual-stream residual decomposition,

$\gamma_{\mathrm{th},l}$0

and, in CASCADE mode, freezes the token stream so that context accumulates only in $\gamma_{\mathrm{th},l}$1. Gated attention further regularizes heads toward discrete activation patterns. The claimed effect is architectural: without per-layer supervision, ablation damage concentrates near zero with low variance; with per-layer supervision, effects spread widely, exposing circuits that can be causally manipulated (Kerce, 8 Mar 2026).

The paper operationalizes layer predictiveness through stability depth,

$\gamma_{\mathrm{th},l}$2

the earliest layer at which a token’s prediction matches the final prediction and remains stable. Under per-layer supervision, 26% of tokens converge at layers $\gamma_{\mathrm{th},l}$3–$\gamma_{\mathrm{th},l}$4, compared with 5% in the control, while the control places more mass in later layers. Ablation effects become 5 to 23 times larger than architecturally identical controls trained with standard objectives, and control leverage on targeted behaviors becomes 4 times greater; on Winograd, the standard deviation of ablation effects increases from $\gamma_{\mathrm{th},l}$5 to $\gamma_{\mathrm{th},l}$6 (Kerce, 8 Mar 2026).

The associated threshold is therefore causal rather than purely statistical. A layer or head has crossed it when its local prediction has stabilized, when ablating or scaling it changes behavior in a smooth and predictable way, and when those changes are task-specific rather than distributed across redundant pathways. In this framing, the Hydra effect is the below-threshold regime: components can be correlated with behavior, but not predictively or controllably responsible for it (Kerce, 8 Mar 2026).

3. Thresholds as local optimization and credit signals

In large-scale training, a per-layer predictive threshold can be a control quantity rather than a structural cutoff. "Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers" defines the Gradient Noise Scale by

$\gamma_{\mathrm{th},l}$7

and shows that tracking only normalization-layer per-example gradients is sufficient to predict the total model GNS (Gray et al., 2024). The method computes per-example norms during LayerNorm backward with zero throughput overhead, and the resulting LayerNorm GNS correlates strongly with total GNS, with a regression slope of approximately $\gamma_{\mathrm{th},l}$8. The paper uses this signal to guide a practical batch size schedule on a 111M Chinchilla-optimal LLM, reducing training time by 18% (Gray et al., 2024).

In that setting, the threshold is the comparison between current effective batch size and a layer-derived critical batch size. The paper’s details explicitly interpret $\gamma_{\mathrm{th},l}$9 as a threshold or control signal: if the effective batch is well below it, increasing batch size can improve optimization efficiency; if it is well above it, further increases are not warranted. This is a predictive threshold because a local statistic from one layer type forecasts global training behavior (Gray et al., 2024).

A different local interpretation appears in "Augmented Lagrangian Predictive Coding" (Seely et al., 29 May 2026). Each layer maintains a constraint error

$\delta = 10^{-3}$0

and a Lagrange multiplier updated by

$\delta = 10^{-3}$1

The effective local credit signal becomes

$\delta = 10^{-3}$2

The paper explicitly describes this composite quantity as the per-layer predictive threshold or credit signal: it drives both activation updates and weight updates, and, at equilibrium in linear predictive-coding networks, the multipliers equal the negative backpropagation adjoints (Seely et al., 29 May 2026). The contrast with standard predictive coding is depth propagation: ordinary predictive coding is diffusive, whereas PC-ALM exhibits ballistic credit propagation with group velocity $\delta = 10^{-3}$3. Here threshold means the local magnitude at which a layer’s mismatch is no longer transient noise but accumulated, actionable credit.

4. Layer-specific pruning and sparsity boundaries

Pruning literature contains the most literal uses of per-layer predictive threshold. "Learned Threshold Pruning" assigns one trainable scalar threshold $\delta = 10^{-3}$4 to every prunable layer and replaces hard masking,

$\delta = 10^{-3}$5

with the differentiable soft-pruning operator

$\delta = 10^{-3}$6

Its objective,

$\delta = 10^{-3}$7

uses a soft $\delta = 10^{-3}$8 penalty so that thresholds and weights co-evolve under gradient descent (Azarian et al., 2020). The layer threshold is predictive in a direct sense: it estimates which magnitudes in that layer are worth retaining. The method reports, for example, $\delta = 10^{-3}$9 compression on ResNet-50 in 30 total epochs and $\tau_l$0 compression on AlexNet with no Top-5 drop from the $\tau_l$1 baseline (Azarian et al., 2020).

"Channel Pruning via Optimal Thresholding" computes a layer-dependent threshold from batch-normalization scales rather than learning it end to end (Ye et al., 2020). After training with an $\tau_l$2 penalty on BN scaling factors, each layer’s threshold $\tau_l$3 is chosen so that the cumulative squared sum of scales below the threshold accounts for a fraction $\tau_l$4 of the total squared sum. The method is explicitly motivated by the failure of global thresholds to account for inter-layer variation. Its DenseNet-121 result on CIFAR-100—$\tau_l$5 accuracy with only $\tau_l$6 FLOPs and $\tau_l$7M parameters—illustrates the practical role of a layer-specific boundary between negligible and important channels (Ye et al., 2020).

A more recent reformulation replaces local importance by downstream compensation. "Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective" defines the relative drift

$\tau_l$8

and the absorption coefficient

$\tau_l$9

If $\gamma_{\mathrm{th},l}$0, downstream layers absorb perturbations from layer $\gamma_{\mathrm{th},l}$1; if $\gamma_{\mathrm{th},l}$2, they amplify them (Jing et al., 13 Jun 2026). The paper reports that early layers usually amplify, middle and late layers often absorb, and that absorption is a large-perturbation phenomenon rather than a small-perturbation one. Using $\gamma_{\mathrm{th},l}$3 as an absorption-aware correction on OWL and AlphaPruning reduces perplexity by 7.13% and boosts zero-shot accuracy by 1.02% across multiple model families at 70% sparsity (Jing et al., 13 Jun 2026). In this formulation, the threshold is not a mask value but a layer-specific tolerance to pruning-scale perturbation.

5. Runtime routing, quantization, and cache-compression thresholds

Inference-time compression extends the threshold idea from weights to runtime state. "PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-LLMs" computes a layer-level Vision Attention Score,

$\gamma_{\mathrm{th},l}$4

then classifies each decoder layer as vision-attentive, vision-balanced, or vision-indifferent by thresholds $\gamma_{\mathrm{th},l}$5 and $\gamma_{\mathrm{th},l}$6, and assigns a retention rate

$\gamma_{\mathrm{th},l}$7

Head-level top-$\gamma_{\mathrm{th},l}$8 pruning then keeps only the most attended vision tokens in each head’s KV cache (Meng et al., 20 Feb 2025). The method is training-free and reports 18% faster decoding, more than 50% KV-cache reduction, and only a 0.46% average performance drop, together with notable improvements in multi-image tasks (Meng et al., 20 Feb 2025). The per-layer threshold is the retention policy itself: a layer crosses from aggressive to conservative token preservation when its measured vision attention crosses $\gamma_{\mathrm{th},l}$9 or $$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$0.

"MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression" generalizes this to a constrained routing problem over eviction and quantization (Sun et al., 20 Apr 2026). For layer $$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$1 and configuration $$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$2, it defines a sensitivity

$$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$3

then minimizes predicted quality loss

$$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$4

subject to a global memory budget $$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$5 (Sun et al., 20 Apr 2026). The final per-layer threshold is the chosen $$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$6 tuple. On a 4-task subset of LongBench-v1 with 16k inputs, the hetero variant matches the uncompressed 1.9 GB baseline at 14x compression (136 MB), while the other compressed baselines tested stay under 8/100 at comparable or smaller memory; on AIME it improves by +6 to +27 points over the strongest per-layer-quantization baseline across eight configurations (Sun et al., 20 Apr 2026). The same paper also reports null results on MATH-500 and TREC because the solver selects $$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$7 on most layers, showing that the relevant threshold is budget- and input-length-dependent.

Post-training weight quantization yields another family of thresholds. "A Hardware-Aware, Per-Layer Methodology for Post-Training Quantization of LLMs" searches per-layer codebook pairs, layer-specific scale shifts, and per-layer promotion profiles scored by activation-weighted cosine,

$$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$8

then uses a multiple-choice knapsack allocator to decide which layers receive higher-precision formats or corrections (Killian, 14 May 2026). Its outlier correction is explicitly thresholded on standardized magnitude, with default quantile $$\mathbf{z}^{(\ell)}=\mathrm{lm\_head}(\mathrm{LayerNorm}(\mathbf{x}^{(\ell)})),$$9 giving $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$0, and its sparse-residual correction retains only the top $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$1 activation-weighted residuals (Killian, 14 May 2026). The recommended FP6 operating point, E2M3sUE4M4 at 6.5 bpw, achieves lower weight reconstruction error than the conventional per-layer-POT FP8 baseline E4M3 at 8.0 bpw, at 1.5 bpw lower storage cost (Killian, 14 May 2026). In "Ternary MobileNets via Per-Layer Hybrid Filter Banks," the layer-level knobs are the fraction $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$2 of full-precision filters and the Strassen hidden width $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$3; the selected hybrid design delivers 27.98% energy savings and a 51.07% reduction in model size while achieving comparable accuracy and no degradation in throughput on specialized hardware (Gope et al., 2019). In both cases, threshold means a layer-specific resource boundary that predicts whether low precision is sufficient.

6. Representational phase transitions and theoretical thresholds

Some of the strongest threshold results concern representation dynamics rather than compression. "Neural Collapse Dynamics: Depth, Activation, Regularisation, and Feature Norm Threshold" identifies a model-dataset-specific critical mean feature norm $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$4 in the penultimate layer,

$$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$5

where $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$6 is the first epoch at which NC1 falls below the collapse criterion (Rupa, 31 Mar 2026). The paper reports that $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$7 concentrates tightly within each model-dataset pair with CV $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$8, that crossing $$\mathcal{L}=\sum_{\ell=0}^{L-1}\lambda_\ell \mathcal{L}_\ell,$$9 below $\lambda_{L-1}=1$0 precedes neural-collapse onset with a mean lead time of 62 epochs and MAE 24 epochs, and that direct perturbations of feature scale converge back to the same value with $\lambda_{L-1}=1$1 (Rupa, 31 Mar 2026). Its grid also shows the strongest architecture effect: ResNet-20 on MNIST yields $\lambda_{L-1}=1$2, a +458% architecture effect versus only +68% on CIFAR-10, while width changes $\lambda_{L-1}=1$3 by at most 13% (Rupa, 31 Mar 2026). Here the threshold is a genuine attractor of the gradient flow.

A geometric depth-based version appears in "A Geometric Perspective on Next-Token Prediction in LLMs: Three Emerging Phases" (Lombardo et al., 9 May 2026). The paper constructs a predictive readout subspace $\lambda_{L-1}=1$4 from the dominant singular subspace of a representation lens and tracks its trajectory on the Grassmann manifold via layer-to-layer readout subspace similarity. The resulting similarity profile shows a rise, near-plateau, and descent across eight models from Qwen2.5 and OLMo2. The authors identify three phases—Seeding Multiplexing, Hoisting Overriding, and Focal Convergence—and report that Phases 1 and 3 grow slowly with model depth while Phase 2 expands linearly (Lombardo et al., 9 May 2026). The paper does not define a scalar threshold per layer, but it explicitly treats the phase boundaries as depths at which predictive geometry crosses qualitative boundaries. In that sense, the boundaries are predictive thresholds on where candidate seeding ends, disambiguation dominates, and final readout alignment begins.

A more theoretical use of threshold occurs in "Recovering the Lowest Layer of Deep Networks with High Threshold Activations" (Goel et al., 2019). There the threshold is an activation bias $\lambda_{L-1}=1$5 in the lowest layer, for example in high-threshold ReLU $\lambda_{L-1}=1$6, under Gaussian input and a polynomial upper network. The paper shows that if $\lambda_{L-1}=1$7 for sufficiently large constant $\lambda_{L-1}=1$8, linear coefficients are at least $\lambda_{L-1}=1$9, and all coefficients are $\lambda_\ell = 0.1\cdot \frac{\ell+1}{L}$0, the lowest-layer weights can be recovered to within $\lambda_\ell = 0.1\cdot \frac{\ell+1}{L}$1 in polynomial time (Goel et al., 2019). High threshold suppresses higher-order interactions and effectively linearizes the lower layer’s contribution. This is a threshold in the original, literal sense, but it already exhibits the later recurring theme: a layer-specific scale parameter can mark the onset of a qualitatively different predictive regime.

Taken together, these strands show that per-layer predictive threshold is best understood as a family of layer-indexed regime markers rather than a single formal object. In some cases the threshold is a learned scalar or a post-training cutoff; in others it is a critical norm, a routing boundary, a local credit signal, or a phase breakpoint in predictive geometry. The unifying feature is methodological: each threshold is meant to predict when a particular layer ceases to be merely latent structure and becomes an actionable determinant of behavior, optimization, or compression.