Dynamic Layer Saliency Probing (DLSP)

• The paper demonstrates that DLSP identifies the most informative intermediate layer by comparing criteria like KL divergence, LDR, and entropy.
• DLSP improves anomaly detection performance by selecting a single optimal layer over standard fixed or final-layer approaches in video analysis.
• The methodology inspires broader applications such as dynamic token pruning, multi-layer interpretability, and adaptive sampling in deep learning models.

Dynamic Layer Saliency Probing (DLSP) is the mechanism in HiProbe-VAD that identifies which intermediate hidden layer of a pre-trained multimodal LLM is most useful for video anomaly detection, then uses the hidden states from that layer for downstream anomaly scoring and temporal localization (Cai et al., 23 Jul 2025). In the narrower, paper-specific sense, DLSP is therefore a layer-selection procedure over internal representations. In a broader methodological sense, related work suggests a family of DLSP-like designs in which saliency is inferred from intermediate activations or hidden states and then used to select layers, decode token importance, redistribute spatial sampling density, or prune computation, rather than relying exclusively on final-layer outputs (Tao et al., 6 Apr 2025, Hou et al., 2023, Recasens et al., 2018).

1. Definition and scope

The term “Dynamic Layer Saliency Probing” is introduced explicitly in HiProbe-VAD, where “Dynamic” means that the method does not assume a manually fixed layer, “Layer” means that the unit of selection is the hidden layer $l$ of the MLLM, “Saliency” means that each layer is assigned a saliency score based on anomaly sensitivity, class separability, and information concentration, and “Probing” means that hidden states from candidate layers are extracted and externally evaluated for anomaly detection utility (Cai et al., 23 Jul 2025). The same source also clarifies an important boundary condition: despite the word “dynamic,” the actual procedure is not per-test-sample adaptive. DLSP is performed in an offline preparation stage on about $1\%$ of the training set, after which a single optimal layer $l^*$ is fixed for scorer training and inference.

This definition distinguishes DLSP from several adjacent strategies. It differs from using only the final output layer, because HiProbe-VAD argues that final layers are less informative than certain intermediate layers for anomaly detection. It also differs from averaging hidden states across layers; the paper does not propose layer averaging, but instead seeks a single best intermediate layer. A plausible implication is that DLSP is most precise when it denotes a procedure that compares candidate depths and commits to a saliency-validated probing depth, rather than any generic use of hidden states.

Related literature broadens the concept without collapsing its distinctions. In “Decoded Grad-CAM,” saliency is computed at arbitrary transformer layers and decoded back into token space, but the layer choice is treated mostly as a dataset-level empirical selection rather than a dynamic per-input routing policy (Hou et al., 2023). In SDTP, lightweight modules are inserted at selected transformer depths to estimate token importance from hidden states and progressively prune low-saliency tokens, which is dynamic at inference time and actionable rather than purely diagnostic (Tao et al., 6 Apr 2025). By contrast, EML-NET uses a fixed, hand-selected multi-layer aggregation scheme, making it highly relevant to DLSP but not itself a dynamic layer selection method (Jia et al., 2018).

2. Historical precursors and methodological lineages

Several earlier lines of work establish the technical premises from which DLSP emerges. “Learning to Zoom” introduces a saliency-based distortion layer that predicts a task-specific saliency map from a uniformly downsampled image and converts that map into a differentiable sampling field over the original high-resolution input (Recasens et al., 2018). Its saliency is learned, task-conditioned, and input-conditional, and the resulting warped intermediate image redistributes a fixed output pixel budget toward task-relevant regions. This is not hidden-layer probing, but it is an early example of saliency-conditioned adaptive computation.

EML-NET establishes a different precursor. It is an encoder-decoder saliency predictor whose encoder can contain more than one CNN model in parallel, specifically NasNet-Large pretrained on ImageNet and DenseNet-161 pretrained on PLACE365 in the main instantiation (Jia et al., 2018). The method extracts a fixed set of intermediate layers, compresses each selected layer to one feature map by applying conv1@1 and ReLU, aligns them to the largest intermediate resolution, concatenates them, and linearly fuses them in a lightweight decoder. The architecture is therefore a strong demonstration that saliency benefits from probing semantically different layers and different pretrained backbones, while leaving open the stronger DLSP question of whether layer selection or weighting should be input-dependent.

A more explicitly diagnostic precursor appears in “Understanding and Visualizing Deep Visual Saliency Models,” which analyzes intermediate activation maps in VGG-16-based saliency networks and shows that categories such as person head, animal head, text, symbol, and vehicle are already encoded in pretrained object-recognition layers (He et al., 2019). The paper further shows that fine-tuning for saliency prediction changes inner representations unevenly, increasing sensitivity to some categories such as heads more than others such as text, and that internal layers can respond strongly to synthetic pop-out structure even when the final saliency output remains poor. This establishes a core DLSP principle: output behavior alone can conceal informative intermediate saliency structure.

In language transformers, “Decoding Layer Saliency in Language Transformers” provides a direct mechanism for probing arbitrary encoder layers and restoring token interpretability through the pretrained MLM head (Hou et al., 2023). SDTP extends this general direction from explanation to systems optimization by training saliency-prediction heads on hidden states and using them to hierarchically prune redundant tokens across selected transformer stages (Tao et al., 6 Apr 2025). Taken together, these papers form a continuum from static multi-layer inspection, through decoded intermediate attribution, to dynamic saliency-conditioned inference control.

3. Core mechanisms and mathematical formulation

In HiProbe-VAD, DLSP operates by extracting hidden states $\mathbf{H}^{(v,l)}$ from each layer $l$ for videos $v$ in a small labeled subset, partitioning them into normal and anomalous sets, and computing three layer-wise criteria: KL divergence for anomaly sensitivity, Local Discriminant Ratio for class separability, and entropy for information concentration (Cai et al., 23 Jul 2025). For feature dimension $d$ at layer $l$, the paper models normal and anomalous hidden states as Gaussians and defines

$$D_{\text{KL}}^{(d)}(l)=\frac{1}{2}\left[\log\left(\frac{(\sigma_{l,d}^A)^2}{(\sigma_{l,d}^N)^2}\right)+\frac{(\sigma_{l,d}^N)^2+(\mu_{l,d}^N-\mu_{l,d}^A)^2}{(\sigma_{l,d}^A)^2}-1\right],$$

with layer average

$$D_{\text{KL}}(l)=\frac{1}{D}\sum_{d=1}^{D}D_{\text{KL}}^{(d)}(l).$$

It also defines

$1\%$0

and entropy

$1\%$1

After Z-score normalization across layers,

$1\%$2

the layer saliency score is

$1\%$3

and the selected layer is

$1\%$4

The chosen representation is then fed to a logistic-regression anomaly scorer,

$1\%$5

with BCE loss.

A second major formulation appears in decoded layer saliency for language transformers. There, for class $1\%$6, the pre-softmax class score $1\%$7 is differentiated with respect to the hidden representation $1\%$8 at layer $1\%$9, producing feature-level scores

$l^*$0

The paper then uses the pretrained MLM head as a decoder from hidden states into token-space influence weights and defines the decoded token saliency as

$l^*$1

where $l^*$2 restricts the decoded vocabulary distribution to the tokens actually appearing in the input (Hou et al., 2023). This formulation is notable because it makes layer-specific transformer saliency interpretable without retraining.

SDTP formalizes a more operational saliency predictor. At selected pruning layers, a two-layer MLP reads the current token hidden states and outputs token scores

$l^*$3

while the saliency target is gradient-times-input,

$l^*$4

Training uses a joint objective

$l^*$5

where $l^*$6 matches predicted and saliency-derived importance values, and $l^*$7 is a pairwise ranking loss that preserves token ordering rather than only absolute magnitude (Tao et al., 6 Apr 2025). This is a DLSP-like design in which saliency probing directly governs compute allocation.

In adaptive image sampling, “Learning to Zoom” introduces a saliency network $l^*$8, a sampler $l^*$9, and a task network $\mathbf{H}^{(v,l)}$0, with

$\mathbf{H}^{(v,l)}$1

The source coordinates sampled for output location $\mathbf{H}^{(v,l)}$2 are computed as saliency-weighted local centroids,

$\mathbf{H}^{(v,l)}$3

$\mathbf{H}^{(v,l)}$4

followed by differentiable resampling,

$\mathbf{H}^{(v,l)}$5

This method performs saliency-driven resource reallocation in input space rather than across hidden layers, but it shares the DLSP logic of learned, differentiable, input-conditional saliency control (Recasens et al., 2018).

4. Modal instantiations and representative systems

The following systems illustrate how DLSP or DLSP-adjacent ideas are instantiated across modalities.

Setting	Saliency signal	Action
HiProbe-VAD	KL divergence, LDR, entropy	select single optimal layer $\mathbf{H}^{(v,l)}$6
SDTP	gradient-based token saliency approximated by a two-layer MLP	hierarchically prune redundant tokens
Decoded Grad-CAM	layer-specific Grad-CAM decoded by MLM head	produce token-space attribution
EML-NET	conv1@1 + ReLU on 7 selected layers	concatenate and linearly decode saliency
Saliency Sampler	low-resolution saliency map $\mathbf{H}^{(v,l)}$7	redistribute fixed output pixel budget
Deep visual saliency probing	NSS alignment of activation maps and salient regions	quantify category selectivity across layers

In video anomaly detection, DLSP is used as an offline representational search mechanism inside HiProbe-VAD. The backbone is a pre-trained MLLM such as InternVL2.5, with additional evaluations on Qwen2.5-VL, LLaVA-OneVision, and Holmes-VAU (Cai et al., 23 Jul 2025). Hidden states are extracted during the first token generation, a single layer is selected, and a lightweight logistic-regression scorer with LBFGS is trained on that layer only. The inference pipeline then segments videos into chunks of 24 frames, uniformly samples $\mathbf{H}^{(v,l)}$8 keyframes, scores them, applies Gaussian smoothing with kernel width $\mathbf{H}^{(v,l)}$9, and uses the adaptive threshold $l$0 with $l$1 for localization.

In long-context LLM acceleration, SDTP inserts pruning modules between transformer blocks at selected stages rather than every layer, usually with $l$2 pruning stages and a geometric keep schedule $l$3 with $l$4 (Tao et al., 6 Apr 2025). The first four initial tokens and the local $l$5 of tokens are always retained. The method is designed to reduce prefill computation, not merely decode-time KV-cache size, and the original transformer weights are frozen while only the SDTP modules are trained.

In encoder-only language transformers for classification, decoded layer saliency is a probing and interpretability system rather than a pruning or routing mechanism. Its contribution is that hidden-space saliency computed at $l$6 can be decoded back into token space through the MLM head, allowing meaningful comparison of layers whose hidden states no longer map directly to input-token provenance (Hou et al., 2023). The paper treats the layer as a control variable and compares attribution quality across depths.

In visual saliency prediction, EML-NET and intermediate-neuron probing establish static multi-layer baselines that are directly relevant to DLSP. EML-NET uses 7 selected layers total, 13,536 total feature maps, compression of each selected layer into one feature map, bilinear up-sampling to $l$7, concatenation, and a decoder with $l$8 parameters (Jia et al., 2018). “Understanding and Visualizing Deep Visual Saliency Models” instead evaluates activation maps through region-level NSS association and category statistics, revealing how saliency-related semantics emerge in conv4-1 through conv5-3 and drift under fine-tuning (He et al., 2019). Both systems support the view that saliency is distributed across depth and that final outputs alone under-describe the representational structure.

5. Empirical evidence

HiProbe-VAD reports direct evidence for the value of DLSP. On UCF-Crime and XD-Violence with InternVL2.5, full HiProbe-VAD achieves 86.72 AUC and 82.15 AP, whereas the “Without DLSP: Fixed Last Layer” ablation gives 83.21 AUC and 79.28 AP, and the “Without DLSP: Fixed Mid Layer (Layer 16)” ablation gives 78.60 AUC and 75.61 AP (Cai et al., 23 Jul 2025). The same paper reports that KL divergence, LDR, entropy, and Silhouette score peak around layer 20, and that t-SNE visualizations show clearer normal/anomalous separation at layer 20 than at the input layer 0 or output layer 31. Cross-model results are also reported: on UCF-Crime, HiProbe-VAD gives 82.26 with LLaVA-OV, 85.89 with Qwen2.5-VL, 86.72 with InternVL2.5, and 88.91 with Holmes-VAU; on XD-Violence it gives 76.32, 80.94, 82.15, and 89.51, respectively.

SDTP provides a different form of empirical support for DLSP-like hidden-state probing. By hierarchically pruning 65\% of the input tokens, it reports 33\% $l$9 47\% FLOPs reduction and speedup up to 1.75$v$0 during inference while maintaining comparable performance (Tao et al., 6 Apr 2025). At 128K context length on Mistral-7B, prefill latency changes from 29.30s to 13.60s and end-to-end latency from 63.69s to 36.35s, with memory changing from 65.13GB to 42.82GB. On LongBench for Mistral-7B at the same compression ratio $v$1, the average is 42.34 for the original model, 35.16 for LLMLingua-2, and 39.82 for SDTP; the lower-compression SDTP-low variant gives 40.98 at $v$2. Ablations show that removing ranking loss degrades performance, including a drop from 66.82 to 66.27 on Llama2-7B.

Decoded layer saliency in language transformers shows that certain middle-to-late layers are more faithful and semantically coherent than the embedding layer. On SST-2, the best Decoded Grad-CAM layers are $v$3, with layer 7 strongest, improving Revealing AUC from 0.797 for vanilla Grad-CAM $v$4 to 0.867 and improving Hiding AUC from 0.713 to 0.609, where lower is better for Hiding (Hou et al., 2023). On AG News, the best layers are $v$5, with layer 6 strongest overall, giving Revealing AUC 0.895 and Hiding AUC 0.739. The same paper reports dramatically lower class-token overlap for Decoded Grad-CAM, including none among the top 50 overlapping tokens for SST-2 at $v$6.

Static multi-layer saliency systems show that intermediate-depth and multi-backbone information are already valuable even before dynamic selection is introduced. On SALICON validation, EML-NET reports DenseNet NSS 1.987, NasNet NSS 1.996, and NasNet + DenseNet NSS 2.024, with corresponding CC 0.873, 0.884, and 0.890, and KLD 0.230, 0.216, and 0.204 (Jia et al., 2018). On SALICON test, EML-NET achieves NSS 2.050, CC 0.886, AUC 0.866, sAUC 0.746, KLD 0.520, IG 0.736, and SIM 0.780. These results support the proposition that saliency benefits from combining complementary prior knowledge across depths and pretraining domains.

Adaptive image sampling provides further evidence that saliency-conditioned computation improves task performance when the input budget is fixed. On GazeCapture with AlexNet as task network, AlexNet + Saliency Sampler gives 3.29 on iPad and 2.03 on iPhone, compared with 5.44 and 2.63 for plain AlexNet, 4.33 and 2.25 for affine STN, 4.44 and 2.39 for TPS STN, and 3.91 and 2.20 for direct grid estimation (Recasens et al., 2018). On iNaturalist 2017 with ResNet-101 at $v$7, the saliency sampler improves from 60% top-1, 83% top-5 to 65 / 86; with Inception-v3, it improves from 64 / 86 to 66 / 87; on CUB-200 with ResNet-50, it improves from 81.6 to 84.5. The reported ablations show none: 60 / 83, 6-layer $v$8: 62 / 84, 10-layer $v$9: 64 / 85, and 14-layer $d$0: 65 / 86.

Intermediate-neuron analysis in deep visual saliency models contributes a different empirical lesson: internal saliency structure can be richer than final behavior. The paper reports a strong Spearman correlation,

$d$1

between category salience in inner representations and category salience in output maps, and a significant ANOVA result,

$d$2

for output-category difference (He et al., 2019). Yet on synthetic pop-out arrays, deep saliency models underperform classical models despite strong intermediate responses: DeepGaze II gives NMM 1.66, SAM 1.25, DVA 1.19, while GBVS gives 2.57 and BMS 3.65. At the representation level, however, top-10 activation-map NMM peaks at 2.12 in conv4-1 and declines to 0.38 in conv5-3, demonstrating that informative saliency can exist in middle layers while being lost at readout.

6. Misconceptions, limitations, and likely directions

A recurring misconception is that “dynamic” always means per-instance online layer choice. The literature does not support a single definition. In HiProbe-VAD, DLSP is dynamic only during offline analysis across layers, after which a single $d$3 is fixed for deployment (Cai et al., 23 Jul 2025). In SDTP, the token-importance estimate is genuinely input-dependent during inference, but pruning is still performed only at selected stages rather than every layer (Tao et al., 6 Apr 2025). In decoded layer saliency, the saliency maps themselves are input-dependent, but the best layer is treated mostly as a dataset-level empirical choice (Hou et al., 2023). EML-NET is even more clearly static: every image uses the same DenseNet and NasNet layers, the same per-layer compression heads, and the same fusion path (Jia et al., 2018).

Another misconception is that the final layer is the most informative place to read saliency. HiProbe-VAD explicitly rejects this for anomaly detection, Decoded Grad-CAM finds middle-to-late encoder layers superior to the embedding layer for textual attribution, and deep visual saliency probing shows that categories and synthetic pop-out cues can be stronger in conv4-x or conv5-1/5-2 than in conv5-3 (Cai et al., 23 Jul 2025, Hou et al., 2023, He et al., 2019). Across these settings, layer utility is depth-dependent and non-monotonic.

The limitations are correspondingly diverse. HiProbe-VAD uses a single selected layer rather than multi-layer fusion, depends on a small labeled subset, and leaves ambiguities regarding prompt text, hidden-state tensor shape, pooling strategy, number of entropy bins $d$4, and the numerical value of $d$5 in LDR (Cai et al., 23 Jul 2025). SDTP does not provide a detailed failure analysis, though it reports that earlier layers are less tolerant to aggressive pruning and that too-early pruning hurts more (Tao et al., 6 Apr 2025). EML-NET has no explicit dynamic layer selection, no attention over depths, and no learned layer dropping or routing (Jia et al., 2018). The saliency sampler introduces adaptive distortion that may be less suitable when geometric fidelity is crucial, offers no explicit theoretical guarantees, and uses a heuristic kernel-based mapping rather than an exact optimal transport solution (Recasens et al., 2018). Intermediate-neuron probing in deep saliency models is not a causal intervention study, covers a limited architecture family, and analyzes lower layers below conv4-1 only minimally (He et al., 2019).

These limitations suggest, rather than establish, several directions. A plausible implication is that future DLSP systems may combine the memory-efficient per-layer compression strategy of EML-NET with image-conditioned weighting, region-conditioned depth selection, or backbone-conditioned saliency routing (Jia et al., 2018). A second plausible implication is that runtime saliency predictors, as in SDTP, could be paired with decoded token-space attribution to produce both actionable pruning and interpretable explanations (Tao et al., 6 Apr 2025, Hou et al., 2023). A third plausible implication is that DLSP should remain decoder-aware: the synthetic-image evidence from deep visual saliency models indicates that useful internal detectors can exist even when the final readout discards them, so probing and readout evaluation should be separated rather than conflated (He et al., 2019).

In its strictest sense, DLSP is the saliency-based layer-selection mechanism defined in HiProbe-VAD. In its broader research significance, it names a growing representational viewpoint: saliency is not merely an output heatmap, but a depth-dependent internal resource that can be measured, decoded, selected, and operationalized for explanation, adaptive sampling, and efficient inference (Cai et al., 23 Jul 2025).