Semantic-Aware Layer-wise Gaussian Process

• The paper introduces SAL-GP, which calibrates deep neural networks by regressing the residual between prediction correctness and softmax confidence using structured Gaussian process kernels.
• SAL-GP integrates semantic awareness by leveraging augmented, layer-indexed features to provide both global and local confidence corrections, highlighting individual layer contributions.
• Empirical results demonstrate that the multi-layer variant robustly reduces calibration errors (ECE and MCE) across architectures, outperforming traditional methods like temperature scaling.

Searching arXiv for the specified SAL-GP and related Gaussian-process layer-wise papers. Semantic-Aware Layer-wise Gaussian Process (SAL-GP) is a semantic-aware, layer-aligned Gaussian-process calibration framework for deep neural networks in which internal layer representations are treated as structured inputs to a single GP with a multi-layer kernel, rather than correcting the network only at its output as a black box. In the 2025 formulation, SAL-GP calibrates the softmax confidence $s(x)$ by modeling the residual $r(x)=c(x)-s(x)$, where $c(x)\in\{0,1\}$ is the correctness of the top-1 prediction, and outputs a calibrated confidence distribution $s'(x)$ with posterior uncertainty (Lee et al., 21 Jul 2025). A plausible conceptual antecedent is the Gaussian-process view of individual layers developed in 2019, where wide-network NNGP and NTK correspondences were used to derive explicit supervised layer-wise losses; that earlier work did not present SAL-GP as calibration, but it established the layer-wise GP machinery from which semantic-aware variants can be constructed (Ben-David et al., 2019).

1. Problem setting and conceptual scope

SAL-GP addresses neural-network miscalibration: predicted probabilities may not match empirical correctness frequencies, and the discrepancy can become severe under domain shift, limited training data, or architectural idiosyncrasies (Lee et al., 21 Jul 2025). The framework is motivated by the observation that overconfidence can be seeded and propagated through intermediate layers, so a single global GP correction at the network output may ignore the internal hierarchical structure that contributes to miscalibration.

The calibration target is typically the maximum softmax confidence. Given logits $z(x)\in\mathbb{R}^K$ and class probabilities $p(x)=\operatorname{softmax}(z(x))$, SAL-GP defines

$$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$

Calibration is then expressed as

$s'(x_*)=s(x_*)+\bar r_*(x_*),$

with predictive uncertainty

$$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$

This formulation makes the calibration object explicit: SAL-GP corrects confidence by regressing residual error on internal features and confidence jointly, while preserving GP uncertainty quantification (Lee et al., 21 Jul 2025).

The framework is post-hoc rather than end-to-end. The base DNN is first trained and then frozen; SAL-GP consumes extracted layer features, softmax confidence, and a layer index as inputs to a structured GP. This distinguishes it from representation-learning methods that optimize hidden layers themselves. It also distinguishes it from standard temperature scaling, isotonic regression, and BBQ, which adjust probabilities but do not quantify calibration uncertainty in the same Bayesian sense (Lee et al., 21 Jul 2025).

2. Gaussian-process antecedents and layer-wise semantics

The 2019 Gaussian-process perspective on layers established the correspondence between wide DNNs and GPs under several regimes: random iid weight priors induce an NNGP at initialization; full-batch gradient descent with added white noise and vanishing learning rate maps SGD to Bayesian inference on the NNGP with observation noise $\sigma^2=T$; and zero-noise, vanishing-learning-rate full-batch gradient descent corresponds to GP regression with the Neural Tangent Kernel (NTK) (Ben-David et al., 2019). For a random network $r(x)=c(x)-s(x)$0 with prior $r(x)=c(x)-s(x)$1, the induced function-space prior is

$r(x)=c(x)-s(x)$2

with covariance

$r(x)=c(x)-s(x)$3

The GP posterior mean for regression on training data is

$r(x)=c(x)-s(x)$4

For a specific layer $r(x)=c(x)-s(x)$5, the representation is denoted $r(x)=c(x)-s(x)$6, and the top network above that layer induces a kernel $r(x)=c(x)-s(x)$7 or, in the NTK regime, $r(x)=c(x)-s(x)$8 (Ben-David et al., 2019). The paper’s Deep Gaussian Layer-wise losses (DGLs) evaluate how well the top network would classify data represented at a given layer using GP leave-one-out prediction. With inverse Gram matrix $r(x)=c(x)-s(x)$9 and label vectors $c(x)\in\{0,1\}$0, the layer-wise objective is

$c(x)\in\{0,1\}$1

The supervision enters only through pairwise label dot products $c(x)\in\{0,1\}$2; for one-hot encodings, this vanishes unless labels are equal. The paper therefore describes a label-aware geometry induced by the top-network kernel and weighted by target similarity (Ben-David et al., 2019).

This structure is directly relevant to SAL-GP. A plausible implication is that semantic awareness can be injected through the same algebraic slot occupied by $c(x)\in\{0,1\}$3. The reconstructed SAL-oriented extension in the 2019 summary replaces the label dot product with a positive semidefinite label kernel $c(x)\in\{0,1\}$4 and defines

$c(x)\in\{0,1\}$5

That replacement is explicitly marked as an extrapolation rather than a claim of the 2019 paper itself (Ben-David et al., 2019).

The same earlier work also supplied a practical greedy optimization procedure, LEGO, in which layers are optimized sequentially using GP-derived objectives. In reported experiments, DGL was competitive with end-to-end training; for example, on CIFAR10$c(x)\in\{0,1\}$6 with $c(x)\in\{0,1\}$7, $c(x)\in\{0,1\}$8, end-to-end accuracy was $c(x)\in\{0,1\}$9 and DGL achieved $s'(x)$0, while on BMNIST$s'(x)$1 with $s'(x)$2, $s'(x)$3, end-to-end accuracy was $s'(x)$4 and DGL achieved $s'(x)$5 (Ben-David et al., 2019). This does not make DGL a calibration method, but it suggests that layer-wise GP objectives can preserve task-relevant semantics with explicit supervision.

3. SAL-GP architecture, augmented inputs, and structured kernels

SAL-GP is defined on a pre-trained DNN with $s'(x)$6 layers and layer features $s'(x)$7 (Lee et al., 21 Jul 2025). In CNNs, $s'(x)$8 is obtained from a feature map $s'(x)$9 by pooling across channels, for example

$z(x)\in\mathbb{R}^K$0

followed by flattening to $z(x)\in\mathbb{R}^K$1 or $z(x)\in\mathbb{R}^K$2. In RNNs, $z(x)\in\mathbb{R}^K$3 can be the hidden state vector at GRU/LSTM layers, with optional pooling across time to obtain fixed-size vectors.

For each layer $z(x)\in\mathbb{R}^K$4, SAL-GP conceptually models a local correction $z(x)\in\mathbb{R}^K$5 to the residual $z(x)\in\mathbb{R}^K$6. An additive interpretation is

$z(x)\in\mathbb{R}^K$7

although the framework realizes this through a single GP over augmented inputs rather than through $z(x)\in\mathbb{R}^K$8 independent GPs. The augmented input at layer $z(x)\in\mathbb{R}^K$9 is

$p(x)=\operatorname{softmax}(z(x))$0

where $p(x)=\operatorname{softmax}(z(x))$1 is the pooled feature vector, zero-padded to width $p(x)=\operatorname{softmax}(z(x))$2 if needed, $p(x)=\operatorname{softmax}(z(x))$3 is the maximum softmax confidence, and $p(x)=\operatorname{softmax}(z(x))$4 is the layer index (Lee et al., 21 Jul 2025).

Two kernel constructions are central.

Variant	Kernel form	Role
SAL-GP(HL)	$p(x)=\operatorname{softmax}(z(x))$5	shared global component plus same-layer correction
SAL-GP(ML)	reduced ICM additive kernel	scalable approximation to multi-output GP coupling

In the hierarchical-layer variant,

$p(x)=\operatorname{softmax}(z(x))$6

This separates a globally shared correlation structure from a layer-specific term active only when the two inputs come from the same layer. In the reduced ICM construction, the full multi-output GP kernel is

$p(x)=\operatorname{softmax}(z(x))$7

with PSD coregionalization matrix $p(x)=\operatorname{softmax}(z(x))$8, and the scalable additive approximation becomes

$p(x)=\operatorname{softmax}(z(x))$9

The base kernels used in the reported experiments include RBF and Matérn, with Matérn $$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$0 used in experiments (Lee et al., 21 Jul 2025).

Because layer indices are explicit inputs and the kernel decomposes into global and layer-specific terms, SAL-GP yields both global calibrated predictions and local per-layer predictions. This is the source of its architecture-aligned interpretability: one can inspect which layers contribute most to correction rather than only observing an end-of-pipeline confidence adjustment (Lee et al., 21 Jul 2025).

4. Posterior inference, optimization, and computational profile

SAL-GP places a GP prior on residuals over all layer-indexed inputs. If all function values are stacked into $$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$1, then

$$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$2

and the residual-regression likelihood is

$$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$3

Kernel hyperparameters are optimized by exact GP marginal likelihood,

$$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$4

The resulting predictive distribution for a test input with layer index $$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$5 is Gaussian in the usual GP sense, and uncertainty is propagated directly to the calibrated confidence $$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$6 (Lee et al., 21 Jul 2025).

The training procedure is explicit. One first trains the base DNN and freezes its weights; extracts per-layer features for all calibration samples; computes $$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$7, $$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$8, and residuals $$s_i=\max_k p_k(x_i), \qquad c_i=\mathbf{1}\!\left[y_i=\arg\max_k p_k(x_i)\right], \qquad r_i=c_i-s_i.$$9; constructs the augmented training set $s'(x_*)=s(x_*)+\bar r_*(x_*),$0; chooses SAL-GP(HL) or SAL-GP(ML); and fits kernel hyperparameters by maximizing marginal likelihood. For test-time global prediction, the layer index is set outside the training layer indices so that only global kernel terms contribute to the cross-covariance; local per-layer predictions are obtained by querying with a specific layer index (Lee et al., 21 Jul 2025).

The computational bottleneck is covariance inversion. A naïve multi-output GP with ICM requires $s'(x_*)=s(x_*)+\bar r_*(x_*),$1 time and $s'(x_*)=s(x_*)+\bar r_*(x_*),$2 memory. Kronecker structure in $s'(x_*)=s(x_*)+\bar r_*(x_*),$3 can reduce some operations to $s'(x_*)=s(x_*)+\bar r_*(x_*),$4, but only with specialized solvers. The additive kernel gives a block-sparse structure with approximate cost near $s'(x_*)=s(x_*)+\bar r_*(x_*),$5 and reduced memory, which is why the reduced ICM variant is emphasized for scalability (Lee et al., 21 Jul 2025). Inducing points and sparse GP methods were not used in the reported experiments but are recommended for very large $s'(x_*)=s(x_*)+\bar r_*(x_*),$6 or $s'(x_*)=s(x_*)+\bar r_*(x_*),$7.

This post-hoc calibration protocol differs sharply from the layer-wise optimization program in DGL. There, the objective is to optimize each hidden layer itself using the GP behavior of the network above it; here, the network is frozen and the GP operates on extracted features to calibrate confidence. The connection is methodological rather than procedural: both rely on kernels defined over layer representations, but they target different outputs and training regimes (Ben-David et al., 2019).

5. Empirical behavior across architectures and shifts

The 2025 SAL-GP study evaluates image classification on MSTAR SAR data using ConvNet, AConvNet, and ResNet-18, and time-series classification on PLAsTiCC astronomy data using a multi-passband bidirectional GRU-based RNN (Lee et al., 21 Jul 2025). Baselines are uncalibrated softmax, temperature scaling (TS), and single-layer GP calibration using a single feature layer together with $s'(x_*)=s(x_*)+\bar r_*(x_*),$8. The primary metric is ECE, with MCE, NLL, and Brier score also reported.

On ConvNet for MSTAR, uncalibrated ECE is $s'(x_*)=s(x_*)+\bar r_*(x_*),$9. TS worsens it to $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$0, which the paper attributes to degradation under perfect validation accuracy when the optimizer pushes the temperature too low. The best single-layer GP reaches ECE $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$1 at Layer 4 with average pooling. SAL-GP(ML) achieves ECE $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$2 for the global prediction with average pooling, reduces MCE to approximately $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$3 versus approximately $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$4 baseline, and improves NLL from $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$5 to $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$6. SAL-GP(HL) is less consistent, with global ECE approximately $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$7 to $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$8 depending on pooling (Lee et al., 21 Jul 2025).

On AConvNet for MSTAR, the setting is strongly overconfident. Uncalibrated ECE is $$s'(x_*)\sim \mathcal{N}\!\left(s(x_*)+\bar r_*(x_*), \operatorname{Var}(r_*(x_*))\right).$$9 and TS reduces it only to $\sigma^2=T$0. The best single-layer GP reaches ECE $\sigma^2=T$1 at Layer 2 with average pooling. SAL-GP(ML) improves this slightly to $\sigma^2=T$2 for the global prediction with average pooling and is reported to be markedly more robust across layers than single-layer GP. SAL-GP(HL) typically lies between $\sigma^2=T$3 and $\sigma^2=T$4 ECE, underperforming both SAL-GP(ML) and the best single-layer GP in this configuration (Lee et al., 21 Jul 2025).

On ResNet-18 for MSTAR, uncalibrated ECE is $\sigma^2=T$5 and TS gives a minor improvement to $\sigma^2=T$6. The best single-layer GP reaches ECE $\sigma^2=T$7 at Layer 4 with max pooling or $\sigma^2=T$8 at Layer 3 with average pooling. SAL-GP(ML) yields ECE approximately $\sigma^2=T$9 to $r(x)=c(x)-s(x)$00 depending on pooling, but can raise MCE because it assigns low confidence to some actually correct samples; the paper interprets this as stronger responsiveness to negative residuals. SAL-GP(HL) gives ECE approximately $r(x)=c(x)-s(x)$01 to $r(x)=c(x)-s(x)$02 and produces smoother residual fits, with less MCE increase but more limited responsiveness to extreme errors (Lee et al., 21 Jul 2025).

The strongest gain appears in the RNN under severe domain shift, class imbalance, and unknown classes. Train accuracy is approximately $r(x)=c(x)-s(x)$03 and test accuracy approximately $r(x)=c(x)-s(x)$04. Uncalibrated ECE is $r(x)=c(x)-s(x)$05, TS reduces it to $r(x)=c(x)-s(x)$06, and single-layer GP across GRU layers remains between approximately $r(x)=c(x)-s(x)$07 and $r(x)=c(x)-s(x)$08, providing no meaningful improvement. SAL-GP(ML) lowers ECE to $r(x)=c(x)-s(x)$09 when using global prediction over layers 1–5, and to $r(x)=c(x)-s(x)$10 with selective early layers 1–3. SAL-GP(HL) remains around $r(x)=c(x)-s(x)$11 to $r(x)=c(x)-s(x)$12 ECE (Lee et al., 21 Jul 2025).

The interpretability analysis is layer-resolved. Residual fit plots show SAL-GP(ML) to be more responsive to negative residuals, i.e., overconfident samples, than single-layer GP; SAL-GP(HL) is smoother but underfits extreme corrections, particularly positive residuals associated with underconfidence. Local per-layer predictions identify which layers contribute most to correction, such as early layers in the RNN under shift and middle convolutional layers in AConvNet (Lee et al., 21 Jul 2025). This suggests that the “semantic-aware” designation refers not merely to labels or classes, but to feature semantics distributed across the network hierarchy.

6. Relations to adjacent methods, misconceptions, limitations, and extensions

SAL-GP is not a deep Gaussian process in the usual latent-function sense. Deep GPs model latent functions at each layer and marginalize through compositions,

$r(x)=c(x)-s(x)$13

which is generally intractable and can amplify predictive variance across layers. SAL-GP instead conditions on deterministic features extracted from a trained DNN and couples layer-wise calibration corrections through a structured kernel, thereby avoiding the intractable posterior marginalization of DGPs while retaining coherent uncertainty propagation (Lee et al., 21 Jul 2025).

SAL-GP is also not identical to general multi-output GP calibration. Its reduced ICM additive kernel is an interpretable, scalable approximation to multi-output dependence specialized to layer-indexed neural features. Relative to standard post-hoc calibration methods such as TS, Platt scaling, or BBQ, SAL-GP provides Bayesian uncertainty over the calibration correction and can expose which internal layers are responsible for confidence misalignment (Lee et al., 21 Jul 2025).

A common misconception is that better average calibration necessarily implies good worst-case calibration. The reported experiments do not support that conclusion. Under severe OOD conditions, SAL-GP(ML) improves ECE substantially, but MCE can remain high, and the paper explicitly notes that complementary OOD handling may still be required (Lee et al., 21 Jul 2025). Another misconception would be to treat SAL-GP(HL) and SAL-GP(ML) as interchangeable. The reported results show that SAL-GP(HL) can underperform in several architectures, whereas SAL-GP(ML) is the more robust variant in the miscalibrated settings studied (Lee et al., 21 Jul 2025).

Practical limitations are also explicit: computational overhead remains substantial; performance is sensitive to kernel choice and hyperparameters; the method relies on a separate calibration set or held-out data; and feature drift between calibration and deployment can degrade performance (Lee et al., 21 Jul 2025). Recommended practice includes pooled feature maps, zero-padding feature vectors to a consistent width, starting with the additive kernel $r(x)=c(x)-s(x)$14, optimizing by marginal likelihood, monitoring ECE and MCE on calibration splits, and using sparse GP methods such as FITC or SVGP when $r(x)=c(x)-s(x)$15 or $r(x)=c(x)-s(x)$16 is large (Lee et al., 21 Jul 2025).

The relation to the 2019 DGL framework remains conceptually important. DGL showed that layer quality can be scored explicitly through the GP or NTK behavior of the upper network, and that supervision enters through label-dependent pairwise terms with strong symmetry constraints, including $r(x)=c(x)-s(x)$17 invariance for fully connected top networks and, in a linear preclassifier limit, a larger $r(x)=c(x)-s(x)$18 symmetry as $r(x)=c(x)-s(x)$19 (Ben-David et al., 2019). A plausible extension from that formalism is to regard SAL-GP not only as post-hoc confidence calibration but also as a broader family of semantic layer-wise GP objectives in which semantic structure is encoded through label kernels, alignment losses, or multi-output constructions. In the supplied reconstruction, such replacements are clearly marked as extrapolations rather than claims of the original DGL paper (Ben-David et al., 2019).

Taken together, the literature defines SAL-GP as a calibration-specific, layer-aligned GP framework with structured kernels and uncertainty-aware residual correction, while also situating it within a wider GP view of deep-network layers. Its main technical contribution is to couple per-layer semantics and global coherence in a single posterior model; its main empirical strength is robust ECE reduction in miscalibrated and shifted settings, especially for SAL-GP(ML); and its main unresolved issues concern scalability, hyperparameter sensitivity, and worst-case calibration under OOD conditions (Lee et al., 21 Jul 2025).