Post-Training Statistical Criterion

• Post-training statistical criterion is a mathematically defined test or metric applied after primary model training to optimize and assess model performance and validity.
• It employs methods such as convex last-layer retraining, selective inference, and surrogate screening to refine model parameters within a fixed representation.
• Its applications span model compression, calibrating quantized models, and robust statistical inference in areas from deep learning to scientific experiments.

A post-training statistical criterion is a mathematically defined test, optimization objective, or significance metric applied after primary model training or statistical estimation. Such criteria serve to optimize, calibrate, or assess models' generalizability, performance, interpretability, or statistical validity without further changing the underlying representation except in a constrained (often convex or isolated) parameter subspace. The concept is deeply embedded in modern statistical machine learning, model compression, calibration, neuroscientific inference, and applied statistics, and is distinguished by being applied after—rather than during—the main training epochs or prior-driven estimation.

1. Theoretical Foundations

The theoretical basis for post-training statistical criteria arises from the need to extract optimality, generalizability, or statistical interpretability from models or parameters that have undergone primary training under complex, non-convex, or overparameterized objectives. A classical example in deep learning is the post-training optimization of a network's terminal affine layer while freezing the upstream layers, whereby the last layer is re-fit as a convex empirical risk minimizer (e.g., ridge regression or logistic regression) on the fixed learned representation. Formally, if the preceding layers define a feature embedding $\Phi_{L-1}(x)$, post-training fixes $\Phi_{L-1}$ and solves the convex problem

$$\min_{W} \frac{1}{N}\sum_{i=1}^N \tilde\ell(\Phi_{L-1}(x_i)W, y_i) + \lambda \|W\|_F^2,$$

where $\tilde\ell$ incorporates the final-layer activation and loss, and $\lambda$ is a ridge penalty (Moreau et al., 2016). Kernel-theoretic interpretations via the representer theorem provide an explicit characterization of the Bayes-optimal linear predictor in the RKHS spanned by $\Phi_{L-1}$, ensuring the fixed embedding is used in a statistically optimal fashion.

Beyond deep nets, post-training statistical criteria are foundational for selective inference in high-dimensional regression (e.g., PTL-SI), post-quantization accuracy prediction, post-pruning calibration, and post-hoc significance screening in inferred graphical models.

2. Methodologies and Algorithmic Instantiations

Post-training criteria may appear as explicit convex optimization subproblems, statistical thresholding, or calibration schemes. Prominent instantiations include:

• Convex last-layer retraining: Given a learned feature map $\Phi_{L-1}$, post-training solves a ridge-regularized empirical risk problem restricted to the top layer, often in closed form via

$$W^* = (\Phi^\top\Phi + \lambda I)^{-1} \Phi^\top Y,$$

ensuring convex optimality within the frozen embedding (Moreau et al., 2016).

• Selective inference and $p$-value computation: After model selection (e.g., Lasso or transfer learning fused lasso), the post-selection distribution of test statistics is no longer standard normal; PTL-SI conditions on the selection event, reduces to a truncated normal law, and evaluates selective $p$-values by inversion along a one-dimensional parameterization (Tam et al., 25 Apr 2025).
• Surrogate significance screening: For Ising or GLM-coupling estimation in neuroscience, the post-processing step compares coupling estimates from real data to those from randomized surrogates, setting a statistical threshold based on empirical percentiles or $p$-values (Terada et al., 2018).
• Post-training quantization/statistical calibration: Approaches such as loss-aware post-training quantization (LAPQ) utilize Taylor expansions of the quantized loss landscape, estimate the curvature and cross-layer interaction terms, and perform joint step-size optimization guided by the curvature-based criterion. Calibration methods for pruning (e.g., Statistical Calibrated Activation Pruning, SCAP) calibrate layerwise mode and quantile thresholds after parameter training to induce controlled sparsity (Nahshan et al., 2019, Chua et al., 2024).
• Statistical diagnostic metrics: In LLM quantization scaling law prediction, post-training performance is forecast by learning a regression from a suite of post-quantization metrics (pre-trained loss, SQNR, RTN-loss, loss landscape slope, bit-precision, block size) to downstream NLL, circumventing expensive trial-and-error (Xu et al., 2024).

3. Statistical Validity, Power, and Interpretability

Rigorous post-training statistical criteria are designed to control Type I error rates, retain optimality under conditional selection, or preserve the information-theoretic structure of the original model. For instance:

• False positive control: PTL-SI provides selective $p$-values that are exactly uniform under the true null, rigorously controlling the false positive rate at pre-specified $\alpha$ for post-selected features, both in synthetic and real-world high-dimensional transfer learning scenarios (Tam et al., 25 Apr 2025).
• Surrogate-based significance: Screening estimated couplings against randomized surrogates in spike inference controls the false positive rate to order $\alpha$ while maintaining high true-positive detection (Terada et al., 2018).
• Information preservation in quantization: Post-training statistical pre-calibration using weight-adaptive penalties approximates minimization of KL divergence $D_{\mathrm{KL}}(f_W\|f_{\hat{W}})$ between original and quantized weights, guaranteeing preservation of Shannon information and matching the performance of data-driven calibration-based PTQ methods at vastly reduced computational cost (Ghaffari et al., 15 Jan 2025).
• Calibrating for practical objectives: In EMOS post-processing for ensemble forecasts, weighted scoring rules such as threshold-weighted continuous ranked probability score (twCRPS) allow focused accuracy improvement for tail events, with the propriety of the criterion guaranteed as long as the weighting function is non-negative. Properly tuned, twCRPS enables tailored risk control for extremes while trading off body and tail accuracy (Wessel et al., 2024).

4. Applications Across Domains

Post-training statistical criteria are deployed across a spectrum of machine learning and statistical modeling pipelines:

• Deep neural network calibration: Last-layer post-training consistently produces accuracy gains in classification and language modeling (e.g., CIFAR-10, PTB, MNIST, Faces), quantifiable as a 0.5–2% error drop or improved perplexity (Moreau et al., 2016).
• Model compression and deployment: PTQ and pruning methods employing post-training statistical calibration—whether via loss landscape characterization or direct distributional matching—enable aggressive weight/activation quantization and structured sparsity without data-intensive retraining, matching or exceeding calibration-based methods (Nahshan et al., 2019, Chua et al., 2024, Ghaffari et al., 15 Jan 2025).
• Scientific inference with ML outputs: Likelihood-ratio–based statistical tests constructed from ML discriminant outputs, with post-training calibration of PDFs and nuisance modeling, form the statistical backbone of high-energy physics hypothesis tests, providing principled $Z$-score significance and explicit error propagation from ML outputs to physics conclusions (Khosa et al., 2022).
• Robustness and generalization assessment: In reasoning LLMs under SFT–RL, post-SFT generalization loss on held-out reasoning data and Pass@large k performance emerge as statistically superior post-training predictors of RLVR outcome, compared to superficial validation accuracy, doubling predictive power ($R^2,\rho$) for end-to-end performance (Kang et al., 2 Oct 2025).
• Probabilistic forecasting: In ensemble model output statistics (EMOS), post-training with twCRPS directly addresses tail event risk, with controlled trade-off between tail and body calibration enabled by weighted score blending and linear pooling (Wessel et al., 2024).

5. Empirical and Practical Considerations

Empirical studies across domains confirm the efficacy and efficiency of post-training statistical criteria:

• Performance gains: Empirically, last-layer post-training yields consistent accuracy or loss improvement in diverse network architectures. In quantization, statistical pre-calibration matches or surpasses calibration-based routines while achieving 10–100× speedup and requiring no calibration data (Moreau et al., 2016, Ghaffari et al., 15 Jan 2025).
• Resource efficiency: Post-training approaches such as adaptive pre-calibration or loss-aware joint step-size optimization for quantization are computationally tractable, data-independent, and resilient—features critical for scaling to large models and deployment scenarios (Ghaffari et al., 15 Jan 2025, Nahshan et al., 2019).
• Statistical interpretability: Selective inference frameworks guarantee that reported $p$-values and confidence intervals reflect conditioning on the selection event, yielding statistically justifiable results even after complex model selection or transfer learning (Tam et al., 25 Apr 2025, Terada et al., 2018).
• Guidelines: Standard practice includes defining post-training significance or calibration criteria with clear, theoretically grounded statistical properties (propriety, type I error control), validating via simulation or empirical benchmarks, and adopting divide-and-conquer or surrogate methods for scalability and interpretability.

6. Limitations and Trade-offs

Post-training statistical criteria, while optimal within their defined scopes, may face trade-offs:

• Body–tail trade-off: Focusing post-training calibration or scoring on the tail (extreme events) can degrade performance on the central (body) distribution, as quantified in EMOS with twCRPS. Weighted training and linear pooling mitigate but do not eliminate this trade-off (Wessel et al., 2024).
• Scope of optimality: Post-training convex reoptimization is only optimal for the parameter subspace being retrained (e.g., last layer); upstream nonconvexities, overfitting, or representational mismatch limit the improvement's ceiling (Moreau et al., 2016).
• Dependence on initial training: Some post-training criteria cannot correct deficiencies in structural representation learned during the main training. For instance, poor feature embeddings upstream of the reoptimized last layer constrain achievable gains.
• Data and computational requirements: While many criteria are fast and data-free, selective inference and some post-calibration methods (e.g., full post-selection p-value computation) incur nontrivial computational overhead in high dimensions or large-scale models (Tam et al., 25 Apr 2025).

7. Representative Methods and Comparisons

Domain/Use Case	Post-Training Criterion	Core Statistical Principle
Deep nets (last-layer)	Convex empirical risk, ridge	RKHS optimality over fixed embedding (Moreau et al., 2016)
Feature selection (TL)	Selective $p$-values, PTL-SI	Conditioning on selection, truncated normal law
Quantization	Loss-aware curvature Lapq, adaptive LASSO	Curvature minimization, KL-divergence preservation
Activation pruning	Mode-centered quantile threshold	Mode-centering for maximal sparse mass
Probabilistic forecasting	twCRPS, weighted scores	Proper scoring focus on tail with body–tail trade-off
HEP ML significance	LLR test statistics on trained scores	Neyman–Pearson and Poisson-likelihood anatomy (Khosa et al., 2022)

These methodologies exemplify the diversity and analytical rigor characteristic of post-training statistical criteria in contemporary research.

---

References

• Post Training in Deep Learning with Last Kernel (Moreau et al., 2016)
• A Statistical Framework for Single Subject Design (Lu et al., 2016)
• Post-Transfer Learning Statistical Inference (Tam et al., 25 Apr 2025)
• Loss Aware Post-training Quantization (Nahshan et al., 2019)
• Rethinking Post-Training Quantization: Introducing a Statistical Pre-Calibration Approach (Ghaffari et al., 15 Jan 2025)
• Post-Training Statistical Calibration for Higher Activation Sparsity (Chua et al., 2024)
• Scaling Laws for Post Training Quantized LLMs (Xu et al., 2024)
• Inferring neuronal couplings … with a statistical criterion (Terada et al., 2018)
• A simple guide from Machine Learning outputs to statistical criteria (Khosa et al., 2022)
• Image Tokenizer Needs Post-Training (Qiu et al., 15 Sep 2025)
• Improving probabilistic forecasts … weighted scoring rules (Wessel et al., 2024)
• Quagmires in SFT-RL Post-Training (Kang et al., 2 Oct 2025)