Pretrain-and-Calibrate (PaC) Workflow

• The PaC workflow is defined by a pretraining stage that generates a model or score, followed by a calibration stage to refine confidence estimates for subsequent tasks.
• It employs diverse calibration methods—such as temperature scaling, mixup, and synthetic pretraining—to reduce miscalibration, improve accuracy, and enable effective sample prioritization.
• The framework supports various applications including uncertainty-based sample selection, PAC confidence sets, and label-free LLM calibration, underpinning reliable downstream performance.

Pretrain-and-Calibrate (PaC) is best understood as an umbrella workflow in which a model, representation, or score generator is obtained first and then calibrated before it is used for downstream selection, confidence estimation, or task adaptation. In the literature, this pattern is not standardized under a single formal algorithmic name: several relevant papers explicitly note that they do not introduce a method called PaC, yet they fit the pattern closely. Across these works, the “pretrain” stage may be a higher-capacity target model, a frozen deep probabilistic predictor, a synthetic random-noise pretraining phase, or an original pre-trained LLM; the “calibrate” stage may be training-time calibration, post-hoc temperature scaling, threshold calibration for confidence sets, or uncertainty shaping before fine-tuning (Tata et al., 2022, Cheon et al., 2024, Park et al., 2019, Luo et al., 22 May 2025).

1. Conceptual scope and recurring workflow

A common PaC structure appears in several distinct settings. In sample prioritization during training, the workflow is: use a current model or a larger-capacity pre-trained target model, ensure that predictive probabilities are calibrated, and then use those probabilities to prioritize samples by uncertainty (Tata et al., 2022). In uncertainty pre-calibration, the workflow is: initialize a classifier conventionally, pretrain it on random noise with random labels so that initial confidence is pushed toward the chance level, and then fine-tune on the downstream task (Cheon et al., 2024). In PAC-valid confidence-set prediction, the workflow is: train a probabilistic predictor, freeze it, apply temperature scaling, and then choose a confidence-set threshold on a held-out calibration split (Park et al., 2019). In LLM post-training, the workflow is: treat the original pre-trained LLM as a source of comparatively trustworthy confidence, and use it to calibrate a post-trained model without labels (Luo et al., 22 May 2025).

Instantiation	Pretrained or frozen object	Calibration target
Sample prioritization	ResNet-34 current model and optional ResNet-50 target (Tata et al., 2022)	Entropy scores for subset selection
Random-noise pre-calibration	Entire classifier after synthetic pretraining (Cheon et al., 2024)	Confidence pushed toward chance level
PAC confidence sets	Frozen probabilistic predictor (Park et al., 2019)	Temperature and threshold $T$
DACA for PoLMs	PLM used as calibrator for PoLM (Luo et al., 22 May 2025)	Temperature $\tau$ on unlabeled data

This suggests that PaC is less a single algorithm than a family resemblance. The shared premise is that downstream decisions based on confidence or uncertainty are only as reliable as the scores that drive them. Consequently, pretraining and calibration are coupled: the first stage supplies a predictor or prior, and the second stage makes its outputs usable for ranking, abstention, confidence-set construction, or controlled downstream adaptation.

2. Calibration objects and formal definitions

The relevant literature calibrates several different mathematical objects. In classification, confidence is typically the probability of the predicted class,

$$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$

and a calibrated model should satisfy that among predictions made with confidence $c$, the empirical accuracy is approximately $c$ (Cheon et al., 2024). Expected Calibration Error is defined using bins $B_m$ as

$$\text{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\mathbf{1}(\hat{y}_i=y_i), \qquad \text{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\text{conf}_i,$$

$$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|\text{acc}(B_m)-\text{conf}(B_m)\right|,$$

with $M=10$ bins in the random-noise pretraining study (Cheon et al., 2024). The LLM calibration study also uses ECE with $G=10$ bins, and additionally reports MCE, AECE, and Brier score (Luo et al., 22 May 2025).

In calibration-driven subset selection, the score being calibrated is not a final confidence threshold but an uncertainty ranking functional. The prioritization criterion is predictive entropy,

$\tau$0

and samples with highest entropy are selected for training in subsequent epochs (Tata et al., 2022). The conceptual dependency is explicit: if probabilities are overconfident, entropy-based ranking can be wrong, so calibration directly affects which examples are considered informative.

In PAC confidence sets, calibration is applied not merely to probabilities but to set-valued predictions. Given a calibrated forecaster $\tau$1, the confidence set is

$\tau$2

and its population error is

$\tau$3

The threshold $\tau$4 is then chosen on a held-out validation set so that the resulting confidence set has a finite-sample PAC guarantee (Park et al., 2019).

Temperature scaling appears in two distinct forms. In PAC confidence sets, a frozen predictor is calibrated by

$\tau$5

with $\tau$6 fitted on a separate calibration split (Park et al., 2019). In DACA, the calibrated class probabilities are

$\tau$7

and $\tau$8 is optimized without labels by aligning the post-trained model’s output distribution to that of the PLM on agreement examples (Luo et al., 22 May 2025).

3. Calibration-driven subset selection as a PaC workflow

The most explicit PaC-style training pipeline is the study of calibration and sample prioritization on CIFAR-10 and CIFAR-100. The setup is: split the training data into $\tau$9 train and $$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$0 validation, train a current model (ResNet-34), use all samples for the first 10 warm-up epochs, then at each subsequent epoch compute predictive entropy for each sample, select the top-$$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$1 or top-$$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$2 most informative examples, and train only on that subset. The calibration methods actually evaluated are Label Smoothing, Mixup, and Focal Loss; the baseline is No Calibration / Cross-Entropy. The target-model variant uses a larger-capacity pre-trained calibrated target model, ResNet-50 trained with Mixup, to compute entropy scores that guide the current model’s training (Tata et al., 2022).

The subset sizes are $$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$3, corresponding to a 90\%, 80\%, and 70\% reduction in the number of examples per epoch after warm-up. The paper’s central empirical claim is that calibration improves sample prioritization, can reduce the number of examples per epoch by at least 70\%, and can thereby speed up the overall training process. The principal metrics are test accuracy and ECE, with additional analyses of validation ECE, class distribution across epochs, and percentage of common samples between consecutive epochs (Tata et al., 2022).

The reported results make the calibration effect concrete. On CIFAR-10, the uncalibrated baseline at 100\% data gives 94.1 accuracy and 4.1 ECE, while Mixup at 100\% gives 95.1 accuracy and 2.1 ECE. At 30\% data, the baseline gives 93.6 accuracy and 5.33 ECE, while Mixup gives 94.39 accuracy and 2.67 ECE. On CIFAR-100, the contrast is sharper: the uncalibrated baseline at 30\% gives 73.13 accuracy and 10.77 ECE, whereas Mixup at 30\% gives 73.57 accuracy and 1.49 ECE. The authors summarize the broader pattern by stating that “Mixup consistently has higher test accuracies and low test ECEs across different subset sizes compared to other calibration techniques for both datasets” (Tata et al., 2022).

The target-model variant strengthens the PaC interpretation. With a pre-trained calibrated ResNet-50 target, Label Smoothing on CIFAR-10 at 30\% improves from 91.74 accuracy and 3.17 ECE to 93.62 accuracy and 2.93 ECE; on CIFAR-100 at 30\%, it improves from 72.21 accuracy and 3.45 ECE to 76.06 accuracy and 2.28 ECE. For Mixup on CIFAR-100 at 30\%, the target-guided variant improves from 73.57 accuracy and 1.49 ECE to 75.62 accuracy and 0.86 ECE. A plausible implication is that, in this class of methods, pretraining contributes a stronger ranking signal and calibration makes that signal operationally reliable for uncertainty-based selection (Tata et al., 2022).

4. Pre-calibration by synthetic pretraining

A more radical PaC interpretation appears in the proposal to calibrate uncertainty by changing the pretraining state itself. Here, the claim is that standard random initialization is an underlying cause of miscalibration, leading to excessively high confidence in untrained networks, and that pretraining on random noise with random labels can “pre-calibrate” the network before any real data are seen (Cheon et al., 2024).

The procedure has two stages. First, the entire classifier is pretrained on synthetic examples with Gaussian noise inputs of shape $$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$4, mean $$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$5, standard deviation $$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$6, and random one-hot labels sampled uniformly across classes; both inputs and labels are re-sampled at every iteration. Second, the same network parameters are trained normally on real labeled data, in the main experiments CIFAR-10. The models are feedforward MLPs with hidden width 256, depths 2 through 6, ReLU nonlinearities, batch normalization in every layer, and a final Softmax. Optimization in both stages uses Adam with learning rate between $$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$7 and $$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$8, betas $$\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),$$9, and weight decay $c$0 (Cheon et al., 2024).

The paper’s mechanistic claim is that random initialization induces arbitrary but confident decision structure, whereas random-noise pretraining washes this out and pushes predictions toward the chance-level distribution over classes. For a $c$1-class problem, the chance-level confidence is $c$2; for CIFAR-10, this is $c$3. The authors report that loss drops during random-noise pretraining while accuracy stays at chance, and that predictive distributions become flatter and less class-biased. Before downstream training, the random-noise pretrained network, when evaluated on CIFAR-10 or SVHN, produces confidence much closer to chance while accuracy remains at chance (Cheon et al., 2024).

Empirically, the method reduces confidence–accuracy mismatch and lowers ECE across data sizes and depths. The reported qualitative summary is: lower test loss, higher accuracy, lower ECE on CIFAR-10 downstream classification; significant ECE reduction across all tested training sizes $c$4; significant ECE reduction across all tested depths; and lower OOD confidence with higher AUROC in the CIFAR-10 versus SVHN setting. The paper does not compare against strong post-hoc calibrators, but it provides direct evidence for a PaC view in which calibration is treated as an initialization or pretraining problem rather than an afterthought (Cheon et al., 2024).

5. Post-hoc PaC: confidence sets and label-free LLM calibration

One line of work realizes PaC as a fully post-hoc pipeline. In PAC confidence sets for deep neural networks, the sequence is explicit: train a deep probabilistic predictor on $c$5, freeze it, fit a scalar calibration parameter $c$6 on a second split $c$7, and then choose the smallest threshold $c$8 on a third split $c$9 such that the confidence set satisfies a PAC constraint. The empirical error is

$c$0

and the paper’s main guarantee is that if $c$1 is chosen appropriately, then

$c$2

This construction is demonstrated on ResNet for ImageNet, a visual object tracking model, and a dynamics model for the half-cheetah reinforcement learning problem. The paper reports that temperature scaling reduces maximum confidence set size noticeably, that the direct finite-sample bound is much tighter than the VC bound, and that incorrectly classified images tend to receive much larger confidence sets than correctly classified ones (Park et al., 2019).

A second post-hoc line uses a pre-trained LLM itself as an unsupervised confidence calibrator for a post-trained LLM. In this setting, the PLM $c$3 outputs $c$4, the PoLM $c$5 is calibrated by temperature scaling, and naive confidence alignment would fit

$c$6

The paper argues that disagreement examples are problematic because PLM confidence tracks the PLM’s own prediction accuracy rather than the PoLM’s. DACA therefore retains only agreement examples,

$c$7

and minimizes the same KL divergence with an indicator gate that is zero on disagreements (Luo et al., 22 May 2025).

This agreement-only filter is motivated both theoretically and empirically. The paper proves that disagreement can induce unavoidable calibration error under naive alignment and gives an extreme condition under which the optimal temperature goes to $c$8. DACA is evaluated on MMLU, MedMCQA, MathQA, and TruthfulQA, across open-source and API-based LLMs. The abstract reports that it improves average ECE by up to 15.08\%. Representative results include Gemma-3-12B-Instruct on MMLU, where Vanilla ECE is $c$9, DACA ECE is $B_m$0, and supervised TS gives $B_m$1; GPT-4o on MedMCQA, calibrated with Gemma-3-12B as PLM, where Vanilla ECE is $B_m$2 and DACA reduces it to $B_m$3; and Qwen2.5-32B-Instruct on TruthfulQA, where Vanilla ECE is $B_m$4 and DACA reduces it to $B_m$5. In this variant of PaC, the pre-trained model functions as a frozen teacher for confidence rather than for labels (Luo et al., 22 May 2025).

6. Related but non-equivalent uses of “PAC”

The PaC literature is complicated by acronym collisions. In fine-tuning pretrained LLMs, “PAC-tuning” uses PAC-Bayes training to learn parameter-wise noise variances and then applies perturbed gradient descent during downstream adaptation. The paper is explicit that its “PAC” means PAC-Bayes / probably approximately correct, not Pretrain-and-Calibrate, and it does not evaluate ECE or other predictive calibration metrics. Its strongest connection to PaC is a broader one: Stage 1 learns a parameter distribution and Stage 2 adapts the model using the learned variances as perturbation scales (Liu et al., 2023).

A similar ambiguity appears in PACTran, which derives transferability metrics for pretrained model selection from PAC-Bayesian bounds. PACTran requires a small labeled target sample $B_m$6, optimizes a PAC-Bayesian surrogate involving empirical target classification loss and $B_m$7, and is evaluated as a ranking metric through Kendall-Tau correlation with downstream error. It does not calibrate predictive probabilities; rather, it “calibrates” transferability only in the broad sense of providing a theoretically grounded estimate of which checkpoint will transfer best (Ding et al., 2022).

PAC-Net is closer to the PaC theme because its three stages are literally Prune, Allocate, and Calibrate. The method first trains on a source task, prunes by magnitude to identify essential and redundant weights, retrains the unpruned subnetwork on the source task, and then calibrates on the target task by updating only the remaining redundant weights while freezing the source-essential subset. The target adaptation stage is

$B_m$8

with the protected source weights $B_m$9 held fixed. The paper explicitly characterizes this as preserving source knowledge completely while adjusting the model with target data (Myung et al., 2022).

There is also a wholly unrelated usage in coding theory: “PAC codes” means polarization-adjusted convolutional codes, a family of short-blocklength channel codes with rate profiling, convolutional precoding, and polar encoding. The acronym there has no connection to calibration or pretraining in machine learning (Yao et al., 2020).

7. Limitations, assumptions, and open questions

A central limitation is terminological. Several of the most relevant papers explicitly do not define a method called PaC; the label is interpretive rather than native to the papers themselves. This matters because the literature does not yet offer a single canonical decomposition of what counts as pretraining, what counts as calibration, and whether calibration is applied to probabilities, entropy rankings, confidence sets, or parameter perturbation scales (Tata et al., 2022, Cheon et al., 2024, Liu et al., 2023).

The empirical scope is also narrow in different ways across subareas. Calibration-driven sample prioritization is limited to CIFAR-10 and CIFAR-100 with ResNet-34 as current model and ResNet-50 as target model, uses only Max Entropy as the prioritization criterion, does not evaluate post-hoc methods such as temperature scaling, and does not report detailed wall-clock speed benchmarks (Tata et al., 2022). Random-noise pre-calibration is demonstrated mainly for feedforward MLPs in classification, with OOD evaluation focused on CIFAR-10 versus SVHN and no head-to-head comparison against strong calibration baselines (Cheon et al., 2024). DACA depends on access to a reasonably calibrated PLM and to option-level probabilities or logprobs, and it reduces sample efficiency by discarding disagreement examples (Luo et al., 22 May 2025).

Some approaches impose strong data assumptions. PAC confidence sets require i.i.d. train, calibration, and test data from the same distribution $$\text{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\mathbf{1}(\hat{y}_i=y_i), \qquad \text{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\text{conf}_i,$$0, need separate held-out splits for model fitting, temperature calibration, and threshold calibration, and provide only marginal coverage guarantees rather than conditional guarantees (Park et al., 2019). PAC-Net assumes the same marginal input distribution $$\text{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\mathbf{1}(\hat{y}_i=y_i), \qquad \text{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\text{conf}_i,$$1 for source and target, requires substantial parameter redundancy so that pruning leaves enough capacity for target calibration, and relies on magnitude pruning as a heuristic indicator of source-essential weights (Myung et al., 2022). PACTran, despite its relevance to checkpoint selection, requires labeled target data and remains a ranking method rather than a predictive calibration method (Ding et al., 2022).

The open design questions are correspondingly heterogeneous. For subset selection, the unresolved issue is whether other prioritization rules beyond Max Entropy benefit similarly from calibration (Tata et al., 2022). For random-noise pre-calibration, the broader question is whether the same effects hold for modern CNNs, transformers, or LLMs (Cheon et al., 2024). For unsupervised PoLM calibration, the critical dependency is the quality and accessibility of the calibrating PLM (Luo et al., 22 May 2025). More generally, the existing evidence suggests that PaC is a productive lens rather than a completed framework: pretraining can supply a representation, prior, or teacher, but the precise calibration object and calibration split remain task-dependent.