---
title: Pretrain-and-Calibrate (PaC) Workflow
url: https://www.emergentmind.com/topics/pretrain-and-calibrate-pac
type: topic
---

# Pretrain-and-Calibrate (PaC) Workflow

Pretrain-and-Calibrate (PaC) is best understood as an umbrella workflow in which a model, representation, or score generator is obtained first and then calibrated before it is used for downstream selection, confidence estimation, or task adaptation. In the literature, this pattern is not standardized under a single formal algorithmic name: several relevant papers explicitly note that they do not introduce a method called PaC, yet they fit the pattern closely. Across these works, the “pretrain” stage may be a higher-capacity target model, a frozen deep probabilistic predictor, a synthetic random-noise pretraining phase, or an original pre-trained language model; the “calibrate” stage may be training-time calibration, post-hoc temperature scaling, threshold calibration for confidence sets, or uncertainty shaping before fine-tuning [2210.06592][2412.17411][2001.00106][2505.16690].

## 1. Conceptual scope and recurring workflow

A common PaC structure appears in several distinct settings. In sample prioritization during training, the workflow is: use a current model or a larger-capacity pre-trained target model, ensure that predictive probabilities are calibrated, and then use those probabilities to prioritize samples by uncertainty [2210.06592]. In uncertainty pre-calibration, the workflow is: initialize a classifier conventionally, pretrain it on random noise with random labels so that initial confidence is pushed toward the chance level, and then fine-tune on the downstream task [2412.17411]. In PAC-valid confidence-set prediction, the workflow is: train a probabilistic predictor, freeze it, apply temperature scaling, and then choose a confidence-set threshold on a held-out calibration split [2001.00106]. In LLM post-training, the workflow is: treat the original pre-trained language model as a source of comparatively trustworthy confidence, and use it to calibrate a post-trained model without labels [2505.16690].

| Instantiation | Pretrained or frozen object | Calibration target |
|---|---|---|
| Sample prioritization | ResNet-34 current model and optional ResNet-50 target [2210.06592] | Entropy scores for subset selection |
| Random-noise pre-calibration | Entire classifier after synthetic pretraining [2412.17411] | Confidence pushed toward chance level |
| PAC confidence sets | Frozen probabilistic predictor [2001.00106] | Temperature and threshold \(T\) |
| DACA for PoLMs | PLM used as calibrator for PoLM [2505.16690] | Temperature \(\tau\) on unlabeled data |

This suggests that PaC is less a single algorithm than a family resemblance. The shared premise is that downstream decisions based on confidence or uncertainty are only as reliable as the scores that drive them. Consequently, pretraining and calibration are coupled: the first stage supplies a predictor or prior, and the second stage makes its outputs usable for ranking, abstention, confidence-set construction, or controlled downstream adaptation.

## 2. Calibration objects and formal definitions

The relevant literature calibrates several different mathematical objects. In classification, confidence is typically the probability of the predicted class,
\[
\hat{y}(x)=\arg\max_k p_\theta(k\mid x), \qquad \text{conf}(x)=\max_k p_\theta(k\mid x),
\]
and a calibrated model should satisfy that among predictions made with confidence \(c\), the empirical accuracy is approximately \(c\) [2412.17411]. Expected Calibration Error is defined using bins \(B_m\) as
\[
\text{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\mathbf{1}(\hat{y}_i=y_i), \qquad
\text{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\text{conf}_i,
\]
\[
\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|\text{acc}(B_m)-\text{conf}(B_m)\right|,
\]
with \(M=10\) bins in the random-noise pretraining study [2412.17411]. The LLM calibration study also uses ECE with \(G=10\) bins, and additionally reports MCE, AECE, and Brier score [2505.16690].

In calibration-driven subset selection, the score being calibrated is not a final confidence threshold but an uncertainty ranking functional. The prioritization criterion is predictive entropy,
\[
\mathbb{H}[y \mid x, D_{train}] := - \sum_c p(y=c \mid x, D_{train}) \log p(y=c \mid x, D_{train}),
\]
and samples with highest entropy are selected for training in subsequent epochs [2210.06592]. The conceptual dependency is explicit: if probabilities are overconfident, entropy-based ranking can be wrong, so calibration directly affects which examples are considered informative.

In PAC confidence sets, calibration is applied not merely to probabilities but to set-valued predictions. Given a calibrated forecaster \(f(y\mid x)\), the confidence set is
\[
C_T(x)=\{y\in\mathcal{Y}\mid f(y\mid x)\ge e^{-T}\}
      =\{y\in\mathcal{Y}\mid -\log f(y\mid x)\le T\},
\]
and its population error is
\[
L(C)=\mathbb{P}_{(x,y)\sim D}[y\notin C(x)].
\]
The threshold \(T\) is then chosen on a held-out validation set so that the resulting confidence set has a finite-sample PAC guarantee [2001.00106].

Temperature scaling appears in two distinct forms. In PAC confidence sets, a frozen predictor is calibrated by
\[
f_{\hat{\phi},\tau}(y\mid x)\propto \exp\!\left(\tau \log f_{\hat{\phi}}(y\mid x)\right), \qquad \tau>0,
\]
with \(\tau\) fitted on a separate calibration split [2001.00106]. In DACA, the calibrated class probabilities are
\[
p(y=i\mid \bm x,\tau)=\sigma_i(f(\bm x)/\tau),
\]
and \(\tau\) is optimized without labels by aligning the post-trained model’s output distribution to that of the PLM on agreement examples [2505.16690].

## 3. Calibration-driven subset selection as a PaC workflow

The most explicit PaC-style training pipeline is the study of calibration and sample prioritization on CIFAR-10 and CIFAR-100. The setup is: split the training data into \(90\%\) train and \(10\%\) validation, train a current model (ResNet-34), use all samples for the first 10 warm-up epochs, then at each subsequent epoch compute predictive entropy for each sample, select the top-\(k\) or top-\(n\%\) most informative examples, and train only on that subset. The calibration methods actually evaluated are Label Smoothing, Mixup, and Focal Loss; the baseline is No Calibration / Cross-Entropy. The target-model variant uses a larger-capacity pre-trained calibrated target model, ResNet-50 trained with Mixup, to compute entropy scores that guide the current model’s training [2210.06592].

The subset sizes are \(n\in\{10,20,30\}\), corresponding to a 90\%, 80\%, and 70\% reduction in the number of examples per epoch after warm-up. The paper’s central empirical claim is that calibration improves sample prioritization, can reduce the number of examples per epoch by at least 70\%, and can thereby speed up the overall training process. The principal metrics are test accuracy and ECE, with additional analyses of validation ECE, class distribution across epochs, and percentage of common samples between consecutive epochs [2210.06592].

The reported results make the calibration effect concrete. On CIFAR-10, the uncalibrated baseline at 100\% data gives 94.1 accuracy and 4.1 ECE, while Mixup at 100\% gives 95.1 accuracy and 2.1 ECE. At 30\% data, the baseline gives 93.6 accuracy and 5.33 ECE, while Mixup gives 94.39 accuracy and 2.67 ECE. On CIFAR-100, the contrast is sharper: the uncalibrated baseline at 30\% gives 73.13 accuracy and 10.77 ECE, whereas Mixup at 30\% gives 73.57 accuracy and 1.49 ECE. The authors summarize the broader pattern by stating that “Mixup consistently has higher test accuracies and low test ECEs across different subset sizes compared to other calibration techniques for both datasets” [2210.06592].

The target-model variant strengthens the PaC interpretation. With a pre-trained calibrated ResNet-50 target, Label Smoothing on CIFAR-10 at 30\% improves from 91.74 accuracy and 3.17 ECE to 93.62 accuracy and 2.93 ECE; on CIFAR-100 at 30\%, it improves from 72.21 accuracy and 3.45 ECE to 76.06 accuracy and 2.28 ECE. For Mixup on CIFAR-100 at 30\%, the target-guided variant improves from 73.57 accuracy and 1.49 ECE to 75.62 accuracy and 0.86 ECE. A plausible implication is that, in this class of methods, pretraining contributes a stronger ranking signal and calibration makes that signal operationally reliable for uncertainty-based selection [2210.06592].

## 4. Pre-calibration by synthetic pretraining

A more radical PaC interpretation appears in the proposal to calibrate uncertainty by changing the pretraining state itself. Here, the claim is that standard random initialization is an underlying cause of miscalibration, leading to excessively high confidence in untrained networks, and that pretraining on random noise with random labels can “pre-calibrate” the network before any real data are seen [2412.17411].

The procedure has two stages. First, the entire classifier is pretrained on synthetic examples with Gaussian noise inputs of shape \(32\times 32\times 3\), mean \(0\), standard deviation \(1\), and random one-hot labels sampled uniformly across classes; both inputs and labels are re-sampled at every iteration. Second, the same network parameters are trained normally on real labeled data, in the main experiments CIFAR-10. The models are feedforward MLPs with hidden width 256, depths 2 through 6, ReLU nonlinearities, batch normalization in every layer, and a final Softmax. Optimization in both stages uses Adam with learning rate between \(10^{-4}\) and \(10^{-5}\), betas \((0.99, 0.999)\), and weight decay \(0.001\) [2412.17411].

The paper’s mechanistic claim is that random initialization induces arbitrary but confident decision structure, whereas random-noise pretraining washes this out and pushes predictions toward the chance-level distribution over classes. For a \(K\)-class problem, the chance-level confidence is \(1/K\); for CIFAR-10, this is \(0.1\). The authors report that loss drops during random-noise pretraining while accuracy stays at chance, and that predictive distributions become flatter and less class-biased. Before downstream training, the random-noise pretrained network, when evaluated on CIFAR-10 or SVHN, produces confidence much closer to chance while accuracy remains at chance [2412.17411].

Empirically, the method reduces confidence–accuracy mismatch and lowers ECE across data sizes and depths. The reported qualitative summary is: lower test loss, higher accuracy, lower ECE on CIFAR-10 downstream classification; significant ECE reduction across all tested training sizes \(500, 1000, 2000, 4000, 8000, 16000, 32000\); significant ECE reduction across all tested depths; and lower OOD confidence with higher AUROC in the CIFAR-10 versus SVHN setting. The paper does not compare against strong post-hoc calibrators, but it provides direct evidence for a PaC view in which calibration is treated as an initialization or pretraining problem rather than an afterthought [2412.17411].

## 5. Post-hoc PaC: confidence sets and label-free LLM calibration

One line of work realizes PaC as a fully post-hoc pipeline. In PAC confidence sets for deep neural networks, the sequence is explicit: train a deep probabilistic predictor on \(Z_{\text{train}}\), freeze it, fit a scalar calibration parameter \(\tau\) on a second split \(Z_{\text{train}'}\), and then choose the smallest threshold \(T\) on a third split \(Z_{\text{val}}\) such that the confidence set satisfies a PAC constraint. The empirical error is
\[
\hat{L}(C;Z_{\text{val}})=\frac{1}{n}\sum_{(x,y)\in Z_{\text{val}}}\mathbb{I}[y\notin C(x)],
\]
and the paper’s main guarantee is that if \(\hat T\) is chosen appropriately, then
\[
\mathbb{P}_{Z_{\text{val}}\sim D^n}\!\left[L(C_{\hat T})>\epsilon\right]<\delta.
\]
This construction is demonstrated on ResNet for ImageNet, a visual object tracking model, and a dynamics model for the half-cheetah reinforcement learning problem. The paper reports that temperature scaling reduces maximum confidence set size noticeably, that the direct finite-sample bound is much tighter than the VC bound, and that incorrectly classified images tend to receive much larger confidence sets than correctly classified ones [2001.00106].

A second post-hoc line uses a pre-trained language model itself as an unsupervised confidence calibrator for a post-trained language model. In this setting, the PLM \(f\) outputs \(\bm p(\bm x)=\sigma(f(\bm x))\), the PoLM \(g\) is calibrated by temperature scaling, and naive confidence alignment would fit
\[
\tau^* = \arg\min_{\tau > 0}
\mathbb{E}_{\bm x\in\mathcal D}\left[\sum_{i=1}^k p_i(\bm x)\log \frac{p_i(\bm x)}{\sigma_i(g(\bm x)/\tau)}\right].
\]
The paper argues that disagreement examples are problematic because PLM confidence tracks the PLM’s own prediction accuracy rather than the PoLM’s. DACA therefore retains only agreement examples,
\[
\arg\max_i f_i(\bm x)=\arg\max_i g_i(\bm x),
\]
and minimizes the same KL divergence with an indicator gate that is zero on disagreements [2505.16690].

This agreement-only filter is motivated both theoretically and empirically. The paper proves that disagreement can induce unavoidable calibration error under naive alignment and gives an extreme condition under which the optimal temperature goes to \(\infty\). DACA is evaluated on MMLU, MedMCQA, MathQA, and TruthfulQA, across open-source and API-based LLMs. The abstract reports that it improves average ECE by up to 15.08\%. Representative results include Gemma-3-12B-Instruct on MMLU, where Vanilla ECE is \(23.679\%\), DACA ECE is \(8.596\%\), and supervised TS gives \(9.746\%\); GPT-4o on MedMCQA, calibrated with Gemma-3-12B as PLM, where Vanilla ECE is \(21.231\%\) and DACA reduces it to \(6.993\%\); and Qwen2.5-32B-Instruct on TruthfulQA, where Vanilla ECE is \(30.955\%\) and DACA reduces it to \(5.244\%\). In this variant of PaC, the pre-trained model functions as a frozen teacher for confidence rather than for labels [2505.16690].

## 6. Related but non-equivalent uses of “PAC”

The PaC literature is complicated by acronym collisions. In fine-tuning pretrained language models, “PAC-tuning” uses PAC-Bayes training to learn parameter-wise noise variances and then applies perturbed gradient descent during downstream adaptation. The paper is explicit that its “PAC” means PAC-Bayes / probably approximately correct, not Pretrain-and-Calibrate, and it does not evaluate ECE or other predictive calibration metrics. Its strongest connection to PaC is a broader one: Stage 1 learns a parameter distribution and Stage 2 adapts the model using the learned variances as perturbation scales [2310.17588].

A similar ambiguity appears in PACTran, which derives transferability metrics for pretrained model selection from PAC-Bayesian bounds. PACTran requires a small labeled target sample \(S\), optimizes a PAC-Bayesian surrogate involving empirical target classification loss and \(D_{KL}(Q\|P)\), and is evaluated as a ranking metric through Kendall-Tau correlation with downstream error. It does not calibrate predictive probabilities; rather, it “calibrates” transferability only in the broad sense of providing a theoretically grounded estimate of which checkpoint will transfer best [2203.05126].

PAC-Net is closer to the PaC theme because its three stages are literally Prune, Allocate, and Calibrate. The method first trains on a source task, prunes by magnitude to identify essential and redundant weights, retrains the unpruned subnetwork on the source task, and then calibrates on the target task by updating only the remaining redundant weights while freezing the source-essential subset. The target adaptation stage is
\[
\mathbf{w}_P(t+1)=\mathbf{w}_P(t)-\eta \hat{\nabla}_{\mathbf{w}_P}[\mathcal{L}_T(\mathbf{w})+\Omega(\mathbf{w}_P)],
\]
with the protected source weights \(\mathbf{w}_U\) held fixed. The paper explicitly characterizes this as preserving source knowledge completely while adjusting the model with target data [2206.05703].

There is also a wholly unrelated usage in coding theory: “PAC codes” means polarization-adjusted convolutional codes, a family of short-blocklength channel codes with rate profiling, convolutional precoding, and polar encoding. The acronym there has no connection to calibration or pretraining in machine learning [2005.13711].

## 7. Limitations, assumptions, and open questions

A central limitation is terminological. Several of the most relevant papers explicitly do not define a method called PaC; the label is interpretive rather than native to the papers themselves. This matters because the literature does not yet offer a single canonical decomposition of what counts as pretraining, what counts as calibration, and whether calibration is applied to probabilities, entropy rankings, confidence sets, or parameter perturbation scales [2210.06592][2412.17411][2310.17588].

The empirical scope is also narrow in different ways across subareas. Calibration-driven sample prioritization is limited to CIFAR-10 and CIFAR-100 with ResNet-34 as current model and ResNet-50 as target model, uses only Max Entropy as the prioritization criterion, does not evaluate post-hoc methods such as temperature scaling, and does not report detailed wall-clock speed benchmarks [2210.06592]. Random-noise pre-calibration is demonstrated mainly for feedforward MLPs in classification, with OOD evaluation focused on CIFAR-10 versus SVHN and no head-to-head comparison against strong calibration baselines [2412.17411]. DACA depends on access to a reasonably calibrated PLM and to option-level probabilities or logprobs, and it reduces sample efficiency by discarding disagreement examples [2505.16690].

Some approaches impose strong data assumptions. PAC confidence sets require i.i.d. train, calibration, and test data from the same distribution \(D\), need separate held-out splits for model fitting, temperature calibration, and threshold calibration, and provide only marginal coverage guarantees rather than conditional guarantees [2001.00106]. PAC-Net assumes the same marginal input distribution \(p(x)\) for source and target, requires substantial parameter redundancy so that pruning leaves enough capacity for target calibration, and relies on magnitude pruning as a heuristic indicator of source-essential weights [2206.05703]. PACTran, despite its relevance to checkpoint selection, requires labeled target data and remains a ranking method rather than a predictive calibration method [2203.05126].

The open design questions are correspondingly heterogeneous. For subset selection, the unresolved issue is whether other prioritization rules beyond Max Entropy benefit similarly from calibration [2210.06592]. For random-noise pre-calibration, the broader question is whether the same effects hold for modern CNNs, transformers, or language models [2412.17411]. For unsupervised PoLM calibration, the critical dependency is the quality and accessibility of the calibrating PLM [2505.16690]. More generally, the existing evidence suggests that PaC is a productive lens rather than a completed framework: pretraining can supply a representation, prior, or teacher, but the precise calibration object and calibration split remain task-dependent.

Source: https://www.emergentmind.com/topics/pretrain-and-calibrate-pac