---
title: Hierarchical Platt Scaling in Automated Code Revision
url: https://www.emergentmind.com/topics/hierarchical-platt-scaling
type: topic
---

# Hierarchical Platt Scaling in Automated Code Revision

Hierarchical Platt scaling is a two-stage post-hoc confidence calibration approach for large language models (LLMs) in automated code revision (ACR) tasks. It addresses the limitations of conventional sequence-level Platt-scaling by incorporating fine-grained, token-level signal extraction combined with local and global calibration layers. This methodology yields calibrated instance-level confidence scores that more faithfully reflect ground-truth correctness, which is critical for downstream tasks such as program repair, vulnerability repair, and code refinement [2604.06723].

## 1. Platt-scaling Formalism

Platt-scaling refers to the application of a logistic regression model to raw model scores to produce calibrated probabilities. Given a score $s$ (such as a logit or model-produced likelihood) for an output $\hat{y}$ and binary ground-truth label $y\in\{0,1\}$, Platt-scaling fits the following transformation:
$$
\hat{p}(y=1 \mid s) = \sigma(A s + B)
$$
where $\sigma(z) = 1/(1+\exp(-z))$, and parameters $A, B \in \mathbb{R}$ are estimated by minimizing regularized negative log-likelihood over a held-out calibration set:
$$
(A, B) = \underset{A,B}{\mathrm{argmin}} \sum_{i=1}^N \left[-y_i \log \sigma(A s_i + B) - (1 - y_i)\log(1 - \sigma(A s_i + B)) \right] + \lambda(A^2 + B^2)
$$
with L2-regularization parameter $\lambda \ge 0$ [2604.06723].

## 2. Fine-grained Confidence Scores

Hierarchical Platt scaling leverages token-level softmax traces from the autoregressive decoding process to compute confidence features, rather than relying solely on sequence-level aggregate scores. Three fine-grained scores are extracted for each predicted sequence:

- **Minimum Token Probability ($s_{\mathrm{min}}$):**
  $$
  s_{\mathrm{min}} = P_{\mathrm{min}}(\hat{y}) = \min_{1 \le t \le T} P(\hat{y}_t \mid \hat{y}_{<t}, X)
  $$
  This feature identifies the least confident token.

- **Lowest-K Token Probability ($s_{\mathrm{lowK}}$):**
  Let $\{ p_{(1)} \le \dots \le p_{(T)} \}$ be sorted token probabilities, $K$ determined via Kneedle elbow detection:
  $$
  s_{\mathrm{lowK}} = P_{\mathrm{low-K}}(\hat{y}) = \frac{1}{K} \sum_{j=1}^K p_{(j)}
  $$

- **Attention-weighted Uncertainty ($s_{\mathrm{attn}}$):**
  Attention rollout is used to propagate downstream attention $w_t$ per token (using Abnar & Zuidema, 2020). Tokens are ranked by $w_t(1-p_t)$, and the top $K$ are used for averaging:
  $$
  s_{\mathrm{attn}} = \frac{1}{K} \sum_{t \in I_K} P(\hat{y}_t \mid \hat{y}_{<t}, X)
  $$
  with $I_K$ the indices of the $K$ largest $w_t(1-p_t)$.

These features enable capture of local uncertainties critical in ACR tasks, where a globally correct-looking sequence may conceal token-level uncertainties indicative of error [2604.06723].

## 3. Hierarchical (Two-stage) Calibration Pipeline

The two-stage calibration pipeline comprises:

### Stage 1: Local Platt-scaling

Each input–output pair $(X, \hat{y})$ is embedded into
$$
\varphi(X,\hat{y}) = [\text{UMAP}_n(\text{embeddings}(X\|\hat{y})); s_j]
$$
where $n=20$, UMAP reduction is applied to concatenated text embeddings of $X$ and $\hat{y}$, and $s_j$ is the scalar score for $j \in \{\mathrm{min}, \mathrm{lowK}, \mathrm{attn}\}$. Embeddings are clustered using HDBSCAN, yielding $K$ clusters.

For each cluster $k$ and score $j$, local sigmoid calibrators are fitted:
$$
p^{(j)}_i = \sigma(A_{j,k}s_{i,j} + B_{j,k}) \quad \text{if } \varphi_i \in \text{cluster } k
$$
Outliers use a default backoff $\psi_j$ (either a global sigmoid or identity).

### Stage 2: Global Platt-scaling over Local Outputs

The locally calibrated probability triplet $p_i = [p^{(\mathrm{min})}_i, p^{(\mathrm{lowK})}_i, p^{(\mathrm{attn})}_i]$ forms the feature vector for a global logistic regression:
$$
\hat{p}_i = \sigma(w_1 p^{(\mathrm{min})}_i + w_2 p^{(\mathrm{lowK})}_i + w_3 p^{(\mathrm{attn})}_i + \beta )
$$
The combined formula is:
$$
\hat{p}_i = \sigma\left(w^\top p_i + \beta\right), \ \text{with} \ p^{(j)}_i = \sigma(A_{j,k(i)}s_{i,j} + B_{j,k(i)})
$$
when $\varphi_i$ falls within cluster $k(i)$.

## 4. Calibration Metrics

Three metrics are utilized to quantify calibration on the test set:

- **Expected Calibration Error (ECE):** Discretizes $\hat{p}_i$ into $B$ equal-width bins. For each bin, $e_b = \text{mean predicted}$, $o_b = \text{fraction correct}$, $P(b) = |\text{bin } b|/N$:
  $$
  \mathrm{ECE} = \sum_{b=1}^B P(b)\,|o_b - e_b|
  $$
- **Brier Score ($\mathcal{B}$):**
  $$
  \mathcal{B} = \frac{1}{N}\sum_{i=1}^N (\hat{p}_i - y_i)^2
  $$
- **Bin Coverage (BC):** Number of bins populated with at least one sample (ideal is $B$).

These metrics collectively assess both global calibration and the discriminative granularity of the confidence estimator [2604.06723].

## 5. Algorithmic Structure and Pseudocode

The pipeline proceeds as follows:

1. For each fine-grained score type $j$:
   - Compute score $s_{i,j}$ for each calibration sample.
   - Compute embedded feature $\varphi_i$.
   - Cluster $\varphi_i$ using HDBSCAN.
   - Fit logistic regression calibrators $(A_{j,k}, B_{j,k})$ for each cluster.
   - Select backoff ${\psi_j}$ for outliers.
2. For each calibration sample:
   - Compute locally calibrated probabilities $p_{i,j}$ using cluster-appropriate or backoff calibrators.
   - Stack $p_i = [p_{i,\mathrm{min}}, p_{i,\mathrm{lowK}}, p_{i,\mathrm{attn}}]$ and fit a global logistic regression $(w, \beta)$.
3. For inference, compute the three fine-grained scores, embed, assign cluster or outlier, calibrate locally, then apply the global logit for final $\hat{p}$.

Pseudocode precisely formalizing these steps is provided in [2604.06723].

## 6. Empirical Results and Recommendations

Extensive evaluation across three ACR benchmarks—DCF-Bug (program repair), DCF-Vul (vulnerability repair), and CR-Trans (code refinement)—using 14 open-source, decoder-only LLMs (Llama-3.1, CodeLlama, Qwen2.5, Qwen2.5-Coder, DeepSeek; 7B–72B params) demonstrates:

- **Sequence-level Platt-scaling** often produces low bin coverage (BC=1–3) and ECE ≳ 0.1, indicating poor calibration granularity.
- **Hierarchical Platt-scaling with fine-grained scores** achieves ECE $0.04–0.08$ and raised BC (≈5–9) for program and vulnerability repair (DCF-Bug, DCF-Vul). Minimum token probability is the most effective feature (lowest ECE, Brier, highest BC).
- **Local Platt-scaling is essential for code refinement (CR-Trans):** It reduces ECE by up to $0.16$ (down to ≈$0.08$) and increases BC by +2–5 bins. For DCF-Bug and DCF-Vul, local gains are smaller but consistent ($\Delta$ECE ≈$0.01–0.07$; $\Delta$BC ≈$+1–2$).
- **Practical guidance:** For DCF-Bug/DC F-Vul, global Platt + minimum token is sufficient for well-calibrated output. Local Platt-scaling should be used for code refinement, or when further ECE reduction is required [2604.06723].

## 7. Significance and Implications

Hierarchical Platt scaling provides a principled and empirically validated framework for confidence calibration in ACR tasks, addressing the shortcomings of global-only approaches. By leveraging local context via token-level features and cluster-specific calibrators, it produces more informative probability outputs. All code, calibrators, and replication scripts are available in the corresponding repository [2604.06723]. This approach enables reliable, instance-level decision-making, facilitating trustworthy integration of LLMs into practical software engineering pipelines.

Source: https://www.emergentmind.com/topics/hierarchical-platt-scaling