---
title: Batched Bootstrap CRC Overview
url: https://www.emergentmind.com/topics/batched-bootstrap-crc-bb-crc
type: topic
---

# Batched Bootstrap CRC Overview

Searching arXiv for the BB-CRC paper and closely related CRC literature.
Batched Bootstrap CRC (BB-CRC) is a batched, bootstrapped variant of Conformal Risk Control in which a small held-out set of calibration trajectories is reused through within-batch bootstrap resampling to calibrate a single threshold \(\hat\lambda_Z\) on a deployment-time scalar score \(Q(y)\). In the formulation introduced for LLM control, CRC is a black-box actuator at the API interface: it does not modify the model, but instead decides whether to ship, abstain, regenerate, or escalate based only on the output and a calibrated policy score. BB-CRC is designed for expensive settings in which each calibration trajectory may require multiple LLM calls, embeddings, or judge evaluations, while still preserving CRC-style finite-sample control of expected loss under exchangeability [2509.23007].

## 1. Placement within Conformal Risk Control

In the underlying CRC framework, the controlled object is a **monotone actionable loss**
$$
L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1],\qquad
R(\lambda)=\mathbb{E}\!\left[L(Y_{\mathrm{new}},\lambda)\right].
$$
Here \(Q(y)\in\mathbb{R}\) is a scalar policy score computed from the output only, \(a_\lambda:\mathbb{R}\to[0,1]\) is a gate monotone in \(\lambda\), and \(m_\beta(y)\in[0,1]\) is a calibration-only severity flag encoding the failure mode to be controlled. In the simplest instantiation, the actuator uses the hard-threshold gate
$$
a_\lambda(u)=\mathbf{1}\{u\ge\lambda\}. \tag{4.2}
$$
As \(\lambda\) increases, the policy becomes stricter and fewer outputs are acted upon [2509.23007].

The operational meaning of the risk is explicit: \(R(\lambda)\) is the expected “acted-while-bad” intensity, such as the probability that an output is shipped and is unfactual. CRC chooses \(\hat\lambda\) so that
$$
\mathbb{E}\bigl[L(Y_{\mathrm{new}},\hat\lambda)\bigr]\le \alpha,
$$
with finite-sample guarantees under exchangeability of calibration and deployment data. Once \(\hat\lambda\) has been calibrated, deployment requires no labels online: one computes \(Q(y)\), evaluates \(a_{\hat\lambda}(Q(y))\), and interprets the action as ship or abstain/escalate/regenerate [2509.23007].

BB-CRC preserves this deployment semantics. Its novelty lies not in changing the online decision rule, but in altering how calibration data are reused to estimate risk efficiently and stably when calibration itself is costly.

## 2. Scores, severity functions, and semantic quantification

The framework is agnostic to the specific definition of \(Q(y)\), provided that larger values correspond to more acceptable outputs. One score used in the paper is the **Gram-geometry consensus score** \(Q_E(y)=E(y)\in[0,1]\). Given unit-norm embeddings \(v_i=\psi(y_i)\in\mathbb{R}^d\) collected into \(V\), with Gram matrix \(G=VV^\top\), the interaction energy of item \(i\) is
$$
e(i;G):=\|G_{:,i}\|_2=\|Vv_i\|_2,
$$
and the normalized energy is
$$
E(i):=\frac{e(i;G)}{\sqrt{n}}\in[0,1].
$$
Large \(E(i)\) indicates strong semantic consensus with the rest of the batch, whereas small \(E(i)\) marks an outlier. The paper characterizes this score as label-free, model-agnostic, permutation-invariant, and numerically stable in Gram space [2509.23007].

A second score is the **LLM-as-judge score** \(Q_J(y)=J_{\text{norm}}(y)\in[0,1]\), obtained by normalizing a judge score \(J(y)\in[0,100]\) based on correctness, faithfulness, completeness, and clarity:
$$
J_{\text{norm}}(y)=J(y)/100.
$$
For calibration, the corresponding severity may be expressed as \(\mathrm{JS}(y)=1-J_{\text{norm}}(y)\). More generally, for hallucination control in question answering, the paper defines **Factuality Severity (FS)** as
$$
\mathrm{FS}(a)=1-\max_{r\in R_q}\mathrm{BERTScoreF1}\!\big(\text{head}(a),r\big)\in[0,1]. \tag{2.1}
$$
This severity enters \(m(y)\) during calibration but is not needed at deployment [2509.23007].

These choices show that BB-CRC is score-agnostic at the algorithmic level. It calibrates a threshold on whatever deployment-time score is chosen, while the calibration-only term \(m(y)\) determines which notion of failure is being controlled. This suggests that the method is best understood as a calibration layer sitting above a semantic or judge-derived signal, rather than as a new scoring mechanism in its own right.

## 3. Batched bootstrap construction and guarantee

BB-CRC begins with \(n\) calibration trajectories \(Y_1,\dots,Y_n\), which are partitioned into \(G\) equal-size batches \(B_1,\dots,B_G\), each of size \(I=n/G\). For each batch \(B_g\), the method draws \(K\) bootstrap replicates \(\{\mathbf Z_j^g\}_{j=1}^K\) by sampling with replacement from that batch. The threshold estimator is then defined by
$$
\hat\lambda_Z \gets
\inf\Bigl\{\lambda :
\tfrac{1}{(G+1)K}\sum_{g=1}^{G}\sum_{j=1}^{K}
L(\mathbf Z^{g}_{j},\lambda) + \tfrac{1}{G+1}\le\alpha
\Bigr\}\,\land\,\lambda_{\max}. \tag{4.3}
$$
The additive term \(+1/(G+1)\) is a bias correction analogous to the \(+1/(n+1)\) term in classical conformal methods [2509.23007].

The paper formalizes the bootstrap mechanism through **Lemma 4.1 (Distributional invariance)**. Informally, conditional on the observed bootstrap replicates from the first \(G\) batches, both a new outcome \(Y_{\mathrm{new}}\) and a bootstrap replicate from an unused \((G+1)\)-st batch behave as draws from the same distribution \(\mathbb P_Y\). This is the bootstrap analog of the exchangeability argument used in standard CRC [2509.23007].

Under this setup, **Theorem 4.2 (Finite-sample BB-CRC)** states that if \(\{B_g\}_{g=1}^{G+1}\) are i.i.d., the items within each batch are exchangeable, \(L\) is right-continuous in \(\lambda\) and bounded in \([0,1]\), and \(L(\cdot,\lambda_{\max})\le\alpha\), then
$$
\mathbb{E}\bigl[L(Y_{\mathrm{new}},\hat\lambda_Z)\bigr]\le\alpha.
$$
The significance is precise: BB-CRC is not presented as a heuristic approximation to CRC, but as a provably valid variant that replaces many independent calibration evaluations with a blockwise bootstrap estimator while retaining the same finite-sample risk-control target [2509.23007].

## 4. Computational motivation and calibration workflow

The motivation for BB-CRC is explicitly computational. In the LLM setting, a single calibration instance may involve generating several responses, running a judge, and computing embeddings and Gram matrices. Naïve CRC requires repeating that heavy pipeline for each calibration point, and calibration may need to be repeated under drift. BB-CRC addresses this by collecting only \(n=G\times I\) real trajectories and then recycling each batch through \(K\) internal bootstrap replicates, with no further LLM calls [2509.23007].

The practical workflow is correspondingly simple. One chooses \(G\), \(K\), \(\alpha\), and \(\lambda_{\max}\), partitions the calibration set into \(G\) batches, generates the within-batch resamples, computes \(L(\mathbf Z_j^g,\lambda)\) as needed, and solves (4.3) for \(\hat\lambda_Z\). The calibrated threshold can then be stored with dataset, model, and date metadata and reused in a “calibrate once, deploy often” regime. At deployment, only the score \(Q(y)\) is computed; if \(Q(y)\ge \hat\lambda_Z\), the system ships the output, and otherwise abstains, regenerates, or escalates [2509.23007].

This reuse of calibration data is the central engineering contribution of BB-CRC. A plausible implication is that its value is greatest when marginal calibration calls are far more expensive than the combinatorial work of resampling and re-aggregating losses. The paper states this directly: the compute overhead of bootstrapping is negligible relative to LLM calls [2509.23007].

## 5. Relation to RBWA-CRC and observed empirical behavior

The paper situates BB-CRC alongside **Randomized Batched Weighted-Average CRC (RBWA-CRC)**, which replaces explicit bootstrap resampling with random simplex weights. For batch \(g\) of size \(I\), RBWA samples
$$
p_g=(p_{g,1},\dots,p_{g,I})\sim\mathcal P_{\mathcal S},
$$
defines
$$
L_g(\lambda)=\sum_{i=1}^I p_{g,i}L(Y_{g,i},\lambda),
$$
and calibrates
$$
\hat\lambda_p \gets \Bigl(\inf\Bigl\{\lambda: \frac{1}{G+1}\sum_{g=1}^{G} L_g(\lambda)+\frac{1}{G+1}\le\alpha\Bigr\}\Bigr)\land \lambda_{\max}. \tag{4.4}
$$
BB-CRC appears as a special case: if the simplex law \(\mathcal P_{\mathcal S}\) is chosen to match empirical bootstrap weights, RBWA reproduces BB-CRC’s resampling scheme [2509.23007].

This relation matters because the paper attributes different strengths to the two methods. BB-CRC’s main contribution is data and compute efficiency with valid guarantees, whereas RBWA-CRC is introduced to improve variance reduction and anti-concentration. With Dirichlet\((\eta\mathbf 1)\) weights and \(\kappa=I\eta\), Theorem 4.4 gives a variance dial,
$$
\mathrm{Var}\!\big(L_g(\lambda)\mid \ell\big)=\frac{\mathrm{Var}_{\mathrm{emp}}(\ell_g(\lambda))}{\kappa+1},
$$
and shows that if the within-batch losses are not all equal, \(L_g(\lambda)\) has no atoms. The paper uses this to explain why RBWA produces smoother calibration curves and lower threshold variance [2509.23007].

Empirically, the framework is assessed on ASQA, NQ-Open, HotpotQA, and AmbigQA, with additional decoding-entropy and vendor/model-swap ablations. In the calibration comparison, RBWA tracks the line \(y=\alpha\) most closely, while both BB-CRC and standard CRC are more conservative, with empirical risk below \(\alpha\). Threshold-stability examples make the distinction concrete: at \(\alpha=0.15\), the reported standard errors are \(7.46\times 10^{-4}\) for BB-CRC, \(5.79\times 10^{-4}\) for CRC, and \(2.89\times 10^{-4}\) for RBWA; at \(\alpha=0.05\), they are \(1.299\times 10^{-3}\), \(1.321\times 10^{-3}\), and \(4.61\times 10^{-4}\), respectively. On the task-level metric of FS reduction, the summary values across risk budgets show that G-Eval-CRC, which uses a BB-CRC threshold, markedly outperforms G-Eval-Naive: for example, at \(\alpha=0.20\), FS reduction rises from \(9.0\) to \(46.5\), while Gram-CRC reaches \(86.0\) [2509.23007].

## 6. Limitations, failure modes, and scope

The limitations described for the overall framework apply directly to BB-CRC. First, all guarantees assume exchangeability between calibration and deployment data. Covariate shift, prompt drift, and temporal dependence therefore threaten the validity of the stated bound. Second, CRC controls the loss defined through \(Q(y)\) and \(m(y)\), not an abstract notion of quality: if the severity flag is misspecified or the score is poorly aligned with that severity, the method may remain formally valid while being practically unhelpful, for example by producing a threshold that ships almost nothing [2509.23007].

Third, BB-CRC does not address model drift by itself. If the relation between \(Q\) and \(m\) changes after calibration, the guarantee is with respect to the old distribution and recalibration may be necessary. Fourth, when judge-based policies are used, performance depends on the rubric and on the judge’s robustness to adversarial prompts or position bias. BB-CRC can quantify and control the resulting risk in the CRC sense, but it does not repair deficiencies in the judge or the rubric [2509.23007].

Within those limits, BB-CRC occupies a specific place in the taxonomy of LLM control methods. It is not a new semantic score, not a modification of the base model, and not a replacement for CRC’s theoretical core. It is a compute-aware calibration mechanism for expensive pipelines that reuse batches through bootstrap replicates, preserve the standard deployment interface of a single threshold on \(Q(y)\), and maintain finite-sample expected-risk control under the same exchangeability assumptions as canonical CRC.

Source: https://www.emergentmind.com/topics/batched-bootstrap-crc-bb-crc