Randomized Batched Weighted-Average CRC

• The paper introduces RBWA-CRC as a compute-aware refinement of CRC and BB-CRC, achieving threshold stability and finite-sample risk control through randomized convex weighting.
• RBWA-CRC employs batched calibration with Dirichlet weights to smooth the empirical risk curve and provides an explicit variance dial to reduce calibration instability.
• The method enables label-free, API-level calibration in LLM pipelines, improving factuality and reliability by enforcing a user-specified risk budget with minimal extra computation.

Randomized Batched Weighted-Average CRC (RBWA-CRC) is a conformal calibration procedure for controlling deployment-time risk in LLM pipelines through a threshold on a scalar policy score at the API boundary. In "Taming Variability: Randomized and Bootstrapped Conformal Risk Control for LLMs" (Pang et al., 27 Sep 2025), RBWA-CRC is presented as a compute-aware refinement of Conformal Risk Control (CRC) and Batched Bootstrap CRC (BB-CRC): it preserves finite-sample marginal validity, replaces discrete bootstrap resampling with random simplex weights, smooths the empirical risk curve, and stabilizes the calibrated threshold while maintaining the CRC objective of enforcing a user-specified risk budget $\alpha$.

1. Formal setting and actuator semantics

RBWA-CRC operates in the CRC formulation where the basic object is a trajectory $Y$ of an LLM pipeline, including prompt, answer, embeddings, and judge score. At runtime, only a scalar policy score $Q(y)\in[0,1]$ is used. The deployment rule is mediated by a thresholded gate $a_\lambda$, and the calibration target is an actionable loss

$$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$

In the experimental instantiation,

$a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$

so the system acts when the score is sufficiently high. The quantity $m_\beta(y)$ is calibration-only and encodes the risk to be controlled, such as Factuality Severity (FS) or Judge Severity (JS); it is not used at deployment (Pang et al., 27 Sep 2025).

The operational objective is to choose a data-dependent threshold $\hat\lambda$ such that

$$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$

under standard exchangeability assumptions. Deployment then reduces to a scalar thresholding rule: ship, trust, or escalate if $Q(y)\ge\hat\lambda$, and abstain, reject, or fall back otherwise. A central feature of the construction is that it is implemented entirely at the API layer, using only $Y$0; no logits or internal LLM states are needed.

This formulation makes RBWA-CRC label-free at inference time and model-agnostic with respect to the underlying LLM. A plausible implication is that the method is best understood not as an intrinsic model modification, but as an external control layer that translates output variability into a calibrated acceptance policy.

2. Motivation and construction of the randomized batched weighted average

The motivation for RBWA-CRC arises from two deficiencies of naïve split-CRC in LLM settings: calibration compute cost and finite-sample instability. Each calibration trajectory may require multiple LLM calls, and the empirical risk curve $Y$1 is piecewise-constant and jagged because the losses are in $Y$2 or small rational values. Small perturbations in the calibration data can therefore move the selected threshold substantially, which is undesirable when small configuration changes or random seeds can flip which responses are shipped (Pang et al., 27 Sep 2025).

RBWA-CRC is defined on a batched calibration set. If $Y$3 calibration trajectories are partitioned as

$Y$4

then batch $Y$5 is

$Y$6

The batches $Y$7 are assumed i.i.d., and the elements within each batch are exchangeable. Instead of bootstrap replicates, RBWA-CRC samples one random weight vector per batch from a distribution $Y$8 on the simplex

$Y$9

The main instantiation is the symmetric Dirichlet law

$Q(y)\in[0,1]$0

Given $Q(y)\in[0,1]$1, the weighted batch loss is

$Q(y)\in[0,1]$2

The calibration statistic is

$Q(y)\in[0,1]$3

where the $Q(y)\in[0,1]$4 term is the usual CRC bias correction. The threshold returned by Algorithm 4.2 is

$Q(y)\in[0,1]$5

The defining idea is that RBWA-CRC averages per-batch weighted losses rather than raw per-item losses or bootstrap-averaged losses. This produces a randomized convex combination of intra-batch losses and thereby a smoothed empirical risk estimate. The method is also strictly more general than BB-CRC: if $Q(y)\in[0,1]$6 is chosen to be the law of multinomial bootstrap frequencies, then RBWA-CRC exactly reproduces the BB-CRC resampling scheme.

3. Statistical properties: validity, moments, and anti-concentration

Theorem 4.3 gives the central guarantee. Under i.i.d. batches, within-batch exchangeability, bounded loss $Q(y)\in[0,1]$7, right-continuity and monotonicity of $Q(y)\in[0,1]$8 in $Q(y)\in[0,1]$9, and the technical condition $a_\lambda$0, the threshold $a_\lambda$1 returned by RBWA-CRC satisfies

$a_\lambda$2

This is a finite-sample marginal risk control statement over the joint randomness of the data and the random weights (Pang et al., 27 Sep 2025).

Theorem 4.1 characterizes the moments of the weighted batch loss under Dirichlet weights. Writing $a_\lambda$3, the batch mean loss is

$a_\lambda$4

and the empirical variance is

$a_\lambda$5

For $a_\lambda$6 with $a_\lambda$7, RBWA satisfies

$a_\lambda$8

and

$a_\lambda$9

The parameter $$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$0 therefore acts as an explicit variance dial: larger Dirichlet precision implies smaller conditional variance. The same theorem also states an anti-concentration property: if the $$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$1 are not all equal, then $$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$2 has a continuous distribution with no atoms. In the terminology of the paper, this removes the lattice structure that appears under bootstrap counts or pure averages of binary losses.

Theorem 4.2 adds an asymptotic calibration CLT for

$$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$3

Under boundedness and i.i.d. batch assumptions, and with $$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$4 approximately constant across folds, as $$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$5,

$$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$6

with

$$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$7

This supports the interpretation of RBWA as a smoothed estimator of the risk curve and motivates the operational rule stated in the source: choose the smallest $$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$8 whose upper CLT confidence band for $$L(y,\lambda)=a_\lambda\!\big(Q(y)\big)\cdot m_\beta(y)\in[0,1].$$9, together with the $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$0 correction, lies below $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$1.

4. Computational role and threshold stability

RBWA-CRC is designed to reduce calibration calls relative to naïve CRC. The calibration set of size $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$2 is collected once, and the number of LLM calls is proportional to $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$3, not to the number of $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$4-evaluations or resampling iterations. In contrast to BB-CRC, which bootstraps within each batch, RBWA-CRC samples a single random weight vector and computes one weighted sum per batch. The added computational cost is therefore essentially a vector-matrix dot per batch (Pang et al., 27 Sep 2025).

The empirical motivation for this design is threshold stability. The source quantifies threshold stability as the standard error of the calibrated $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$5 across multiple calibration runs, such as across random folds or random seeds. In Fig. 4.1(b), RBWA achieves the lowest $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$6 across $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$7, while BB-CRC improves over standard CRC but not as much as RBWA. In Fig. 4.1(a) and the calibration summary table, all three methods control risk below the target $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$8, but RBWA’s empirical risk curve tracks $a_\lambda(u)=\mathbf{1}\{u\ge \lambda\},$9 most closely, meaning that the constraint is more active.

The calibration summary reported for $m_\beta(y)$0 makes the pattern concrete. The empirical risk is approximately $m_\beta(y)$1 for CRC, approximately $m_\beta(y)$2 for BB-CRC, and approximately $m_\beta(y)$3 for RBWA, which is much closer to the target $m_\beta(y)$4. The standard error of the threshold is $m_\beta(y)$5 for CRC, $m_\beta(y)$6 for BB-CRC, and $m_\beta(y)$7 for RBWA. These observations are consistent with the theoretical picture: unbiasedness keeps the estimator centered, while anti-concentration reduces ties and dampens threshold volatility.

A common misunderstanding is to equate smoothing with a loss of validity. In the RBWA construction, smoothing is achieved through randomized convex weighting, but the finite-sample guarantee remains exact in the sense of Theorem 4.3 rather than asymptotic or heuristic.

5. Deployment logic and empirical uses in LLM pipelines

At deployment, RBWA-CRC changes only the calibrated threshold and not the form of the decision rule. The actuator is

$m_\beta(y)$8

In the reported experiments, acting is interpreted as ship, trust, or route through, while the complement is abstain, escalate, or regenerate. The same thresholding logic applies whether $m_\beta(y)$9 is the Gram-energy score $\hat\lambda$0 or the judge score $\hat\lambda$1; RBWA-CRC is agnostic to the semantics of the risk metric $\hat\lambda$2 beyond boundedness and monotonicity of the induced loss (Pang et al., 27 Sep 2025).

The empirical uses emphasized in the source are hallucination mitigation and LLM-as-judge reliability. For hallucination control, the risk flag can be FS; for judge reliability, it can be JS. The framework is assessed using four datasets, and the detailed experimental discussion further reports results across six QA settings: ASQA, NQ-Open, HotpotQA, AmbigQA, plus entropy and vendor-swap ablations. Across these settings, shipped answers are consistently more factual than unshipped ones, and the acceptance and rejection regions remain stable under entropy stress tests and vendor or model swaps.

RBWA-CRC is used as the main calibrator for the Gram-CRC mode with policy $\hat\lambda$3. The reported findings are: all CRC variants keep the empirical acted-while-bad risk at or below the desired $\hat\lambda$4; RBWA is closest to the target; and RBWA yields significantly smaller variability of $\hat\lambda$5 across calibration runs. For aggregated factuality reduction, Gram-CRC with RBWA reports FS reduction from $\hat\lambda$6 at $\hat\lambda$7 down to $\hat\lambda$8 at $\hat\lambda$9. Judge-CRC with BB-CRC and policy $$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$0 reports FS reduction from $$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$1 at $$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$2 down to $$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$3 at $$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$4. Judge-Naive with fixed thresholds and no CRC reports FS reduction around $$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$5–$$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$6 across pseudo-$$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$7 settings. This suggests that calibration itself is a major source of performance gain, while the combination of Gram-energy scoring and RBWA-CRC yields the most uniform factuality gains across tasks and budgets.

6. Relation to CRC variants, practical tuning, and limitations

RBWA-CRC belongs to a family of CRC-style procedures that differ only in how empirical risk is estimated. Standard CRC uses direct per-item averaging,

$$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$8

BB-CRC uses batched bootstrap averaging,

$$\mathbb{E}\big[ L(Y_{\text{new},\hat\lambda})\big]\le\alpha,$$9

and RBWA-CRC uses weighted batch losses,

$Q(y)\ge\hat\lambda$0

All select

$Q(y)\ge\hat\lambda$1

Within this taxonomy, standard CRC is the most conservative and variable in the described LLM setting; BB-CRC reduces calls and stabilizes thresholds to some extent; RBWA-CRC generalizes BB-CRC, adds theoretical smoothing and explicit variance control, and provides better threshold stability (Pang et al., 27 Sep 2025).

The practical guidance in the source is correspondingly specific. Batching parameters $Q(y)\ge\hat\lambda$2 and $Q(y)\ge\hat\lambda$3 should be chosen to match the calibration budget; for a few thousand calibration trajectories, the discussion suggests $Q(y)\ge\hat\lambda$4 in the tens and $Q(y)\ge\hat\lambda$5 in the tens. Larger $Q(y)\ge\hat\lambda$6 improves CLT accuracy and reduces the variance of $Q(y)\ge\hat\lambda$7. The default weight law is symmetric Dirichlet, and $Q(y)\ge\hat\lambda$8 controls the smoothing level: small $Q(y)\ge\hat\lambda$9 gives more noise and more anti-concentration, while large $Y$00 moves the weights toward $Y$01 and reduces the variance of each $Y$02. The recommended usage is RBWA-CRC when stable thresholds and empirical risk near $Y$03 are desired, and BB-CRC when a bootstrap-style discrete resampling scheme is specifically preferred. The deployment pattern is "calibrate once, deploy often."

The limitations are those of the broader framework. The theoretical guarantees rely on i.i.d. batches and within-batch exchangeability, whereas practical LLM systems may exhibit covariate shift, temporal dependence, prompt drift, and feedback loops. Performance also depends on the quality of the scalar score $Y$04: a poorly aligned judge or embedding geometry can limit usefulness even when the formal risk guarantee remains intact. Calibration still requires a reasonably sized set with ground-truth-based $Y$05, which can be expensive. The source also identifies open directions in multi-judge ensembling, adversarial prompting for judges, richer evaluation tasks such as pairwise ranking, critique grading, and safety moderation, and extensions from expected-loss control to tail-risk variants such as CVaR-style objectives.

In this formulation, RBWA-CRC is best characterized as a randomized, batched, conformal threshold calibrator that preserves finite-sample risk validity while reducing calibration cost and threshold instability. Its distinctive contribution is the replacement of discrete resampling by random convex weighting, yielding unbiased smoothing, anti-concentration, and a tunable variance mechanism within the CRC framework.