Dual-Calibration Answer Reward (DCAR)

• DCAR is a composite reward framework that integrates answer-level confidence with consensus credibility for robust multi-step reasoning.
• It employs dual calibration strategies—step-level and path-level—to mitigate overconfidence and cumulative errors in decision-making systems.
• The approach enhances RLHF and pseudo-labeling by generating calibrated, trustworthy reward signals for improved model performance.

Dual-Calibration Answer Reward (DCAR) is a principled framework for the selection and scoring of candidate answers in reasoning and decision-making systems, particularly those involving LLMs and multi-step inference. DCAR mechanisms provide a calibrated, trustworthy reward signal to guide model training and inference, by simultaneously accounting for both the intrinsic confidence in a generated answer and the credibility or consensus among candidate solutions. This approach is increasingly prominent in reinforcement learning from human feedback (RLHF), self-consistent reasoning, and unlabeled data settings, where robust pseudo-label generation and the mitigation of spurious majority effects are vital.

1. Conceptual Foundations

DCAR is fundamentally a composite reward scheme that merges two distinct calibration criteria: answer-level confidence and population consensus credibility. In contrast to conventional majority voting or simple correctness-based scoring, DCAR evaluates each candidate answer not only based on its popularity (i.e., how frequently it appears among candidate paths or samples), but also by how much intrinsic confidence the model expresses in generating that answer. This dual calibration ensures that selected answers are not merely popular but also supported by sufficiently decisive evidence from the model’s generative process.

Formally, DCAR mechanisms involve the calculation of a confidence score for each candidate, often derived from statistics over the model’s token-level decision distributions (e.g., difference between top-1 and top-2 token probabilities), and the subsequent aggregation of these scores to form a consensus pseudo-label that is further evaluated for credibility.

2. Mathematical Formulations and Scoring Functions

A unified DCAR scoring system for reasoning paths {P_i} can be formalized as follows (Deng et al., 2023):

$$D_{(i)} = \alpha \left(\frac{n_i}{N}\right) + (1-\alpha) \left(\frac{m_i}{M}\right)$$

where:

• $n_i$ is the frequency of final answer $A_i$ among $N$ candidate paths (consensus measure),
• $m_i$ counts the correct intermediate steps in path $P_i$ out of $M$ steps (step-level correctness),
• $\alpha\in[0,1]$ is a hyper-parameter balancing consensus and individual path quality.

The final answer $\hat{A}$ is drawn from the path with maximal $D_{(i)}$:

$i^* = \arg\max_{i\in[1,N]} D_{(i)}$

In unsupervised test-time RL and pseudo-labeling, DCAR is implemented via mechanisms such as COMPASS, which utilize token-level decisiveness for answer confidence (Tang et al., 20 Oct 2025):

• Define per-token confidence score $$PD_{\text{topk}}(x_t) = p(x_t^1 | x_{<t}) - p(x_t^2 | x_{<t})$$
• Sequence-level confidence aggregation: $$c(\hat{y}_i) = \exp(-\operatorname{std}_t[PD_{\text{topk}}(x_t)])$$
• Confidence-weighted consensus score: $$S_\mathrm{ccsc}(y) = \frac{\sum_{i: \hat{y}_i = y} c(\hat{y}_i)}{\sum_{i=1}^N c(\hat{y}_i)}$$
• Credibility normalization: $$S_\mathrm{cred}(y^*) = \frac{\mathcal{C}_\mathrm{General}}{\mathcal{C}_\mathrm{Elite}}$$ where $\mathcal{C}_\mathrm{General}$ is the top confidence within the consensus, and $\mathcal{C}_\mathrm{Elite}$ is the global maximum among all candidates.

The continuous answer-level reward is then:

$$R_\mathrm{answer}(\hat{y}_i) = S_\mathrm{cred}(y^*) \cdot \mathbb{1}\{\hat{y}_i = y^*\}$$

3. Step-Level and Path-Level Calibration Strategies

DCAR subsumes two foundational answer calibration strategies (Deng et al., 2023):

• Step-Level Calibration: Individual reasoning steps (intermediate deductions, chain elaboration) are checked and corrected, either by self-verification or aggregation of correct steps across multiple paths. This approach improves robustness against incoherent or low-quality prompts.
• Path-Level Calibration: The entire reasoning trajectory (chain-of-thought) is treated as an atomic unit and calibrated via mechanisms such as majority vote consensus or self-consistency, maximizing overall output reliability.

The unified DCAR framework combines these by adjusting the $\alpha$ parameter; low $\alpha$ regimes favor step-level calibration (mitigating cumulative error propagation), whereas high $\alpha$ prioritizes global consensus and consistency.

Thresholds for $\alpha$ delineate these domains:

• Step-Level Dominance: $\alpha < \frac{M(N-2)}{N+M(N-2)}$
• Path-Level Dominance: $\alpha >$ lower bound specified by task characteristics.

4. Application in Reinforcement Learning from Human Feedback and Pseudo-Labeling

DCAR is central to modern RLHF pipelines and unsupervised learning settings (Leng et al., 13 Oct 2024, Tang et al., 20 Oct 2025):

• In RLHF, DCAR-inspired reward calibration is achieved by training reward models to explicitly recognize and distinguish genuine model confidence from answer quality, either during reward model training (PPO-M) or via reward calculation (PPO-C), thereby avoiding biases toward overconfident but incorrect outputs.
• In test-time RL (COMPASS), DCAR generates pseudo-labels by integrating intrinsic model confidence and answer credibility, stabilizing learning on unlabeled data streams and mitigating spurious consensus.

These methods result in measurable improvements in calibration, consistency, expected calibration error (ECE), and overall accuracy while maintaining or enhancing model capabilities on diverse benchmarks.

5. Pluralistic Alignment and Ensemble Reward Functions

Recent DCAR frameworks extend answer reward calibration to pluralistic settings, where diverse human preferences must be preserved rather than collapsed into a single scalar reward (Halpern et al., 17 May 2025). By constructing ensembles of reward models (each capturing a distinct annotator or value perspective) and calibrating their mixture weights to match pairwise human preferences, DCAR can faithfully reflect minority viewpoints and maintain pluralism.

Mathematically, for an ensemble $r = (r_{\theta_1}, \dotsc, r_{\theta_k})$ with mixture $\alpha_1, \dotsc, \alpha_k$, the pairwise preference on candidate responses $(y_1, y_2)$ in context $x$ is:

$$\hat{p}(x, y_1, y_2) = \sum_{j=1}^k \alpha_j \cdot \mathbb{1}\{r_{\theta_j}(x, y_1) > r_{\theta_j}(x, y_2)\}$$

DCAR-based policy mixtures derived from such ensembles preserve preference diversity and avoid majority collapse.

6. Practical Implementations and Experimental Results

DCAR and its variants have been empirically validated across multi-step reasoning, mathematical problem-solving, code generation, and commonsense QA (Deng et al., 2023, Leng et al., 13 Oct 2024, Tang et al., 20 Oct 2025, Cui et al., 29 Sep 2025):

• In multi-path CoT setups, tuning $\alpha$ between step-level and path-level dominance yields optimal reasoning accuracy.
• RL pipelines augmented with DCAR-based reward calibration (both in reward models and policy optimization) exhibit reduced overconfidence, improved alignment between expressed and actual answer quality, and better robustness to prompt variation.
• In unsupervised reinforcement learning from unlabeled streams (COMPASS), DCAR stabilizes pseudo-label selection, leading to more reliable model evolution and increases in pass@1 scores and reasoning consistency.

Experimental metrics include consistency, faithfulness, perplexity, ECE, AUC, and accuracy across diverse datasets (e.g., GSM8K, MultiArith, CSQA, MT-Bench, Arena-Hard, AIME, AMC, MATH, GPQA).

7. Limitations and Future Directions

DCAR mechanisms depend on careful tuning of balancing parameters (e.g., $\alpha$ in unified scoring, scaling coefficients in reward adjustment), as well as the reliability of intrinsic confidence signals. In RLHF, sensitivity to hyperparameter settings and calibration dataset design remains a central challenge (Leng et al., 13 Oct 2024). Potential future directions include:

• Extending DCAR to broader domains (e.g., healthcare, legal, multimodal data) where calibrated uncertainty is essential.
• Integrating bounded proper scoring rules (e.g., Brier score as in RLCR) to further align confidence with correctness (Damani et al., 22 Jul 2025).
• Investigating inter-chain and inter-answer consistency in calibration.
• Scaling DCAR to capture richer pluralistic value distributions and user-level control via policy ensembles.

DCAR frameworks represent a rigorous, substantiated approach to answer calibration and reward generation that is broadly applicable to complex reasoning systems, robust policy alignment, and reliable self-supervision.