Self-Rationale Calibration (SRC)

• Self-Rationale Calibration (SRC) is an iterative post-training paradigm that optimizes model alignment by using self-generated rationale–output pairs as supervision signals.
• It employs methodologies like Direct Preference Optimization to fine-tune models based on pairwise comparisons of candidate rationales, boosting scoring accuracy and answer consistency.
• The framework has demonstrated significant performance improvements on benchmarks such as Reward Bench and MMStar by converting rationale diversity into effective calibration signals.

Searching arXiv for the two SRC papers to ground the article in the current literature. Self-Rationale Calibration (SRC) denotes an iterative post-training paradigm in which a model improves the alignment between its generated rationales and its final judgments or answers by learning from preference relations constructed over its own outputs. The term is used in two closely related but distinct settings. In "Self-rationalization improves LLM as a fine-grained judge" (Trivedi et al., 2024), SRC refers to a self-rationalization procedure for LLM-as-a-judge models that generates multiple rationale-and-score judgments, curates preference pairs from those self-judgments, and fine-tunes the judge with Direct Preference Optimization (DPO). In "Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration" (Wu et al., 17 Sep 2025), SRC refers to an iterative framework for Large Vision-LLMs (LVLMs) that calibrates rationale–answer alignment through rationale-first fine-tuning, diverse candidate search, pairwise scoring, confidence-weighted preference curation, and DPO-style preference fine-tuning. Across both formulations, the central premise is that rationales are not merely explanatory by-products, but training signals that can calibrate downstream scoring or answering behavior.

1. Conceptual scope and problem setting

In the LLM-as-a-judge formulation, SRC is introduced for fine-grained customizable scoring criteria, specifically Likert-scale scoring with arbitrary evaluation criteria (Trivedi et al., 2024). The judge model produces a rationale $r$ and a score $s$, with the score generated after the rationale so that scoring is explicitly conditioned on reasoning. The method is defined over both pointwise and pairwise evaluation modes. Pointwise evaluation is formalized as

$J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$

where $c$ is context, $a$ is the response, $e$ is the scoring criterion, and $s \in \{1,\dots,5\}$. Pairwise evaluation is formalized as

$$J(c, a_1, a_2, e, i_{\text{pair}}; \theta) \to (p, r),$$

where $p \in \{1,2\}$ is the preferred response index (Trivedi et al., 2024).

In the LVLM formulation, SRC addresses rationale–answer misalignment in visual question answering and related multimodal tasks. For an input $x=(I,q)$ with image $s$0 and question $s$1, the model generates a candidate response $s$2, where $s$3 is a textual rationale and $s$4 is a textual answer. The stated objective is to calibrate the LVLM so that the generated rationale is factually grounded and supports the answer, thereby improving both perception and reasoning (Wu et al., 17 Sep 2025).

The common conceptual structure is that SRC treats rationale generation as part of the decision process rather than as post hoc explanation. This suggests a broader interpretation of calibration: not probabilistic calibration in the narrow sense, but alignment between internalized reasoning structure and externally evaluated outputs. In the judge setting, this alignment concerns rubric-conformant scores; in the LVLM setting, it concerns factual and visual support for answers.

2. Core mechanisms of SRC

The judge-oriented SRC pipeline begins with a supervised fine-tuned seed judge, denoted $s$5, constructed by training pointwise and pairwise SFT models and merging weights (Trivedi et al., 2024). For each input $s$6, the current judge $s$7 samples $s$8 judgments, each of the form $s$9, under a "rationale then score" decoding order. Preference pairs are then constructed from these self-generated judgments, and DPO fine-tuning produces the next judge $J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$0. The reported setup uses $J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$1, temperature $J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$2, two iterations, and training subsets of 5k examples in Iter 1 and 500 examples in Iter 2; a single-stage DPO baseline with 8k samples is also reported (Trivedi et al., 2024).

The LVLM SRC pipeline is likewise iterative but introduces additional components tailored to multimodal reasoning (Wu et al., 17 Sep 2025). It comprises four stages per iteration: rationale fine-tuning, candidate response search, pairwise scoring via a tailored scoring model called R-Scorer, and confidence-weighted preference curation followed by DPO-style preference fine-tuning. Rationale fine-tuning uses lightweight LoRA supervised fine-tuning to induce the model to output "Thinking: [rationale] Output: [answer]" natively, without explicit prompts at inference time. Candidate responses are then generated through sentence-level beam search, ranked for distinctiveness, scored pairwise, and reduced to preferred and non-preferred responses for preference learning (Wu et al., 17 Sep 2025).

Despite their architectural differences, both versions of SRC share three structural commitments. First, the model must generate multiple candidate rationale-conditioned outputs for the same input. Second, preference information is derived from these outputs rather than collected as external human preference data. Third, the calibrated model is obtained by preference optimization over complete rationale-output objects, not over scores or answers alone. A plausible implication is that SRC should be understood less as a single algorithm than as a design pattern for rationale-conditioned alignment.

3. Preference construction and scoring strategies

In the LLM-as-a-judge setting, preference pairs are constructed at the combined judgment level, meaning rationale plus score rather than score-only (Trivedi et al., 2024). Three curation strategies are described.

Correct-Answer Preference Pairing identifies judgments whose score matches ground truth and designates them as chosen, while mismatched judgments become rejected. Optional score margins of at least 1 or at least 2 are used to ensure a clearer preference signal (Trivedi et al., 2024).

Meta-Judge uses an LLM-as-meta-judge to rate or rank judgments based on overall judgment quality, including both score correctness and rationale quality. The paper notes that the seed judge was biased toward high scores and struggled to meta-judge reliably (Trivedi et al., 2024).

Self-consistency or majority voting treats the majority score across self-judgments as chosen and minority-score judgments as rejected. This is easy to implement but can be noisy when consistency is misaligned with ground truth (Trivedi et al., 2024).

The LVLM setting replaces these heuristics with an explicit pairwise scoring model, R-Scorer (Wu et al., 17 Sep 2025). R-Scorer is built by LoRA fine-tuning Qwen-2.5-1.5B on 21k pairwise examples and outputs a scalar score $J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$3 indicating the relative superiority of candidate $J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$4 over candidate $J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$5. It takes as input the question and reference answer, two candidate responses containing "Thinking" and "Output", and factual checks that probe visual claims made in the rationales. Bidirectional pairwise scoring is used to mitigate order sensitivity (Wu et al., 17 Sep 2025).

The LVLM formulation further defines winning-score aggregation and confidence-weighted ranking. Given a pairwise score matrix $J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$6, the sum-based winning score is

$J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$7

and the count-based alternative is

$J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$8

These scores are rank-normalized and combined with the model's generation confidence $J(c, a, e, i_{\text{point}}; \theta) \to (s, r),$9 through

$c$0

where $c$1 controls the emphasis on scorer versus confidence (Wu et al., 17 Sep 2025). Preferred and non-preferred responses are selected as the candidates with maximal and minimal $c$2.

These two formulations differ in how preference quality is operationalized. The judge formulation relies primarily on ground-truth score agreement, margins, or proxy aggregation over self-judgments. The LVLM formulation inserts an externalized comparative scorer and explicit factual checks. This suggests that SRC can absorb varying degrees of auxiliary supervision while preserving its self-calibration character.

4. Optimization objectives and training dynamics

The judge formulation uses standard DPO over preference pairs $c$3, where $c$4 is the preferred judgment and $c$5 the rejected judgment (Trivedi et al., 2024). The reported loss is

$c$6

where $c$7 is the current policy, $c$8 is the reference model, and the SFT judge $c$9 serves as the natural reference model (Trivedi et al., 2024). The paper does not report optimizer, batch size, learning rate, number of steps, or the exact $a$0 value.

The LVLM formulation also uses a DPO-style objective, but instantiates the preference ratio explicitly against the previous iteration's model (Wu et al., 17 Sep 2025):

$a$1

with $a$2 in experiments, together with a small SFT loss of weight 0.02 during preference fine-tuning (Wu et al., 17 Sep 2025). Here the probabilities are computed over the concatenated "Thinking" and "Output" tokens, so the optimization target directly couples rationales and answers.

The judge paper emphasizes that training with rationales via DPO significantly improves judging, whereas training on long rationales via SFT or RLHF alone can dilute the training signal (Trivedi et al., 2024). It also reports that prompting the final model to output only scores, without rationales, reduces performance. The LVLM paper similarly distinguishes its rationale-oriented alignment from prompt-based chain-of-thought control, stating that prompting for rationales yields limited or mixed benefits and that the full SRC pipeline with rationale SFT delivers the largest improvements (Wu et al., 17 Sep 2025).

Taken together, these results frame DPO not as a generic replacement for SFT, but as a mechanism for converting rationale diversity into contrastive supervision. This suggests that the efficacy of SRC depends on the quality and separability of candidate rationale-output pairs more than on rationale generation alone.

5. Empirical performance and ablations

The judge-oriented SRC reports strong benchmark performance for the final model $a$3 after two DPO iterations (Trivedi et al., 2024). On Reward Bench Total, $a$4 reaches 0.76, compared with Prometheus-2 7B at 0.72 and Prometheus-2 8x7B at 0.74. Its per-category Reward Bench scores are Chat 0.88, Chat-Hard 0.56, Safety 0.86, and Reasoning 0.74. On BiGGen Bench, it achieves Human Pearson 0.52 and GPT-4 Pearson 0.65, outperforming Prometheus-2 7B at 0.50 and 0.62, Auto-J-13B at 0.30 and 0.38, and Prometheus-2-BGB 8x7B at 0.44 and 0.55. On Feedback Bench, it reaches GPT-4 Pearson 0.93, compared with Prometheus-2 7B at 0.88 and Prometheus-2 8x7B at 0.84 (Trivedi et al., 2024).

The same paper reports that a single-stage DPO baseline with 8k samples already improves substantially over SFT, reaching Reward Bench Total 0.73, BiGGen Human 0.50, and BiGGen GPT-4 0.63, but the iterative SRC procedure performs better (Trivedi et al., 2024). The abstract states improvements of 3% to 9% over models trained using SFT with rationale, self-consistency, or best-of-$a$5 sampling. Concrete values reported in the details include Reward Bench Total improving from 0.68 for SFT and 0.68 for self-consistency to 0.76 for $a$6, while best-of-$a$7 attains 0.67 (Trivedi et al., 2024).

Human evaluation in the judge paper focuses specifically on rationale quality. Three independent annotators conducted blind side-by-side comparisons on held-out items filtered to cases where both models assigned the same score, so that only rationale quality was judged (Trivedi et al., 2024). The abstract reports an average win rate of 62% for the SRC-calibrated judge against the SFT model. A more detailed figure reports average win rate 55% versus $a$8 and 69% versus Best-of-$a$9, averaged over three annotators on BigGen and Feedback Bench (Trivedi et al., 2024).

The LVLM formulation reports improvements on MMStar and several VQA-related benchmarks (Wu et al., 17 Sep 2025). For LLaVA-1.5-7B, baseline MMStar overall is 32.1 and SRC improves it to 38.9, with Fine-grained Perception improving from 25.2 to 33.2, Math from 24.0 to 33.2, and Science & Technology from 20.4 to 28.8. ChartQA VAL improves from 17.8 to 24.4, LLaVABench from 61.8 to 65.5, and HallusionBench from 24.7 to 27.3. For LLaVA-Next-8B, MMStar overall improves from 42.6 to 47.6, Coarse Perception from 63.6 to 69.2, Fine-grained Perception from 41.2 to 45.6, and Math from 29.6 to 39.6. ChartQA VAL improves from 68.7 to 71.4, LLaVABench from 64.1 to 67.4, and HallusionBench from 32.1 to 37.5 (Wu et al., 17 Sep 2025).

A compact comparison of representative reported results is given below.

Setting	Baseline	SRC result
Reward Bench Total, judge model	0.68 (SFT)	0.76
BiGGen GPT-4 Pearson, judge model	0.60 (SFT)	0.65
Feedback Bench GPT-4 Pearson, judge model	0.86 (SFT)	0.93
MMStar overall, LLaVA-1.5-7B	32.1	38.9
MMStar overall, LLaVA-Next-8B	42.6	47.6

Ablation findings are central to both papers. In the judge setting, ground-truth matching with higher margin threshold performs best: margin $e$0 yields Reward Bench Total 0.75 versus 0.73 for margin $e$1, and BiGGen GPT-4 Pearson 0.65 versus 0.57 (Trivedi et al., 2024). Self-consistency underperforms margin-based selection, and meta-judge is worse than margin-based approaches because the base judge is biased toward higher scores (Trivedi et al., 2024). In the LVLM setting, sum-based winning score outperforms count-based scoring, factual checks substantially boost Fine-grained Perception, and increasing $e$2 improves overall performance, with notable gains in math (Wu et al., 17 Sep 2025). Semantic entropy decreases over iterations while MMStar performance improves, which the paper interprets as greater consistency and reduced uncertainty (Wu et al., 17 Sep 2025).

6. Relation to neighboring methods

The judge paper positions SRC against several alternative alignment and inference strategies (Trivedi et al., 2024). SFT with rationales provides only positive examples and lacks explicit negative signals, whereas SRC adds contrastive learning through DPO over preferred and rejected judgments. Self-consistency and best-of-$e$3 improve inference-time robustness by sampling multiple outputs and aggregating or selecting among them, but they do not explicitly train the model to distinguish better from worse rationales or scores. RLHF and RLAIF require external human reward models, human data, or auxiliary judges, whereas SRC is described as self-contained because it uses the judge's own generations to curate preference pairs before applying DPO (Trivedi et al., 2024).

The LVLM paper draws a related distinction from chain-of-thought prompting and multimodal alignment approaches (Wu et al., 17 Sep 2025). It argues that chain-of-thought prompting often depends on explicit prompts or distillation from expert models, while SRC modifies the response format through lightweight rationale SFT so that "rationale-then-answer" becomes native decoding behavior. It also distinguishes SRC from prior vision-text alignment methods that target consistency in descriptions or captions using human or expert feedback. Its claimed novelty lies in rationale-first fine-tuning without explicit prompts, a tailored R-Scorer with pairwise bidirectional scoring and factual checks, confidence-weighted preference curation, and iterative calibration of rationale–answer alignment in VQA-like scenarios (Wu et al., 17 Sep 2025).

Across both papers, the contrast with selection-only methods is especially pronounced. A plausible synthesis is that SRC converts diverse generations from an inference-time trick into a training-time signal. On that view, the essential innovation is not diversity itself, but the transformation of diversity into calibrated preference supervision over entire reasoning-output traces.

7. Limitations, failure modes, and broader implications

The judge paper identifies several limitations (Trivedi et al., 2024). Preference data quality depends on the model's ability to compare its own outputs, and the underperformance of the meta-judge strategy demonstrates that inaccurate self-comparisons can introduce bias. The base SFT judge was biased toward higher scores. The paper also does not provide explicit out-of-distribution analysis, so robustness to unseen criteria or domains is not studied. Because SRC reuses training-set examples, overfitting to the training distribution remains a consideration, even though held-out benchmark results are reported as encouraging. The paper further notes potential self-confirmation: if the initial judge systematically errs, curated preference pairs may encode those errors, though margin-based heuristics mitigate this by emphasizing clearer separations (Trivedi et al., 2024).

The LVLM paper reports analogous limitations (Wu et al., 17 Sep 2025). It requires rationale-augmented supervision, and behavior depends on the quality of rationale augmentation and filtering performed by Qwen2-VL-72B and Qwen-2.5-72B. R-Scorer, as an LLM-as-judge, may introduce residual scoring biases; bidirectional scoring reduces but does not eliminate order effects or judge preferences. Pairwise scoring scales as $e$4 per input, making efficiency a practical concern even with $e$5. Domain transfer beyond VQA-style tasks is left open, particularly for multi-turn dialogue, detailed scientific reasoning, or specialized domains (Wu et al., 17 Sep 2025).

Both papers also indicate that rationale alignment is not equivalent to correctness. The LVLM paper explicitly discusses plausible-but-wrong rationales, hallucinations, and spurious correlations in which the answer may be correct but the rationale references unrelated cues (Wu et al., 17 Sep 2025). The judge paper similarly distinguishes higher-quality rationales from mere score agreement by filtering human comparisons to equal-score cases (Trivedi et al., 2024). This suggests that SRC addresses a narrower but important problem: improving the congruence between reasoning traces and outputs, rather than guaranteeing faithful internal reasoning in a stronger causal sense.

Possible extensions listed in the papers include multi-dimensional rubrics, continuous scales and ordinal-aware losses, domain adaptation, lightweight human-in-the-loop checks, multimodal scoring with stronger visual grounding, end-to-end joint training of scorer and task model, weighting preference losses by confidence margin, and curriculum strategies that increase rationale difficulty over time (Trivedi et al., 2024, Wu et al., 17 Sep 2025). These proposals indicate that SRC is currently best understood as a modular post-training framework rather than a closed methodology.

In current usage, "Self-Rationale Calibration" therefore encompasses two convergent research programs: calibration of LLM judges through self-rationalization and DPO, and calibration of LVLM reasoning through rationale–answer alignment, pairwise scoring, and preference optimization. The shared research claim is that rationale quality can be operationalized as a trainable alignment signal, and that optimizing over preferred versus non-preferred rationale-conditioned outputs can improve both decision quality and the interpretability of those decisions (Trivedi et al., 2024, Wu et al., 17 Sep 2025).