System One models Are Not Better Rerankers

Abstract: System One models, or decision models, answer a question about an input with a calibrated probability instead of generated text. Within weeks of the first release, in September 2026, they were used as rerankers. We test whether the decision fine-tuning behind them makes a better reranker than plain reranker fine-tuning of the same model. For six Qwen backbones from 0.6B to 27B parameters and ten TREC DL and BEIR collections, we compare the untuned model, asked zero-shot, with its decision fine-tunes and its reranker fine-tunes, two of which we train ourselves. In our work, we draw the conclusion that decision fine-tuning never makes the better reranker. From 0.6B to 9B, reranker fine-tuning wins on every backbone by 5 to 29 points nDCG@10, also on zero-shot collections, and at 27B the two tie. Up to 9B, decision fine-tuning adds nothing over simply asking the untuned model. Moreover, we find that the calibration advantage of decision models depends on the question. Asked whether a passage answers the query, they are better calibrated than the published reranker fine-tunes from 4B on. Asked a slightly different question from prior work, they lose this advantage. Lastly, calibration does not make up for a worse ranking: up to 9B, a reranker fine-tune keeps the better passages at a fixed threshold. If you need a reranker, fine-tune a reranker.

• The paper found that dedicated reranker fine-tuning improves ranking by 5.2 to 28.8 nDCG@10 points over decision fine-tuning for models up to 9B parameters.
• Calibration advantage of decision models separates from their ranking performance and varies based on question wording."
• Decision fine-tuning up to 9B typically adds little improvement to nDCG@10 over zero-shot prompting, while reranker fine-tuning consistently delivers 6 to 30 NDCG@10 points benefit.
• post_hoc_calibration_highlighting_the_dependency_of_calibration_on_data_distribution

Research question and experimental premise

“System One Models Are Not Better Rerankers” (2610.32318) examines whether decision fine-tuning—training a LLM to return calibrated probabilities for explicit binary questions—produces a better passage reranker than conventional relevance-oriented reranker fine-tuning. The comparison is deliberately controlled at the backbone level. Rather than comparing a decision model with a reranker built from a different model family, the study evaluates untuned Qwen models, decision fine-tunes, and reranker fine-tunes derived from the same Qwen backbones.

The central claim is strong and directly contrary to early reports that System One models constitute competitive or superior rerankers: decision fine-tuning never produces the best reranker on any evaluated backbone. From 0.6B to 9B parameters, dedicated reranker fine-tuning improves ranking substantially over decision fine-tuning; at 27B, the two approaches are approximately tied. The result is accompanied by a second, more qualified finding: decision models generally provide better native calibration for a relevance question that matches the evaluation labels, but that advantage is highly sensitive to question wording and does not compensate for inferior ranking quality.

The paper separates two objectives that are often conflated in retrieval evaluation. Ranking depends primarily on the relative ordering of scores, whereas calibration concerns the correspondence between numerical probabilities and empirical relevance frequencies. A model may therefore rank effectively while producing poorly calibrated scores, or produce calibrated probabilities while ordering passages poorly. The experiments test both properties independently.

Decision-model reranking formulation

A decision model receives a query, one or more candidate passages, and a binary question. The model returns the probability of the “yes” answer. For isolated scoring, each state contains a query and one candidate passage, yielding a pointwise score of the form $P(\text{yes}\mid q,d_i)$. In joint scoring, the state contains the query and all 100 BM25 candidates, while the model is queried separately for each passage. The latter formulation allows the score for one passage to depend on the other candidates, although the final output remains a pointwise score for ranking.

The primary question is deliberately aligned with the relevance criterion used in the evaluation: “Does document {doc} answer the query?” The authors also reproduce the wording used by S1Rank: “Is document {doc} relevant to the query? Relevant means it answers the query or contains information that directly addresses it.” This ablation is methodologically important because it tests whether calibration is a stable model property or a consequence of the semantic relationship between the prompt and the labels.

Three model classes are compared:

• Untuned models: released Qwen chat models prompted directly and scored using the probability of “yes” as the first answer token.
• Decision fine-tunes: Kev models across all six backbones, supplemented by Clef and Clef-Flash at larger sizes, evaluated in isolated and joint modes.
• Reranker fine-tunes: Qwen3-Reranker, Prism, and two rerankers trained by the authors.

The authors’ 9B and 27B rerankers provide an additional control. Both use a standard Qwen3-Reranker-style prompt, LoRA with rank 32, one epoch of training on 100,000 MS MARCO queries, one positive passage, and 15 hard negatives per query. The objective combines groupwise InfoNCE ranking loss with binary cross-entropy. No decision-model training data or hyperparameter search is used. This design prevents the comparison from depending exclusively on the undisclosed training procedures of published rerankers, although it does not eliminate all differences between the released decision models and the authors’ single-recipe controls.

Evaluation design

The evaluation uses BM25 top-100 candidates from TREC Deep Learning 2019–2023 and five BEIR collections: TREC-COVID, NFCorpus, SciFact, Touché, and DBPedia. The zero-shot subset consists of TREC-COVID, Touché, and DBPedia, which were not included in the training data of the fine-tuned systems. Ranking is measured with nDCG@10, with paired $t$-tests and Holm correction across collections.

The six Qwen backbones range from 0.6B to 27B parameters:

• Qwen3-0.6B and Qwen3-8B;
• Qwen3.5-0.8B, 4B, and 9B;
• Qwen3.8-27B.

Calibration is evaluated on judged query–passage pairs from the five TREC DL collections using ECE and Brier score. Two operational uses of probabilities are tested. First, the system retains passages with probability at least 0.5, and the resulting set is evaluated using $F_1$, threshold regret, and the number of retained passages. Second, the probabilities assigned to a query’s judged passages are summed to estimate the number of relevant passages. These evaluations appropriately treat calibration as useful only when scores are consumed on an absolute scale, such as filtering, truncating, or estimating relevance prevalence.

Ranking results

The ranking results consistently favor relevance-specific reranker fine-tuning at model sizes up to 9B. The margin over the stronger of the two decision-model modes ranges from 5.2 nDCG@10 points for the 9B backbone to 28.8 points for the 0.6B backbone. The comparison is statistically significant for 48 of 60 reranker–collection pairs, and the decision fine-tune is never significantly superior.

Backbone	Best decision fine-tune	Reranker fine-tune	Reranker advantage
Qwen3-0.6B	27.3	56.1	28.8
Qwen3-8B	51.7	59.4	7.7
Qwen3.5-0.8B	34.2	57.1	22.9
Qwen3.5-4B	49.9	58.7	8.8
Qwen3.5-9B	53.8	59.4	5.6
Qwen3.8-27B	60.1	59.6	−0.5

The same pattern holds on the zero-shot collections. The reranker advantage ranges from 5.4 to 23.9 points, demonstrating that the result is not simply a consequence of rerankers having encountered the evaluation domains. This is a consequential distinction: the superiority of reranker fine-tuning persists even when the test collections are outside the fine-tuning data.

The authors’ independently trained rerankers reinforce the comparison. Ours-9B achieves an average nDCG@10 of 59.4, essentially matching Prism-9B at 59.0, despite using a simple MS MARCO-only recipe. Ours-27B reaches 59.6, while Kev-27B reaches 60.1. Thus, the result does not appear to depend on a proprietary or unusually sophisticated reranker recipe.

A particularly strong finding is that decision fine-tuning adds almost nothing over zero-shot prompting up to 9B. Relative to the corresponding untuned model asked the same question, the stronger decision configuration remains within −2.5 and +0.7 nDCG@10 points. In contrast, reranker fine-tuning adds between 6 and 30 points, depending on the backbone. For example, the untuned Qwen3.5-4B reaches 52.4 average nDCG@10, Kev-4B reaches 49.9, and Prism-4B reaches 58.7. Consequently, for these model sizes, decision fine-tuning is not merely inferior to reranker fine-tuning; it often fails to improve the underlying model’s ranking capability at all.

The 27B results qualify rather than overturn the conclusion. Kev-27B improves over untuned Qwen3.8-27B from 56.4 to 60.1, while the authors’ reranker reaches 59.6. The two approaches are statistically indistinguishable on nearly all collections. However, the paper explicitly notes that the 27B comparison is confounded: Kev-27B is a full fine-tune of a released chat checkpoint, whereas smaller Kev models use LoRA on pretrained checkpoints, and the 27B model was trained with longer states. Clef, another 27B decision fine-tune, does not improve over the untuned model. Therefore, model scale alone cannot explain the 27B behavior.

Joint decision scoring is also not uniformly beneficial. It improves Kev-27B by approximately one nDCG@10 point and produces a similar gain for Clef, but it degrades Kev-0.6B by 12.2 points and Kev-8B by 13.0 points. The authors attribute this behavior plausibly to context length: 100-passage states contain approximately 13,000 tokens on TREC DL and at least 40,000 on some BEIR collections, exceeding the reported training-state lengths of smaller Kev variants. The evidence supports a context-length explanation, but the experiments do not isolate context truncation, attention degradation, or distributional mismatch as separate causes.

Calibration results and the role of question wording

Decision models do show a calibration advantage under the primary question, especially at moderate and large model sizes. From 4B onward, decision fine-tunes are generally better calibrated than the published reranker fine-tunes of the corresponding backbone. For example, Kev-8B has ECE 0.113 compared with 0.257 for Qwen3-Reranker-8B, and Kev-27B reaches ECE 0.076. Jev provides the lowest reported ECE at 0.075.

This advantage is not present for the smallest models. At 0.6B, Kev has ECE 0.503 and Qwen3-Reranker has 0.511; at 0.8B, Kev reaches 0.336 while Prism reaches 0.243. Untuned small models are particularly pathological: Qwen3-0.6B has ECE 0.750 and Qwen3.5-0.8B has ECE 0.617, reflecting a tendency to assign high “yes” probabilities indiscriminately.

The calibration result changes substantially under the S1Rank wording. On the Qwen3-8B backbone, for example, Kev-8B’s ECE rises from 0.113 under “Does document answer the query?” to 0.243 under the looser relevance definition. Jev’s joint-mode ECE increases from 0.075 to 0.224. The paper emphasizes that a single sentence in the prompt can eliminate the apparent calibration advantage of decision models. The change is not limited to decision fine-tunes: the untuned Qwen3-8B model’s ECE changes from 0.383 under the S1Rank wording to 0.231 under the primary wording.

The implication is that calibration is conditional on a particular relevance semantics and base-rate distribution. A decision model is not calibrated in an unconditional sense; it is calibrated with respect to the question it is asked and the labels against which its probabilities are interpreted. The primary wording matches the TREC relevance criterion more closely, which explains why both untuned and decision-tuned models improve under it. Decision fine-tuning contributes additional calibration, but prompt-label alignment is at least as important in several cases.

The results also show that calibration can be obtained through ordinary reranker training. Ours-9B reaches ECE 0.111, essentially matching Clef-Flash’s 0.112, using a ranking objective plus binary cross-entropy. This weakens the claim that calibrated relevance probabilities are a distinctive consequence of System One training. Nevertheless, Ours-27B has ECE 0.110 compared with Kev-27B’s 0.076, so decision fine-tuning can still provide a calibration benefit when ranking quality is comparable.

Operational value of calibrated probabilities

The fixed-threshold experiment tests whether better calibration improves passage selection. The threshold is $p \geq 0.5$, and threshold regret measures how much $F_1$ would improve if the best collection-specific threshold were selected after evaluation.

Up to 9B, reranker fine-tunes retain the better passage set on every backbone despite generally worse native calibration. For example, Prism-4B achieves $[email protected]$ of 51.0, compared with 42.6 for Kev-4B. Decision models have relatively low threshold regret, typically between 0.9 and 3.1 points, indicating that 0.5 is already near their optimal threshold. However, this does not compensate for their poorer ordering: a well-placed threshold applied to a worse ranking still produces a worse retained set.

At 27B, where ranking quality is approximately tied, calibration becomes operationally valuable. Kev-27B achieves $[email protected]$ of 57.2, whereas the authors’ underconfident reranker reaches only 48.3 and retains just 7.9 passages per query. This result demonstrates the intended division between ranking and calibration: calibration has little ability to rescue a weak ranking, but it can materially improve threshold-based selection when the rankings are already competitive.

The count-estimation experiment produces a similar pattern. Summed relevance probabilities from decision models often estimate the number of relevant passages more accurately than the median baseline error of 8.6 passages. Kev-8B has error 7.6 compared with 12.9 for Qwen3-Reranker-8B. Yet decision fine-tuning is not necessary: untuned Qwen3.8-27B obtains the best error, 5.8, and Ours-9B outperforms Clef-Flash, with errors of 7.3 and 8.1 respectively. Under the S1Rank wording, no decision model improves meaningfully over the median baseline, again showing that the usefulness of probabilities depends on prompt-label alignment.

Post-hoc calibration partially narrows the difference. Temperature scaling fitted on MS MARCO development data reduces the ECE of poorly calibrated rerankers, such as Qwen3-Reranker-0.6B from 0.511 to 0.326. Platt scaling, however, performs poorly at the fixed 0.5 threshold because MS MARCO has approximately one relevant passage per query, whereas TREC DL queries contain substantially more relevant passages. This result highlights a base-rate limitation: calibration learned on one collection and relevance distribution does not automatically transfer to another.

Interpretation and limitations

The paper’s explanation for the ranking gap is that reranker fine-tuning optimizes the tested task directly: discriminating relevant passages from hard negatives using ranking supervision. Decision fine-tunes instead learn a broader decision interface across questions and states. For passage relevance, that general capability transfers approximately as well as the untuned model’s instruction-following behavior, at least up to 9B. The zero-shot results support this interpretation, although they do not prove that task specialization is the only mechanism responsible.

The empirical comparison has several limitations. The study evaluates released decision models whose training procedures and data differ from those of the rerankers. In particular, it does not reproduce RLCD, the unpublished method associated with Jev. The authors’ reranker controls are trained once with a deliberately simple recipe and without hyperparameter optimization, so a stronger reranker-training protocol could alter the numerical margins. Calibration is measured only on TREC DL, at one relevance grade, with one principal threshold. Only two question wordings are tested, and the primary wording, although fixed before evaluation, was not selected using a development set. Finally, the 27B models differ in fine-tuning method and context length, preventing a clean causal attribution of their convergence.

These limitations constrain the scope of the conclusion. The evidence establishes that the currently available decision fine-tunes are not better rerankers than same-backbone relevance fine-tunes under the reported setup. It does not establish that every possible calibrated-decision objective is inferior, nor that decision models lack value in thresholding, relevance counting, or other tasks where absolute probabilities are required. The open technical question is whether a decision model trained with relevance-specific data, controlled prompt semantics, and a ranking-sensitive objective could retain its calibration advantages without sacrificing ranking performance.

Conclusion

The paper provides a controlled empirical comparison between decision fine-tuning and conventional reranker fine-tuning across six Qwen backbones and ten retrieval collections. Its principal result is consistent: from 0.6B through 9B, reranker fine-tuning improves nDCG@10 by 5–29 points over decision fine-tuning, while decision fine-tuning adds little over zero-shot prompting; at 27B, the approaches tie. Decision models can offer better native calibration, but this advantage depends sharply on the wording of the relevance question and does not offset inferior ranking in fixed-threshold retrieval. The practical conclusion follows directly from the evidence: when the primary requirement is passage reranking, relevance-specific reranker fine-tuning is the stronger default; decision models are most defensible when calibrated absolute probabilities, rather than ranking alone, are the central requirement.