System One Models Are Not Better Rerankers
This presentation examines a controlled experimental comparison between decision fine-tuning and conventional reranker fine-tuning for passage retrieval. The research directly challenges early claims that System One models constitute competitive rerankers, demonstrating that decision fine-tuning never produces the best reranker on any evaluated backbone from 0.6B to 9B parameters. While decision models can provide better native calibration under specific question wordings, this advantage does not compensate for inferior ranking quality in standard retrieval tasks. The findings clarify when each approach is appropriate and reveal how sensitive calibration advantages are to prompt design.Script
Decision fine-tuning trains a language model to return calibrated probabilities for binary questions, while conventional reranker fine-tuning optimizes passage ordering directly. The authors test whether decision models trained on explicit yes-or-no questions produce better rerankers than models fine-tuned specifically for relevance ranking.
Across backbones from 0.6 billion to 9 billion parameters, reranker fine-tuning outperforms decision fine-tuning by margins ranging from 5 to 29 nDCG at 10 points. At 27 billion parameters, the two approaches tie, but the comparison at that scale is confounded by differences in training method and context length.
Decision models show better native calibration under the primary question wording, especially at 4 billion parameters and above. However, changing a single sentence in the relevance prompt eliminates this advantage, revealing that calibration depends sharply on how the question aligns with evaluation labels.
When a fixed threshold of 0.5 is applied to select passages, reranker fine-tunes retain better passage sets on every backbone up to 9 billion parameters, despite generally worse native calibration. A well-placed threshold applied to a worse ranking still produces a worse retained set, so calibration cannot rescue inferior ordering.
Decision fine-tuning adds almost nothing over zero-shot prompting at model sizes up to 9 billion parameters. In contrast, reranker fine-tuning improves ranking by 6 to 30 nDCG at 10 points, demonstrating that decision training does not generally enhance the underlying model's ability to discriminate relevant passages from hard negatives.
When the primary requirement is passage reranking, relevance-specific fine-tuning is the stronger default across model sizes up to 9 billion parameters. Decision models are most defensible when calibrated absolute probabilities, rather than ranking alone, are required for tasks like threshold selection or prevalence estimation. To explore this research further and create your own videos, visit EmergentMind.com.
