Think When Needed: Model-Aware Reasoning Routing for LLM-based Ranking

Abstract: LLMs are increasingly applied to ranking tasks in retrieval and recommendation. Although reasoning prompting can enhance ranking utility, our preliminary exploration reveals that its benefits are inconsistent and come at a substantial computational cost, suggesting that when to reason is as crucial as how to reason. To address this issue, we propose a reasoning routing framework that employs a lightweight, plug-and-play router head to decide whether to use direct inference (Non-Think) or reasoning (Think) for each instance before generation. The router head relies solely on pre-generation signals: i) compact ranking-aware features (e.g., candidate dispersion) and ii) model-aware difficulty signals derived from a diagnostic checklist reflecting the model's estimated need for reasoning. By leveraging these features before generation, the router outputs a controllable token that determines whether to apply the Think mode. Furthermore, the router can adaptively select its operating policy along the validation Pareto frontier during deployment, enabling dynamic allocation of computational resources toward instances most likely to benefit from Think under varying system constraints. Experiments on three public ranking datasets with different scales of open-source LLMs show consistent improvements in ranking utility with reduced token consumption (e.g., +6.3\% NDCG@10 with -49.5\% tokens on MovieLens with Qwen3-4B), demonstrating reasoning routing as a practical solution to the accuracy-efficiency trade-off.

• The paper introduces a plug-and-play router that dynamically selects between reasoning and non-reasoning modes in LLM-based ranking tasks, improving NDCG by up to 6.3% on MovieLens and reducing token usage by up to 75.3% on MS-MARCO
• The router combines ranking-aware features (candidate complexity, context alignment) and model-aware difficulty signals (yes/no checklist diagnostic answers) to predict the advantage of using a reasoning or Non-Think mode
• The study concludes that instance-level routing, not universal reasoning, is crucial for enhancing efficiency in large language model (LLM) ranking tasks, offering a practical solution for balancing accuracy and cost

Motivation and problem statement

Reasoning prompting (Think mode) improves LLM-based ranking on average, but the paper's diagnostic experiments show that its benefits are inconsistent and expensive. On MovieLens with Qwen3-4B, Think improves NDCG@10 by 2.1% over Non-Think while consuming roughly seven times more tokens. More strikingly, per-instance analysis on MS-MARCO with Qwen3-14B shows that Non-Think outperforms Think in 35% of instances, and 3.6% of instances yield zero accuracy under both modes, indicating cases beyond model capacity where reasoning is pure waste. An oracle ("Ideal") router that picks the better mode per instance increases NDCG@10 by 32% over either single mode on MovieLens while using 55% fewer tokens than Think — a large headroom that motivates instance-level routing.

The authors further show that routing decisions are largely model-dependent: the same MS-MARCO instance may favor Think under Qwen3-4B but Non-Think under Qwen3-14B, ruling out a universal, model-agnostic router. Self-Select — letting the LLM choose its own mode via prompt — is unreliable, remaining well below always-Think in utility, which indicates current LLMs cannot self-assess ranking difficulty.

Framework

The proposed router is lightweight, plug-and-play, and executes within a single forward pass of the backbone without modifying architecture or decoding. It combines two feature families:

Ranking-aware features. Segment embeddings for the context and each candidate are pooled from the backbone's final hidden states, then summarized into statistics along two dimensions: context/candidate complexity (e.g., candidate dispersion, cluster-size entropy) and context–candidate alignment (e.g., centroid cosine similarity). A two-stage refinement — sparsity-inducing feature selection followed by cross-dataset/cross-model consistency filtering — yields a compact, robust subset.

Model-aware difficulty signals. A masked checklist of diagnostic yes/no questions is appended to the prompt suffix. A block-diagonal causal mask enforces probe isolation so checklist tokens cannot interfere with ranking generation. Yes/No probabilities at an "Answer:" anchor via the LM head capture the model's internal confidence. To counter known LLM yes-bias, questions are constructed in direction-balanced pairs (one phrased toward Non-Think, one toward Think), and ChatGPT-generated candidates are pruned via reflective elicitation for redundancy.

Router head. Rather than binary classification, routing is formulated as regression on a compute-aware advantage:

$$A_p = \big(U_p^{Think} - U_p^{Non}\big) - \lambda\big(T_p^{Think} - T_p^{Non}\big)$$

A gradient-boosted regressor $g_\phi$ predicts $A_p$, trained with robustness weights and a monotonicity constraint ensuring predicted advantage does not increase with extra cost $\Delta T_p$. Feature selection follows ManiFeSt-style intra-class relational analysis.

Deployment-time policy selection. Sweeping a density parameter $\eta$ in the routing rule $a_p(\eta)$ = Think iff $A_p - \eta(T_p^{Think}-T_p^{Non}) \ge 0$ traces a validation Pareto frontier over token cost and utility. Four anchors map operational goals to $\eta$: Knee (efficiency-first, farthest point from the anti-diagonal), Utopia (weighted distance to ideal low-cost/high-utility), Epsilon (minimum cost meeting a utility target $U_{base}+\epsilon$), and UMax (utility-first). The chosen $\eta$ is frozen at deployment for stable, auditable behavior.

Experimental results

Experiments cover three datasets (MS-MARCO for IR; MovieLens and Amazon-VG for recommendation with leave-one-out and 49 sampled negatives), five open-source backbones (Qwen3-4B/8B/14B, Gemma3-12B, GPT-Oss-20B), and four baselines: Non-Think, Think, Random, and Self-Select, plus the ARM family of adaptive-reasoning models.

Key findings under the UMax policy:

Setting	Metric	Baseline → Ours	Token change
MovieLens, Qwen3-4B	NDCG@10	0.190 → 0.202 (+6.3%)	−49.5%
Amazon-VG, Qwen3-4B	NDCG@10	0.203 → 0.211 (+3.9%)	−20.5%
MS-MARCO, Qwen3-4B	NDCG@10	0.880 → 0.882	−75.3%
MovieLens, Gemma3-12B	NDCG@10	matches Think to 99.37%	−34%

The router exceeds always-Think utility in most settings while cutting tokens substantially. The Gemma3-12B case is notable as a stress test: there, Non-Think beats Think overall on MovieLens, yet the router still tracks Think-level quality at lower cost, indicating robustness when reasoning is not globally beneficial. The phenomenon of Non-Think winning persists even at scale — on MS-MARCO with Qwen3-8B, Non-Think slightly surpasses Think (0.883 vs. 0.877) despite Think costing ~8× more tokens.

Comparison with ARM is unfavorable to ARM: ARM classifies nearly all ranking instances as "hard," defaulting to long reasoning chains that both increase cost and degrade utility relative to the proposed router, suggesting adaptive-reasoning designs built for math/code transfer poorly to listwise ranking.

Pareto-frontier analysis shows the Epsilon anchor reaches Think-level NDCG with roughly 10–30% fewer tokens on the Rec datasets, Utopia typically halves token usage with small utility loss, Knee achieves 60–80% savings with controlled accuracy loss, and UMax often exceeds peak Think NDCG by avoiding harmful over-reasoning.

Ablations confirm both feature groups matter, with task-dependent weighting: SHAP importance shows ranking-aware features dominate on Rec datasets (preference structure and alignment cues), whereas model-aware checklist signals dominate on IR (per-query ambiguity drives reasoning benefit).

Prompt-sensitivity analysis on MS-MARCO with Qwen3-14B shows prompt edits cause substantial instance-level shifts in which mode wins, yet the router retains its favorable trade-off under a modified CoT-style prompt, indicating robustness to realistic prompt engineering choices.

Limitations and open questions

The authors concede that the hand-crafted feature group and checklist probes may miss nuanced semantic difficulty and require re-tuning across domains or prompt formats. Additional caveats bear on interpretation: the router is trained on supervised advantage labels obtained by running both modes offline, so it inherits the cost of dual inference during training; evaluation is limited to open-source instruction-tuned models with fixed prompts and candidate sets provided by upstream retrievers, leaving candidate-generation effects unexamined; and direct comparison with fine-tuning-based adaptive-reasoning methods was infeasible due to lack of public ranking datasets or fine-tuned checkpoints. Whether the advantage-regression formulation transfers to settings where reasoning depth (not just mode) is controllable remains open, as does reinforcement-based training of the router against downstream ranking objectives rather than supervised targets.

Conclusion

This paper formulates reasoning routing for LLM-based ranking and demonstrates that deciding when to reason matters as much as how. By combining pre-generation ranking-aware features with model-aware checklist difficulty signals in a lightweight regressor, and by exposing deployment-time policy selection through a validation Pareto frontier, the framework consistently improves ranking utility while reducing token consumption across three datasets and five backbones. Its main contribution is a practical, non-intrusive mechanism for resolving the accuracy–efficiency trade-off in latency-sensitive ranking pipelines, with clearly stated limits around domain-specific re-tuning of features and probes.