Close the adaptive-routing performance gap

Develop practical adaptive-routing signals that close the approximately 14-percentage-point accuracy gap between the best tested routing method and the oracle under the evaluated multilingual natural language inference cascade, thereby making routing preferable to always-expensive inference under comparable compute budgets.

Background

The paper evaluates adaptive inference for African-language natural language inference using a cheap model, an expensive model, representation-geometry features, and cheap-model confidence. Although an oracle using the true per-example correctness benefit of escalation substantially outperforms always-expensive inference at a reduced compute budget, neither tested practical signal recovers enough of this available benefit.

At a 60% compute budget, the oracle achieves 0.688 accuracy, whereas the best practical routing method achieves 0.545 and always-expensive inference achieves 0.577. The authors therefore identify the roughly 14-point gap between practical routing and the oracle as an unresolved problem: finding signals or routing methods that can exploit the learnable allocation opportunity without access to the outcome being predicted.

References

Whether the remaining gap relative to computationally expensive approaches such as cross-encoder reranking or LLM-based verification can be closed without additional inference remains an open question.

— The Magnitude Mirage: Rethinking Confidence for Reasoning-Intensive Retrieval  (2609.15578 - Holdcroft et al., 14 Sep 2026) in Section titled “Limitations,” final paragraph

What the evidence supports is that the tested representation and confidence signals do not recover enough of that headroom to make routing worthwhile under the tested models, signals, and compute budgets---no practical method we evaluated exceeds simply running the expensive model on every input. The gap between what is achievable and what these signals achieve is roughly 14 accuracy points, and closing it is an open problem rather than a closed one.

— Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLI  (2608.19003 - Ogunade, 19 Aug 2026) in Discussion, paragraph "Practical routing"