Determine whether hidden-state routing gains arise from improved cost representation

Determine whether the routing gains obtained by replacing GTE semantic embeddings with Qwen2.5-7B-Instruct layer-21 hidden-state embeddings arise from a better representation of response cost in the embedding space.

Background

The paper compares semantic GTE embeddings with the last-token hidden state at layer 21 of Qwen2.5-7B-Instruct for out-of-distribution LLM routing. The hidden-state representation improves routing utility, particularly for positive cost penalties. Separate reward and cost analyses show that the relative contribution of cost prediction differs between RouterBench and SPROUT, motivating—but not definitively establishing—the conjecture that hidden states encode cost-relevant information more effectively.

The authors explicitly present this explanation as a conjecture rather than a demonstrated result. The subsequent controls indicate benchmark-dependent contributions from reward and cost estimates, leaving the general mechanism unresolved.

References

Furthermore, we note that most gains from the methods above come for $\lambda>0$. This hints at the possibility, which we conjecture, that the gains come from a better representation of cost by the embedding space.

— Embedding Geometry Shapes LLM Routing: Tracing Failure Modes and Improving Generalization  (2610.19198 - Torti et al., 13 Oct 2026) in Section 3.1, “Changing the Embedding Improves Routing”