Cost-Spectrum Contrastive Routing (CSCR)

• The paper introduces a unified framework for cost-sensitive routing of prompts in LLMs by mapping both prompts and models into a shared Euclidean space.
• It employs dual descriptors, cost-banded contrastive losses, and FAISS k-NN to achieve microsecond latency and improved accuracy–cost trade-offs.
• Empirical evaluation shows up to 25% improvement in AUDC over existing methods, ensuring robust generalization under varying expert pools.

Cost-Spectrum Contrastive Routing (CSCR) is a framework for efficient, cost-sensitive routing of prompts to LLMs within diverse and dynamic expert pools. CSCR addresses the limitations of traditional model routing—such as prompt-context ignorance, prohibitive profiling requirements, and reliance on fixed expert sets—by constructing a shared embedding space for models and prompts. This enables rapid model selection that explicitly manages the tradeoff between inference cost and predicted accuracy, using a contrastive learning approach informed by model-specific cost bands. The method leverages fast descriptors (logit footprints or perplexity fingerprints), banded cost-aware InfoNCE losses, and FAISS-index-based approximate nearest neighbor search to deliver microsecond-latency routing without retraining upon pool changes, while empirically outperforming existing schemes in accuracy–cost tradeoff and generalization (Shirkavand et al., 17 Aug 2025).

1. Embedding-Based Representations and Model Descriptors

CSCR's central innovation is mapping both prompts and expert models into a common Euclidean embedding space in which distance encodes cost-aware suitability. Model representations fall into two categories, depending on model access:

• Logit Footprints (open-source LLMs): For $h \in H$ (the expert pool), descriptors are computed by recording next-token distributions over a fixed probe prompt set $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$ and a short generation horizon $T$. At each of $T$ steps and for each prompt, the $K$ most frequent tokens form the vector:

$$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$

• Perplexity Fingerprints (black-box APIs): Where logits are inaccessible, closed-source models are “fingerprinted” using a reference scorer $S$. Each expert’s output is scored under $S$ on the probe set:

$$\tilde{\ell}_h(x) = -\frac{1}{|\hat{y}_h(x)|} \sum_{j=1}^{|\hat{y}_h(x)|} \log p_S(\hat{y}_{h,j} | \hat{y}_{h,<j}).$$

The descriptors are mean-centered, normalized vectors:

$$d_{\mathrm{PPL}}(h) = \text{normalize}([\ell_h(x^{(i)})]_{i=1}^N) \in \mathbb{R}^N, \quad \text{then normalized to the unit sphere.}$$

This dual-descriptor system unifies open and black-box LLMs under a shared retrieval mechanism.

2. Contrastive Encoder Architecture and Loss Function

Prompts are encoded using a frozen MiniLM-L6-v2 backbone ($S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$0) followed by a two-layer trainable MLP head $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$1, $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$2-normalized to $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$3. Expert keys $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$4 inhabit the retrieval space.

CSCR generalizes classical InfoNCE contrastive loss by incorporating costs $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$5:

• Cost Bands: Experts are partitioned into $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$6 percentile bands $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$7 by cost, with positives $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$8 per query in each band, and adaptive band-specific temperature $S_{\mathrm{probes}} = \{x^{(i)}\}_{i=1}^N$9, where $T$0 is the mean cost in band $T$1.
• Cost-Spectrum InfoNCE Loss:

$T$2

where $T$3 is the “negative-cost penalty,” controlling the penalization of incorrect, costly experts.

This structure ensures the learned metric ranks cheap correct experts above expensive or wrong ones, accommodates band-specific gradient allocation (critical for high-cost bands), and extends InfoNCE with cost-sensitive weighting.

3. Routing Inference via k-NN and FAISS

At inference, CSCR computes the query embedding for a given prompt, and identifies the top-$T$4 nearest experts via cosine similarity using a FAISS IndexFlatIP (inner-product) search over $T$5-normalized expert descriptors.

Routing proceeds as follows:

1. $T$6
2. Retrieve $T$7 top-$T$8 experts using FAISS: $T$9
3. For each $T$0 compute $T$1
4. Return the expert maximizing $T$2: $T$3

In practice, $T$4 suffices, and lookup takes $T$5s for thousands of experts. New experts are incorporated by updating the descriptor set, eliminating the need for retraining.

$$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$9

4. Theoretical Properties

CSCR provides theoretical guarantees in terms of excess risk and embedding consistency:

• Excess-Risk Bound: Under a Lipschitz-in-query embedding assumption, the $T$6-NN routing rule on $T$7 i.i.d. prompts achieves:

$T$8

where $T$9 is embedding dimension, $K$0 is Bayes-optimal, $K$1 is a constant. This recovers the standard $K$2-NN rate.

• Cost-Spectrum InfoNCE Consistency: At any stationary point of $K$3, for each query $K$4 and correct expert $K$5 in a cheaper band than any incorrect $K$6, $K$7, ensuring the induced metric aligns with the desired cost-adjusted ranking.

5. Empirical Evaluation

CSCR is benchmarked on three routing datasets:

Benchmark	Model Access	Descriptor	Metric(s)
EmbedLLM	Open-source	Logit-footprint	AUDC, QNC, Peak accuracy
MixInstruct	Open-source	Logit-footprint	Exponentiated BARTScore, Quality/Cost
RouterBench	Black-box APIs	PPL fingerprint	USD cost, binary accuracy

Deferral curves plot response quality vs. cumulative cost as $K$8 varies. Key reported metrics:

• AUDC: Area under deferral curve (higher = better)
• QNC: Query-normalized cost to match best expert (lower = better)
• Peak Accuracy: Maximum observed for routing

Across all datasets, CSCR improves the accuracy–cost trade-off by up to 25% in AUDC over baselines such as UMR, Thompson, Soft-MoE, and others. Generalization experiments—holding out two-thirds of experts or switching from STEM to non-STEM prompts—show that CSCR retains top AUDC performance for novel experts and out-of-distribution prompts. Ablation studies confirm that logit/perplexity descriptors and cost-aware losses are both critical for these results. Optimal performance is achieved with $K$9 cost bands, $$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$0 for $$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$1-NN, $$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$2–$$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$3 for cost penalty, and moderate smoothing of band temperatures ($$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$4–$$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$5, $$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$6–$$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$7).

6. Extensions, Limitations, and Future Directions

CSCR has been validated on pools of approximately 5–15 experts. Further evaluation on larger, heterogeneous expert pools ($$d_{\mathrm{logit}}(h) = \frac{1}{N T} \sum_{i=1}^N \sum_{t=1}^T [p_h(v_k | x^{(i)}, t)]_{k=1}^K \in \mathbb{R}^K, \quad \|d_{\mathrm{logit}}(h)\|_2 = 1.$$8) is necessary for operational-scale deployments. Fixed probe prompt sets underpin descriptor construction; adopting adaptive or learnable probes could result in further footprint compression. Integration of dense user feedback, such as pairwise preference data, could refine label sets and margins within cost bands. Broadening cost-awareness to dynamic, per-prompt cost budgets or multi-objective contexts, such as latency or API quota optimization, represents another unexplored direction. Finally, exploration of reinforcement learning for routing within LLMs, potentially achieving token-level granularity, is proposed for future research.

CSCR establishes a plug-and-play, lightweight paradigm for cost-aware routing among LLMs by embedding prompts and models in a unified space, deploying a cost-banded contrastive loss, and performing microsecond-scale FAISS k-NN inference. It achieves superior accuracy–cost profiles and robust generalization relative to existing methods, while remaining compatible with a wide range of LLM deployment scenarios (Shirkavand et al., 17 Aug 2025).