Multi-Tier LLM Routing

• Multi-tier LLM routing is a dynamic selection of heterogeneous models and components based on explicit performance, cost, latency, and compliance constraints.
• Architectural patterns include pre-generation, cascade, and distributed routing that effectively balance latency, resource usage, and output quality.
• Empirical studies show that multi-tier routing can achieve similar quality to large models at 2–5× lower cost and enhanced inference efficiency.

Multi-tier LLM routing is the dynamic selection, per query, of one or more models or adjacent pipeline components from a heterogeneous pool under explicit quality, cost, latency, energy, or compliance constraints. In contrast to monolithic single-model deployments, routing treats an LLM-based system as a dynamic architecture in which tiers can include small and large generalist models, domain experts, retrieval and embedding modules, prompting or reasoning modes, tools, and geographically constrained endpoints. Routing may occur before generation, after an initial answer in a cascade, or across the full retrieval-and-generation pipeline (Varangot-Reille et al., 1 Feb 2025, Moslem et al., 23 Feb 2026).

1. Formal problem statements

The canonical formulation treats routing as a resource-aware decision problem over a set of candidate models. The survey formalizes the task as selecting, for each query, the model or component that maximizes a task-specific score under a budget constraint, with cost defined broadly enough to include \$/token, latency, energy, or hardware usage (Varangot-Reille et al., 1 Feb 2025). A closely related scalarization appears in several routing methods: $$Performance_{i,j} = \lambda \cdot P_{i,j} - cost_j,$$ where $P_{i,j}$ is the predicted quality of model $M_j$ on query $q_i$, $cost_j$ is the cost of using that model, and $\lambda$ is the user’s willingness to pay (Varangot-Reille et al., 1 Feb 2025).

Subsequent work specializes this general view. IRT-Router scores each query–model pair with a weighted performance–cost objective,

$$\mathcal{S}(q_i, M_j) = \alpha \cdot \hat{\mathcal{P}}(q_i, M_j) - \beta \cdot \mathcal{C}(M_j),$$

and chooses the model with maximal score, where $\hat{\mathcal{P}}(q_i, M_j)$ is predicted performance and $\mathcal{C}(M_j)$ is a fixed cost derived from output price per million tokens (Song et al., 1 Jun 2025). PROTEUS instead poses routing as constrained optimization with an explicit accuracy floor $\tau$,

$P_{i,j}$0

so that the runtime control variable is the requested accuracy target rather than an opaque trade-off knob (Bhatti et al., 27 Jan 2026).

The literature increasingly generalizes the “tier” concept beyond model size. The survey’s DynamicRAG perspective treats embedding models, retrieval strategies, context lengths, prompting styles, reasoning modes, and tools as routing targets alongside the generator itself (Varangot-Reille et al., 1 Feb 2025). This suggests that multi-tier routing is best understood not as a binary small-versus-large switch, but as structured policy selection over an entire inference stack.

2. Architectural patterns

A basic distinction separates pre-generation routing from post-generation routing. In pre-generation routing, the system decides before any answer is produced which model, expert, retriever, prompt family, or toolchain should handle the query. This is the dominant design for low-resource production systems because it avoids unnecessary calls to expensive models (Varangot-Reille et al., 1 Feb 2025). In post-generation routing, a cheaper model answers first, a quality or uncertainty evaluator decides whether the output is good enough, and the query is escalated if needed; the survey describes this as cascade routing and notes that it is inherently more expensive because multiple generations may be required (Varangot-Reille et al., 1 Feb 2025).

Several papers broaden the architectural scope of “tiers.” RecServe implements recursive offloading across device, edge, and cloud, with progressively larger models $P_{i,j}$1 and task-specific confidence thresholds at each tier. A request is answered locally if confidence exceeds a dynamic threshold; otherwise the original input is offloaded to the next tier (Wu et al., 22 May 2025). ComplianceGate moves the routing decision even earlier: a 12-layer BERT-like encoder classifier evaluates each query for complexity and sensitivity before any decoder inference, then routes it to a model tier and a geographic endpoint. Because PII-containing queries are sent to local endpoints before any LLM computation begins, the paper states that data residency violations become structurally impossible (Dey, 30 Jun 2026).

A different architectural line replaces centralized gating with distributed control. DiSRouter organizes models as a cascade of self-aware agents, each trained to either answer or emit a rejection string, “I don’t know!”, which routes the query upward to a more capable model. The final model always answers, so the process terminates without a separate central router (Zheng et al., 22 Oct 2025). This suggests a useful distinction between centralized routing policies, which infer the competence of all models externally, and distributed self-routing, in which each model judges its own competence locally.

3. Routing mechanisms and decision signals

The survey organizes routing strategies along two axes: routing stage and resource requirement. High-resource methods include supervised cascades such as FrugalGPT, prompt-based routers that use a strong LLM to select a downstream model or tool, repeated-call self-consistency schemes such as Automix and Mixture-of-Thoughts, and task-specific evidence mechanisms such as code execution for code tasks (Varangot-Reille et al., 1 Feb 2025). Low-resource methods include similarity-based routing, preference-based routing, supervised classification or regression, knowledge-graph routing, reinforcement learning, and token-probability-based escalation (Varangot-Reille et al., 1 Feb 2025).

Similarity-based and preference-based methods are especially prominent in multi-tier settings. OrchestraLLM uses kNN over encoder embeddings to choose between small and large LLMs for dialogue state tracking and reports a 50–80% reduction in calls to the large tier while slightly improving accuracy (Varangot-Reille et al., 1 Feb 2025). RouteLLM learns a Bradley–Terry preference model to decide when a larger model is likely to be preferred, while Eagle combines global and local ELO scores over query neighborhoods (Varangot-Reille et al., 1 Feb 2025). Universal Model Routing extends clustering-based routing to dynamic pools in which previously unseen LLMs appear at test time; its central device is a correctness-based model representation computed on a modest validation set, allowing routing among more than 30 unseen LLMs without retraining model-specific heads (Jitkrittum et al., 12 Feb 2025).

Interpretability-oriented routing appears in IRT-Router. It maps LLMs to latent ability vectors and queries to difficulty and discrimination parameters, then predicts success probability with an Item Response Theory model. In its multidimensional form, the response model is

$P_{i,j}$2

which makes routing decisions interpretable in terms of ability and difficulty rather than only opaque regression scores (Song et al., 1 Jun 2025).

Recent task-specific systems show that the routing signal need not be generic difficulty. RouteLMT, in hybrid machine translation deployment, argues that the correct signal under a fixed large-model budget is the marginal gain

$P_{i,j}$3

that is, the large model’s improvement over the small one on that sample (Luo et al., 24 Apr 2026). Triage, for software engineering, routes tasks across light, standard, and heavy tiers using code health metrics and verification outcomes, explicitly tying routing to repository maintainability signals rather than generic uncertainty (Madeyski, 8 Apr 2026).

4. Cost, QoS, and operational control

Multi-tier routing is fundamentally multi-objective. The survey emphasizes four cost dimensions—monetary cost, latency, compute, and energy or environmental impact—and notes that larger models imply higher ecological impact as well as higher direct cost (Varangot-Reille et al., 1 Feb 2025). Quality is likewise heterogeneous: exact match, F1, BLEU or ROUGE, pass@1, execution success, SQL validity, retrieval correctness, and user preference all appear as routing objectives (Varangot-Reille et al., 1 Feb 2025).

Several systems expose explicit operational controls. PROTEUS accepts an accuracy target $P_{i,j}$4 at runtime and uses Lagrangian dual control so that a single trained policy can operate across $P_{i,j}$5 without retraining; the paper reports target-response correlations of 0.97 to 0.98 (Bhatti et al., 27 Jan 2026). LQM-ContextRoute, for functionally equivalent tool providers, rejects the common additive reward $P_{i,j}$6 and instead ranks providers by expected answer quality per service cycle,

$P_{i,j}$7

arguing that low latency should not compensate for poor answers (Chu et al., 14 May 2026). RecServe regulates communication and compute through a quantile parameter $P_{i,j}$8 that approximately controls per-tier offload probability, while its confidence thresholds adapt online via sliding windows (Wu et al., 22 May 2025).

Regulated deployments add a further control dimension: jurisdiction and sensitivity. ComplianceGate classifies each query into one of six classes, formed by the cross-product of three complexity levels and two sensitivity levels, and applies a confidence threshold $P_{i,j}$9; if confidence falls below threshold, the query is forced onto the complex_pii route, כלומר the largest local model path (Dey, 30 Jun 2026). This indicates that operational multi-tier routing may be governed as much by compliance and residency policies as by pure performance–cost trade-offs.

5. Empirical performance and deployment patterns

Across studies, routing typically improves the cost–quality frontier relative to any fixed model choice. The survey’s cross-paper synthesis states that multi-tier routing often achieves similar quality to large single models at 2–5× lower cost and lower latency, and that domain-based experts plus a router can approach frontier-model performance with far cheaper components (Varangot-Reille et al., 1 Feb 2025).

Concrete results illustrate the magnitude of these gains. On in-distribution evaluation, MIRT-Router reaches 80.67% performance at total cost \$M_j$012.93 for always using GPT‑4o, and 48.70$M_j$10.14, while GPT‑4o achieves 84.90$5.30 (Song et al., 1 Jun 2025). ComplianceGate reports 39% median latency reduction, 33–52% cost savings depending on query distribution, and generation throughput of 122–200 tokens/second versus 50–64 for the single large-model baseline; its encoder classifier reaches 99.2% accuracy with near-perfect PII recall at 7 ms inference overhead (Dey, 30 Jun 2026). In device–edge–cloud serving, RecServe reduces the communication burden by over 50% compared to centralized cloud-based serving while outperforming CasServe in service quality and communication efficiency across eight datasets (Wu et al., 22 May 2025).

Routing gains also appear when the model pool evolves. Universal Model Routing shows effective routing among more than 30 unseen LLMs, and its cluster-based strategies nearly match a clairvoyant fixed-pool router on EmbedLLM despite being trained only on training models (Jitkrittum et al., 12 Feb 2025). Hyperion, which addresses multi-tier partitioning and request scheduling rather than semantic model choice, reports end-to-end latency reductions of up to 52.1% and 31.2% for Phi-3-medium relative to GPipe and HEFT, respectively, in edge–fog–cloud inference (Ma et al., 18 Nov 2025). The broader empirical pattern is that gains come not only from picking a better model, but from exploiting complementary capacities—model size, specialization, retrieval policy, locality, and hardware placement.

6. Evaluation pitfalls, misconceptions, and open directions

A recurrent misconception is that routing headroom is determined only by model heterogeneity. An empirical study of the “unsolvability ceiling” argues that a substantial portion of reported unsolvability stems from evaluation artifacts: systematic judge biases favoring verbosity over correctness, truncation under fixed generation budgets, and output format mismatches (Garg et al., 8 May 2026). On the studied benchmarks, standard routers collapse to majority-class prediction, with about 79% of queries labeled as smallest-tier optimal, and random-feature or shuffled-label controls confirm that the collapse is driven by the label marginal rather than by meaningful features (Garg et al., 8 May 2026). This underscores that routing quality is inseparable from evaluation protocol.

The survey and later papers identify several open problems. Standardized benchmarks and baseline protocols remain limited; the survey recommends common frameworks such as RouteLLM and a standard set of baselines, including random routing, gold routing, the best standalone model, and other state-of-the-art routing methods (Varangot-Reille et al., 1 Feb 2025). Candidate sets are often non-complementary, containing only similar small generalist models, which restricts the achievable routing gain (Varangot-Reille et al., 1 Feb 2025). Many routers remain static and must be retrained when new models, tools, or distributions appear, motivating adaptive and autonomous policies based on contextual bandits, reinforcement learning, or inductive graph representations (Varangot-Reille et al., 1 Feb 2025, Jitkrittum et al., 12 Feb 2025).

The field is also expanding outward from generator selection. DynamicRAG argues for routing across embedding, retrieval, prompting, tools, and memory rather than only across LLMs (Varangot-Reille et al., 1 Feb 2025). ShardMemo applies a learned tier gate over working memory, sharded evidence memory, and a versioned skill library, with masked MoE shard routing under a fixed probe budget $M_j$2; under $M_j$3, it improves over cosine-to-prototype shard routing by +6.87 F1 while reducing VecScan from 521 to 414 and p95 latency from 95 to 76 ms (Zhao et al., 29 Jan 2026). This suggests that the long-term trajectory of multi-tier LLM routing is toward unified orchestration over models, memory, tools, and infrastructure, with explicit budgets and structured constraints throughout.

In that broader sense, multi-tier LLM routing is less a single algorithmic family than a systems doctrine: allocate just enough capability, in the right place and form, for each query. The literature increasingly treats that doctrine as a joint problem of prediction, optimization, and systems control rather than merely a heuristic small-versus-large switch.