Cost-Aware LLM Selection

• Cost-aware LLM selection is an algorithmic framework that optimizes the choice of language models by balancing predictive quality and resource expenditure.
• Techniques leverage bandit theory, constrained optimization, and reinforcement learning to dynamically route queries and manage multi-stage pipelines.
• Empirical benchmarks demonstrate significant reductions in cost and latency while achieving higher overall accuracy compared to static model deployment.

Cost-Aware LLM Selection

Cost-aware LLM selection refers to the principled, algorithmic choice of one or several LLMs to serve each user query or subtask in order to optimize a trade-off between predictive quality (e.g., accuracy, utility, or reward) and resource cost (e.g., API charge, latency, or compute). Approaches in this area leverage methods from bandit theory, constrained optimization, supervised/active/routing learning, multi-objective decision analysis, preference modeling, and pipeline orchestration. Current research formalizes selection both for single queries as well as in multi-stage pipelines where LLM outputs are chained, introducing context and dependency structures that must be learned online or from offline logs. Empirical results demonstrate that with appropriate selection methodology, significant improvements in system-level quality–cost trade-offs can be achieved, far surpassing naïve baselines such as always using a fixed model or static escalation rules.

1. Formal Problem Definitions and Objectives

Nearly all approaches to cost-aware LLM selection are formulated as constrained maximization or multi-objective optimization problems. The canonical formulations are:

For a single query $q$ from a stream $Q$, let $L = \{\ell_1, \ldots, \ell_k\}$ be a pool of LLMs, each with per-inference cost $c(\ell) > 0$ and output quality $s(q, \ell)$. The cost-aware selection objective is

$$\max_{\pi} \; \mathbb{E}_{q\sim Q} \Bigl[ s(q, \pi(q)) - \lambda c(\pi(q)) \Bigr]$$

or, as a constraint,

$$\max_{\pi} \; \mathbb{E}_{q\sim Q} \bigl[ s(q, \pi(q)) \bigr] \quad \text{s.t.} \quad \mathbb{E}_{q\sim Q}[c(\pi(q))] \leq B,$$

where $\lambda$ is a cost penalty and $B$ is a cost budget.

In pipeline and sequential settings, task $T$ is split into subtasks $Q$0 forming a directed acyclic pipeline. At each subtask $Q$1, one selects $Q$2 for stage $Q$3, and the composition $Q$4 comprises a "super-arm" in the bandit sense. The net reward is

$Q$5

where $Q$6 is typically the accuracy or reward of the pipeline output, $Q$7 is the cost of the selected model at subtask $Q$8, and $Q$9 allows fine-grained cost-reward weighting across stages (Atalar et al., 13 Aug 2025).

Pareto optimization is widely used for multi-objective selection, identifying the set of model(s) $L = \{\ell_1, \ldots, \ell_k\}$0 for which no other candidate dominates in both cost and performance, and supporting flexible user-driven trade-off selection (Gonzalez, 6 Feb 2026, Okamoto et al., 2 Feb 2026).

2. Algorithmic Methodologies

2.1 Contextual Bandit and Neural Bandit Methods

Cost-aware LLM selection has been extensively framed as an online contextual bandit with cost-aware upper-confidence bound (UCB) strategies. In the pipeline setting, each candidate LLM for subtask $L = \{\ell_1, \ldots, \ell_k\}$1 is an 'arm', with a separate neural network $L = \{\ell_1, \ldots, \ell_k\}$2 learning to predict expected subtask reward from the current input context and model embedding. The bandit UCB for candidate $L = \{\ell_1, \ldots, \ell_k\}$3 is

$L = \{\ell_1, \ldots, \ell_k\}$4

Sequential arm selection is performed, with policy and network updates after observing realized rewards. This structure models inter-stage dependencies, learning context-conditional success rates per model per subtask and integrating explicit cost penalties (Atalar et al., 13 Aug 2025, Dai et al., 2024, Poon et al., 21 Jun 2025).

2.2 Joint History-Model Embedding and Routing

For multi-turn scenarios, joint embeddings of interaction history $L = \{\ell_1, \ldots, \ell_k\}$5 and model attributes are computed: $L = \{\ell_1, \ldots, \ell_k\}$6, where distinct encoders aggregate dialogue context and model-level metadata into a unified representation. Outcome estimators $L = \{\ell_1, \ldots, \ell_k\}$7 predict expected utility per candidate per turn, and the policy greedily selects the model with highest predicted score under a global cost budget (Zhang et al., 26 Apr 2026). This approach supports specialization—models are adaptively preferred for certain action types—and reduces unnecessary model switching and cost.

2.3 Query- and Skill-Aware Selection

Query-aware selection relies on per-query classification or regression: for each query $L = \{\ell_1, \ldots, \ell_k\}$8, a lightweight classifier predicts which LLM(s), potentially from a small subset, are likely to yield a correct response while remaining within cost constraints. Policies such as WeightedMaxConf balance classifier confidence and candidate cost, and ensemble voting or confidence reweighting mitigates over-confidence in low-quality models (Maurya et al., 2024). Skill- and requirement-aware routing, as in BELLA, uses skill matrices and task requirement vectors to solve structured cost-constrained selection problems using interpretable, critic-annotated skill decompositions (Okamoto et al., 2 Feb 2026).

2.4 Preference-Conditioned and RL-Based Orchestration

Some methods allow explicit user preferences to steer the cost-accuracy trade-off by exposing a parameter $L = \{\ell_1, \ldots, \ell_k\}$9 or $c(\ell) > 0$0 that weights model quality and cost in reward computation. Preference-conditioned policies are trained by reinforcement learning (RL), including PPO and reward-shaping, to trace out the full Pareto frontier: the same policy generalizes across preference regimes and adapts to new (previously unseen) models by virtue of model identity embeddings (Li, 4 Feb 2025, Qian et al., 9 Oct 2025).

2.5 Cascading, Fusion, and Hybrid Routers

Cascading strategies query inexpensive or specialized models first, escalating to more expensive LLMs only if confidence or aggregate judge signals indicate high risk of failure. Fusion frameworks, such as EMAFusion, jointly leverage taxonomy-based routing for well-understood query classes and learned routing for ambiguous cases, further refining answers by multi-judge cascading based on confidence signals. These hybrid architectures achieve state-of-the-art cost–accuracy curves by combining manual knowledge (taxonomy) and data-driven prediction with robust runtime orchestration (Shah et al., 14 Apr 2025).

3. Cost Models and Reward Metrics

Cost-aware LLM selection assumes access to, or prediction of, both cost and quality metrics:

• Cost Terms:
  • Direct API charge (monetary per-token cost)
  • Inference latency
  • Resource usage (GPU hours or energy)
  • Composite cost models: $c(\ell) > 0$1(input tokens) $c(\ell) > 0$2(output tokens)], with output length estimated via regression (Atalar et al., 13 Aug 2025, Zhang et al., 26 Apr 2026, Shekhar et al., 2024).
• Quality/Reward Terms:
  • Task-specific reward (accuracy, correctness, utility, F1, BERTScore, pass@k for code)
  • Aggregated pipeline reward: typically, quality of last stage or overall output.
  • For ensembles: correctness probability under model output voting (Huang et al., 9 Jan 2025).
• Net Reward: Scalarized to $c(\ell) > 0$3 or exponential forms $c(\ell) > 0$4 for robust preference sensitivity (Pulishetty et al., 11 Sep 2025).
• Pareto and Utility Metrics:
  • Utility score: $c(\ell) > 0$5 for multi-objective system evaluation (Gonzalez, 6 Feb 2026).
  • Area under cost–quality frontier (AIQ) (Pulishetty et al., 11 Sep 2025).

Monitoring and optimizing these metrics under annotated benchmarks, deployment constraints, and application budgets is integral to evaluation and policy tuning.

4. Empirical Findings and Benchmarks

Across diverse application domains, cost-aware LLM selection robustly improves system efficiency and quality:

• Pipeline QA/medical tasks: Neural contextual bandits reduce evaluation cost by 10–15% and improve net reward by 6–7.6% compared to static policies or non-cost-aware bandits (Atalar et al., 13 Aug 2025).
• Multi-turn science/QA (MTRouter): Up to 58.7% cost reduction over single-LLM baselines with improved or preserved performance, and manifesting adaptive specialization and avoidance of spurious switching (Zhang et al., 26 Apr 2026).
• Text classification (fine-tuned vs. prompted): Encoder-based classifiers (DistilBERT, RoBERTa) achieve similar or higher F1 at 100–200x lower cost and 2–5x lower latency than LLM prompting, making encoders the Pareto-optimal choice for fixed-label tasks (Gonzalez, 6 Feb 2026).
• Classification ensembles: DP-based selection of a model subset (ThriftLLM) achieves up to 94% of the global optimum accuracy at ~50% of the cost, outperforming greedy and cascade baselines (Huang et al., 9 Jan 2025).
• Cascading/fusion: EMAFusion hybridizes taxonomy, learning, and cascading to deliver 94.3% accuracy at ~\$5 per 1,000 queries, exceeding leading single models by up to 17 percentage points at 1/20th the cost (Shah et al., 14 Apr 2025).

These results are corroborated by extensive ablations, revealing that context-conditioning, error-penalty feedback, and model-attribute embeddings are essential to overall gains.

5. Implementation and Theoretical Guarantees

• Complexity:
  • DP-based subset selection, MC-based estimation, and contextual bandit/knapsack methods scale polynomially in pool size and input dimension; rounding-procedures ensure budget feasibility with minimal overhead (Dai et al., 2024, Huang et al., 9 Jan 2025).
  • Policy inference typically adds negligible latency (<1 ms per query for attention-based routers) relative to LLM serving times (Pulishetty et al., 11 Sep 2025).
• Regret and Approximation:
  • LinUCB-style contextual bandits and C2MAB-V admit $c(\ell) > 0$6 (sublinear) regret under standard assumptions (Poon et al., 21 Jun 2025, Dai et al., 2024).
  • ThriftLLM achieves instance-dependent additive approximation guarantees; greedy bandits approach (1-1/e) optimal for submodular surrogates; Select-LLM enjoys near-optimal annotation efficiency (Durmazkeser et al., 24 May 2026, Huang et al., 9 Jan 2025).
• Skill coverage and transparency: Skill/profile-based methods (BELLA) produce interpretable capability matrices and cost-justified recommendations, enabling trust and auditability (Okamoto et al., 2 Feb 2026).

6. Applications, Deployment Guidelines, and Trade-Offs

• Pipeline orchestration: Decompose tasks into subtasks and model inter-stage dependencies; optimize per-stage router models with cost-trade-off awareness (Atalar et al., 13 Aug 2025, Zhang et al., 26 Apr 2026).
• Production deployment: Use simple utility- or constraint-based decision rules; update routers to reflect drift in queries, cost changes, or pool membership (Gonzalez, 6 Feb 2026, Poon et al., 21 Jun 2025).
• Model pool management: Maintain a spectrum (cheap/mid/premium), periodically cluster/re-embed for robust cross-pool selection (Pulishetty et al., 11 Sep 2025).
• Human-in-the-loop auditing: Cost- and confidence-aware components support intervention where uncertainty, fairness, or budget deviations are detected (Amin, 4 Jan 2026).

Final recommendations are application- and deployment-dependent. Cost-aware selection should be tightly integrated with monitoring, policy adaptation, and clarity regarding user/system preferences for quality versus cost.

7. Open Challenges and Research Directions

Open challenges in cost-aware LLM selection include:

• Ensuring robust adaptation to dynamic pools with new/deprecated models and fast-changing cost regimes.
• Robustness to prompt evolution and black-box context transitions in interactive or multi-step settings (Poon et al., 21 Jun 2025, Zhang et al., 26 Apr 2026).
• Active query selection and efficient annotation (fewer labels for performance evaluation) via information gain methods (Durmazkeser et al., 24 May 2026).
• Incorporating richer cost models, such as energy, privacy, or explainability constraints.
• Scalable skill analysis for open-ended, multidimensional tasks.
• Fairness, calibration, and value-of-information-driven information gathering in high-stakes domains (Amin, 4 Jan 2026).
• Extending principled, cost-aware orchestration to agentic workflows and tool-augmented LLM systems.

Cost-aware LLM selection, as developed in recent literature, delivers provable and empirical advances over naïve static or fixed-pool deployments, enabling practical, scalable, and economically viable LLM-driven systems across diverse domains.