Dynamic Tool Routing in LLM Selection

• Dynamic Tool Routing is a framework that assigns user queries to optimal language models using multi-objective optimization based on accuracy, latency, cost, and ethical factors.
• It employs real-time task analysis with methods like k-NN search and hierarchical filtering to extract query features and compute complexity scores.
• Empirical results indicate that the approach improves cost efficiency and latency while maintaining high accuracy, making it suitable for scalable and ethically constrained AI deployments.

Dynamic tool routing, as formalized in the context of LLM selection and orchestration, refers to the automated process of assigning computational tasks—typically user queries expressed in natural language—to the most suitable model or combination of models. Suitability is determined dynamically by optimizing over multiple criteria, including accuracy, latency, cost, and ethical considerations such as helpfulness, harmlessness, and honesty. Unlike static mapping, dynamic tool routing incorporates both explicit user-defined weights and implicit task analysis to make per-task routing decisions, adaptively balancing sometimes competing objectives (Piskala et al., 23 Feb 2025).

1. Formalization of Dynamic Tool Routing

Dynamic tool routing is operationalized as a multi-objective optimization problem. Let $\mathcal{Q}$ denote the space of user queries and $\mathcal{M} = \{m_1, \ldots, m_N\}$ be the set of available LLMs indexed in the Model Registry & Evaluation Store (MRES).

For a given query $q \in \mathcal{Q}$, explicit user preference weights are represented by $$P_{\text{exp}} = (\alpha_{\text{acc}}, \alpha_{\text{lat}}, \alpha_{\text{cost}}, \alpha_{\text{help}}, \alpha_{\text{harm}}, \alpha_{\text{hon}}) \in [0,1]^6$$, corresponding to accuracy, latency, cost, helpfulness, harmlessness, and honesty. These weights can be normalized or unconstrained. Implicit preferences $$P_{\text{imp}} = (\text{task\_type}, \text{domain}, c \in [0,1])$$ are derived via automatic task analysis.

The selection objective is to choose $m^* \in \mathcal{M}$ that maximizes a weighted score under hard constraints: $$m^* = \arg\max_{m\in\mathcal{M}_\text{filtered}} w \cdot g(m)$$ where $$w = (\alpha_{\text{acc}}, \alpha_{\text{lat}}, \alpha_{\text{cost}}, \alpha_{\text{help}}, \alpha_{\text{harm}}, \alpha_{\text{hon}})$$ and $$g(m) = (\text{acc}(m), 1-\text{lat}(m), 1-\text{cost}(m), \text{help}(m), \text{harm}(m), \text{hon}(m))$$ with $m$’s metrics supplied by the MRES. Hard constraints (e.g., $\text{cost}(m) \leq \theta_{\text{cost}}$) are enforced prior to scoring (Piskala et al., 23 Feb 2025).

2. Task Analysis and Complexity Estimation

A central component is lightweight, real-time task analysis, implemented in OptiRoute via a quantized, instruction-fine-tuned FLAN-T5 (400M). Given a query $q$, the Task Analyzer (TA) maps it to:

• $$\text{task\_type} \in \{\text{classification}, \text{summarization}, \ldots\}$$
• $$\text{domain} \in \{\text{legal}, \text{medical}, \ldots\}$$
• $c \in [0,1]$: predicted task complexity

Feature extraction includes normalized token counts, number of subclauses, a sarcasm score, estimated reasoning steps, and semantic domain similarity (via maximum cosine similarity between domain and query embeddings). Complexity is computed with a lightweight regressor: $$c = \sigma(w_c^T x(q) + b_c), \quad \sigma(t) = \frac{1}{1 + e^{-t}}$$ This process achieves latency of 50–150 ms using quantized inference and context pruning (Piskala et al., 23 Feb 2025).

3. Hybrid k-NN Search and Hierarchical Filtering

After task analysis, OptiRoute constructs a query embedding $e_q \in \mathbb{R}^K$, concatenating TA outputs and user preferences. Dynamic tool routing proceeds as follows:

1. Approximate k-NN Retrieval: Find $K$ nearest candidate models $m$ by cosine similarity on precomputed model embeddings.
2. Hierarchical Filtering: Sequentially enforce hard constraints on task type, domain, cost, latency, and specific user rules (e.g., $\text{harm}(m) \geq 0.9$ for ethical alignment).
3. Weighted Scoring: From filtered candidates, select $m^*$ maximizing $w \cdot g(m)$.

If filtering yields no valid candidates, constraints are relaxed or $K$ is expanded. The process admits real-time latency (routing engine: 10–30 ms) and is suitable for cloud ML and regulated environments (Piskala et al., 23 Feb 2025).

Pseudocode Extract

Input: e_q, P_exp, P_imp, MRES, K, HardConstraints, UserProfiles Output: m* 1. candidates = ApproxKNN(MRES.model_embeddings, e_q, K) 2. filtered = [m for m in candidates if m satisfies all hard constraints] 3. if not filtered: filtered = RelaxConstraintsAndReRun() 4. m* = argmax_{m in filtered} w · g(m) return m*

4. Real-Time Data Flow and Modular Architecture

The dynamic routing pipeline comprises the following sequential steps:

1. User submits $q$ and potentially explicit weights $P_{\text{exp}}$.
2. Task Analyzer tokenizes and encodes $q$, outputting $(\text{task\_type}, \text{domain}, c)$.
3. Query embedding $e_q$ is constructed.
4. Routing Engine performs k-NN and hierarchical filtering, then weighted scoring.
5. Inference Engine dispatches $q$ to $m^*$, returning the model prediction.
6. User feedback is logged in the MRES, reinforcing or updating routing policy weights (Piskala et al., 23 Feb 2025).

In batch settings, a single model may be selected for a fraction (2–5%) of queries to optimize throughput at the expense of per-query granularity.

5. Empirical Performance and Benchmarking

The OptiRoute dynamic routing framework is evaluated on SST-2 (sentiment classification), AG News (topic classification), MATH (grade-school math problems), and XSum (summarization). Baselines include always using GPT-4, always using LLaMA2-7B, random selection, and uniform-weight OptiRoute variants.

Table: Key Metrics (SST-2)

Method	Accuracy	Cost/query (USD)	Latency
GPT-4	94.1%	0.12	2.8 s
LLaMA2-7B	89.7%	0.01	1.1 s
Random	91.5%	0.06	1.7 s
OptiRoute D	93.6%	0.05	1.6 s
OptiRoute E	92.8%	0.04	1.5 s

With a cost-focused policy, OptiRoute reduces cost by ~67% over GPT-4, incurring a 1.3 percentage point accuracy drop and ~46% lower latency. Ablation studies report $K=5$ is sufficient to retrieve top models >98% of the time. On bias-sensitive subsets, selecting high-harmlessness models reduces flagged harmful outputs by 32% with only 0.8 pp accuracy loss (Piskala et al., 23 Feb 2025).

6. Limitations, Ethics, and Prospective Extensions

Dynamic tool routing as instantiated in OptiRoute presents several limitations:

• MRES metrics can become stale as models or fine-tuned variants are updated.
• The complexity score from TA is coarse and sometimes mispredicts nuanced task demands (e.g., legal contract review).
• When no suitable candidate satisfies hard constraints, relaxation may result in transient SLA violations.

Ethical considerations include the need for transparency in model selection rationale, fairness in serving diverse domains or dialects, and accountable logging, particularly in regulated verticals.

Proposed extensions comprise:

• On-the-fly model merging ("Dynamic Model Soups") using weight merging (e.g., via low-rank LoRA adapters) to satisfy otherwise unmet criteria.
• Bandit-style online learning to personalize routing by updating $w$ from cohort-based feedback.
• Integration with sparse Mixture-of-Experts (MoE) models to route fine-grained sub-tasks to experts.
• Enriching task analysis with semantic parsing features to yield finer-grained complexity and reasoning assessments (Piskala et al., 23 Feb 2025).

A plausible implication is that such modular and adaptive routing architectures could underpin future AI platforms that provide both cost-efficient and ethically constrained deployments at scale.