Green AI Dynamic Model Routing

• Green AI dynamic model routing is an inference-time framework that adaptively selects among heterogeneous models based on task difficulty and cost metrics.
• It uses routing signals like energy consumption, latency, and complexity to achieve substantial efficiency gains, often saving up to 25% energy with minimal quality loss.
• The approach leverages multiple architectural patterns, including supervised oracle approximation and distributed cross-site scheduling, to optimize performance and sustainability.

Green AI dynamic model routing denotes a family of inference-time orchestration methods that adaptively choose among multiple independently trained models, model configurations, or serving sites so that computational resources are spent only when task difficulty, service-level constraints, or policy requirements justify them. In this literature, routing is distinct from mixture-of-experts, which routes within a single model, and from cascading, which evaluates progressively stronger models in sequence until a stopping rule is met (Moslem et al., 23 Feb 2026, Cruciani et al., 24 Sep 2025). Across current work, the central premise is that smaller models suffice for many routine queries, whereas complex or high-stakes requests require larger or more specialized systems; Green AI routing attempts to exploit that asymmetry to reduce energy use, carbon emissions, latency, and monetary cost without unacceptable degradation in quality (Piskala et al., 23 Feb 2025, Sheshanarayana et al., 12 May 2026).

1. Conceptual scope and problem definition

Dynamic model routing is usually formulated over a pool of candidate models with heterogeneous quality and cost profiles. One common setting assumes models $M_1,\ldots,M_k$ ordered by increasing energy consumption, with a routing function $R : X \to \{1,\ldots,k\}$ that selects exactly one model per input; at inference time, $i = O(\mathbf{x})$ and the system returns $M_i(\mathbf{x})$ (Cruciani et al., 24 Sep 2025). This single-shot formulation emphasizes front-loaded decision making: the router must predict which model will succeed while minimizing unnecessary computation.

The same idea appears in preference-aware LLM routing. OptiRoute treats routing as selection over a model registry whose entries store normalized metrics such as accuracy, latency, cost, helpfulness, harmlessness, honesty, and reliability. It augments explicit user preferences with a low-footprint Task Analyzer that predicts task type, domain, and a scalar complexity in $[0,1]$, then converts them into a shared query embedding used for retrieval and ranking (Piskala et al., 23 Feb 2025). In that formulation, “Green AI” is not an additional layer added after deployment; it is built into the routing decision through cost and latency terms and through a complexity signal that escalates only when needed.

The survey literature broadens the scope further. Dynamic model routing may operate before generation, after generation, or in multi-stage form; it may rely on query-only signals, model metadata, intermediate outputs, uncertainty measures, human preferences, or online feedback; and it may be implemented by heuristics, supervised classifiers, probabilistic models, bandits, reinforcement learning, or LLM meta-routers (Moslem et al., 23 Feb 2026). This suggests that Green AI dynamic model routing is better understood as a systems paradigm for conditional allocation of inference resources than as a single algorithmic family.

2. Optimization objectives and routing signals

A recurring formal objective is to minimize expected resource use subject to quality preservation. In the proof-of-concept formulation of Green AI dynamic model selection, the design goal is to minimize expected energy per inference while keeping accuracy close to that of the most accurate model. The idealized oracle selects, for each validation input, the lowest-index model that predicts correctly: $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$ A learned router $O$ then approximates this oracle and is used at inference by the rule $i \gets O(\mathbf{x})$, followed by $M_i(\mathbf{x})$ (Cruciani et al., 24 Sep 2025). The structure is explicitly energy-aware because model indices encode increasing energy consumption.

Universal routing generalizes the same principle to dynamic candidate sets. For a possibly changing pool $H$, the router $R : X \to \{1,\ldots,k\}$0 minimizes expected loss under a cost budget: $R : X \to \{1,\ldots,k\}$1 Under $R : X \to \{1,\ldots,k\}$2–$R : X \to \{1,\ldots,k\}$3 loss, the Bayes-optimal dynamic rule becomes

$R : X \to \{1,\ldots,k\}$4

making the loss–cost trade-off explicit (Jitkrittum et al., 12 Feb 2025).

Carbon-aware routing sharpens these constraints. GAR defines a feasible set

$R : X \to \{1,\ldots,k\}$5

where $R : X \to \{1,\ldots,k\}$6 is a per-dataset accuracy floor and $R : X \to \{1,\ldots,k\}$7 is a p95-latency target, and then selects

$R : X \to \{1,\ldots,k\}$8

In GAR-PD, this is extended to a rolling-window carbon budget through an online primal-dual objective that penalizes emissions via a dual variable $R : X \to \{1,\ldots,k\}$9 (Sheshanarayana et al., 12 May 2026). The resulting formulation elevates CO$i = O(\mathbf{x})$0 from an evaluation metric to a routing objective.

Preference-aware systems use richer descriptors. OptiRoute’s query embedding,

$i = O(\mathbf{x})$1

is matched against model embeddings in a normalized latent space, with hard filtering by task type and domain and soft ranking by user-specified weights (Piskala et al., 23 Feb 2025). In agentic systems, EvoRoute instead maintains step-level records of role, model, tools, cost, delay, and trajectory performance, and routes each step by multi-faceted retrieval, Pareto filtration, and Thompson sampling over a scalarized utility $i = O(\mathbf{x})$2 that balances performance, cost, and delay (Zhang et al., 6 Jan 2026).

3. Architectural patterns and routing mechanisms

Current systems exhibit a small number of recurring architectural patterns. OptiRoute exemplifies a two-tier design: a small, quantized Task Analyzer predicts task type, domain, and complexity; a model registry stores normalized per-model metrics; and an in-memory vector database plus approximate $i = O(\mathbf{x})$3-nearest neighbors performs the actual routing (Piskala et al., 23 Feb 2025). This pattern reduces routing overhead while preserving per-query adaptivity.

A second pattern is supervised oracle approximation. Green AI dynamic model routing trains a lightweight router to imitate the oracle that chooses the least energy-consuming correct model, while cascading retains a sequential fallback mechanism based on confidence thresholds (Cruciani et al., 24 Sep 2025). The distinction matters operationally: routing makes one upfront decision, whereas cascading may evaluate multiple models on the same input.

A third pattern is universal behavioral representation. Universal Model Routing represents each LLM by a correctness vector over a fixed validation set, compresses prompt space with clustering, and estimates per-cluster model error. At inference, a query is embedded, mapped to a cluster or soft cluster weights, and routed to the model minimizing estimated error plus cost (Jitkrittum et al., 12 Feb 2025). Because the model representation is behavior-based rather than identity-based, new LLMs can be added without retraining the router, provided they are evaluated once on the representative validation set.

A fourth pattern is experience-driven step-wise routing. EvoRoute inserts a routing layer into multi-agent systems and chooses the LLM backbone for each agent step rather than for the whole trajectory. It retrieves prior step records by agent role, semantic similarity, and tool congruence; computes per-model estimates of performance, cost, and delay; removes dominated models by Pareto filtration; and then samples among the remaining candidates with Thompson sampling (Zhang et al., 6 Jan 2026). This step-level granularity is especially relevant when task complexity varies substantially across sub-steps within a single query.

A fifth pattern is cross-site green scheduling. Heron is a logically centralized software control plane that routes AI inferencing workload among wind-powered modular data centers using a long-term ILP-based planner, a short-term planner that adjusts frequency and load, and a request scheduler that performs weighted round-robin across sites and configurations (Reddy et al., 15 May 2025). Here the routed object is not merely “which model” but also “which site, tensor-parallel configuration, and frequency,” extending Green AI dynamic routing into infrastructure control.

4. Empirical evidence and reported efficiency gains

Empirical results are heterogeneous because they span classification, LLM inference, agentic systems, network routing, and production ensemble serving. Even so, several consistent patterns emerge. In the proof-of-concept experiment for Green AI dynamic model selection, dynamic model selection can achieve substantial energy savings (up to ~25%) while substantially retaining the accuracy of the most energy greedy solution (up to ~95%). In the specific routing configuration $i = O(\mathbf{x})$4, the method uses the cheap decision tree for $i = O(\mathbf{x})$5 of instances, reaches accuracy $i = O(\mathbf{x})$6, and reduces energy to $i = O(\mathbf{x})$7 relative to $i = O(\mathbf{x})$8 for always using the neural network (Cruciani et al., 24 Sep 2025).

Production ensemble evidence points in the same direction. In DocQMiner, static selection improves the F1 score beyond the baseline while reducing average energy usage from $i = O(\mathbf{x})$9 from the full ensemble to $M_i(\mathbf{x})$0, and dynamic selection further enhances F1 scores while using on average $M_i(\mathbf{x})$1 compared to $M_i(\mathbf{x})$2 of the full ensemble. The GreenQuotientIndex-based energy-aware method decreases the average energy usage of the Static strategy from approximately $M_i(\mathbf{x})$3 to $M_i(\mathbf{x})$4, and for the Dynamic strategy, from around $M_i(\mathbf{x})$5 to $M_i(\mathbf{x})$6 (Nijkamp et al., 2024). These results are operational rather than synthetic, which makes them especially relevant for production Green AI practice.

Carbon-aware routing introduces explicit CO$M_i(\mathbf{x})$7 accounting. GAR reports substantial carbon reductions while maintaining competitive accuracy and p95 latency guarantees; in the reported macro comparison, GAR-PD achieves $M_i(\mathbf{x})$8 macro accuracy at $M_i(\mathbf{x})$9 g/request, compared with $[0,1]$0 and $[0,1]$1 g/request for always using the largest LLM (Sheshanarayana et al., 12 May 2026). Cross-site green scheduling shows similar system-level gains: using one week of coding and conversation production traces from Azure and real variable wind power traces, Heron improves aggregate goodput of AI compute by up to $[0,1]$2 compared to the state-of-the-art (Reddy et al., 15 May 2025).

Agentic and networking results reinforce the same efficiency logic. EvoRoute reports execution cost reductions by up to $[0,1]$3 and latency reductions by over $[0,1]$4 while sustaining or enhancing system performance on benchmarks such as GAIA and BrowseComp+ (Zhang et al., 6 Jan 2026). In dynamic LEO satellite routing, a GAT-LSTM-DQN policy processes $[0,1]$5 routing decisions with energy $[0,1]$6 kWh and carbon $[0,1]$7 g CO$[0,1]$8, while also achieving proactive congestion avoidance with up to $[0,1]$9 queue reduction (Chou et al., 4 May 2026). These measurements are not directly comparable, but they consistently support the claim that adaptive routing can trade modest routing overhead for substantial reductions in wasted downstream computation or network inefficiency.

5. Distributed, decentralized, and infrastructure-aware routing

Green AI dynamic model routing is not confined to a centralized model zoo. In mobile and edge settings, placement, selection, and routing are tightly coupled. The decentralized mobile-network framework models the infrastructure as a directed graph, jointly optimizes AI service placement, selection, and request routing, and uses traffic tunneling to support user mobility without costly AI service migrations (Zhang et al., 4 Nov 2025). Its nonconvex objective balances service quality against end-to-end latency under nonlinear queuing delays, and its decentralized Frank–Wolfe algorithm plus messaging protocol yields node-level updates based on local KKT conditions. This converts model routing into a networked resource-allocation problem.

Cross-site scheduling provides another systems interpretation. Heron routes requests across multiple wind-powered modular data centers using power forecasts $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$0, per-site GPU capacities $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$1, workload forecasts $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$2, and profiling tables $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$3 and $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$4. Planner-L runs every $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$5 minutes to decide routing fractions and configurations, while Planner-S runs every few seconds to adjust only frequency and load (Reddy et al., 15 May 2025). The routing decision therefore depends jointly on renewable power availability, GPU capacity, request class, and application SLOs.

Adjacent networking work extends the same logic to distributed routing under local observability. In dynamic LEO satellite networks, each node uses local state $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$6, a GAT for spatial features, an LSTM for temporal history, and a DQN for next-hop selection, yielding a distributed framework that maintains low computational overhead with negligible carbon emissions (Chou et al., 4 May 2026). In vehicular Opportunistic Networks, CR-DRL integrates an Actor–Critic learning framework with a heuristic function, extends node lifetimes by up to $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$7, reduces overall energy use by $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$8, and keeps nodes active for $$O^*(\mathbf{x}) = \begin{cases} \min \left\{ i \in \{1,\ldots,k\} \,:\, M_i(\mathbf{x}) = y \right\} & \text{if } \exists i \text{ s.t. } M_i(\mathbf{x}) = y, \ 1 & \text{otherwise}. \end{cases}$$9 longer (Sani et al., 24 Nov 2025). These are not model-selection systems in the narrow sense. This suggests that Green AI dynamic model routing belongs to a broader class of dynamic workload-routing methods in which computation, communication, and energy budgets are optimized jointly.

6. Limitations, misconceptions, and open problems

A recurring misconception is that routing is always beneficial. The evidence is more conditional. Dynamic selection is most beneficial when input difficulty is heterogeneous; if most inputs are inherently difficult, dynamic selection yields limited gains (Cruciani et al., 24 Sep 2025). Similarly, routing overhead is not automatically negligible: the proof-of-concept paper notes that in other settings a more complex router may have noticeable energy and latency, and the production ensemble study emphasizes model loading and maintenance overheads (Cruciani et al., 24 Sep 2025, Nijkamp et al., 2024).

Confidence and quality estimation remain central bottlenecks. For generative tasks, “confidence” is harder to define, outputs are variable-length, and quality metrics are not trivial, making cascading particularly challenging and routing dependent on proxies such as reward models, judges, or calibrated correctness predictors (Cruciani et al., 24 Sep 2025, Moslem et al., 23 Feb 2026). OptiRoute adds complexity estimation, but the paper explicitly notes that the $O$0M Task Analyzer might misestimate complexity: underestimation routes complex tasks to small models, whereas overestimation overuses large models (Piskala et al., 23 Feb 2025). Universal routing reduces dependence on fixed model pools, but its accuracy depends on a representative validation set and on clustering quality; if per-cluster errors do not approximate per-query errors well, routing quality degrades (Jitkrittum et al., 12 Feb 2025).

Scalability and metric fidelity remain open issues. OptiRoute notes that HuggingFace has $O$1 models, with $O$2 added daily, so model registries and vector search require careful engineering (Piskala et al., 23 Feb 2025). GAR uses public grid data, hardware-specific energy models, and a five-model pool, so direct generalization to larger or more heterogeneous deployments still requires additional measurement (Sheshanarayana et al., 12 May 2026). The survey literature likewise emphasizes open challenges in developing routing mechanisms that generalize across diverse architectures, modalities, and applications (Moslem et al., 23 Feb 2026).

Several future directions are already explicit in the literature. OptiRoute suggests incorporating FLOPs per inference, energy per token, or carbon cost as explicit metrics in the registry, as well as carbon-aware routing and data-center-aware deployment (Piskala et al., 23 Feb 2025). Heron outlines a natural extension in which latency, carbon intensity, and monetary cost are jointly optimized across sites (Reddy et al., 15 May 2025). GAR points toward real-time carbon APIs, forecasting, training-time routing, and dynamic model ensembles (Sheshanarayana et al., 12 May 2026). Taken together, these directions indicate that Green AI dynamic model routing is moving from cost-aware orchestration toward explicit sustainability-aware control of models, hardware, and geography.