Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching

Abstract: Optimizing the trade-off among predictive performance and computational cost is a central focus in the deployment of LLMs. Current routing methods primarily rely on direct mapping from queries to models based on surface-level features, making them susceptible to the memorization trap and leading to poor generalizability on out-of-distribution (OOD) data. In this paper, we propose DecoR, a novel routing framework that recasts the routing task as a matching process of sifting similar queries from historical logs, effectively mitigating the memorization trap. To enhance matching accuracy, we introduce a query capability deconstruction method that decouples linguistic surface forms from task-intrinsic requirements, directing matching toward capability dimensions to ground decisions in essential task attributes. Furthermore, we develop CodaSet, a comprehensive benchmark for assessing routing generalization, where experimental results demonstrate that DecoR maintains superior accuracy while substantially lowering inference costs across both in-distribution and OOD settings. All the codes and data are available at https://github.com/lvbotenbest/DecoR.

• The paper introduces DecoR, a three-stage router that decomposes queries into skills, knowledge, and difficulty before matching them with historical model-performance logs.
• On CodaSet, DecoR achieves 89.35% in-distribution performance at 2.1× relative cost and 86.49% out-of-distribution performance, outperforming existing routers while approaching the strongest model.
• The framework’s fallback mechanism prevents OOD collapse, while ablations show query decomposition drives the largest accuracy and efficiency gains without requiring router retraining when the model pool changes.

DecoR (Decomposition-based Routing) is a framework for LLM model routing that departs from the dominant paradigm of learning a direct mapping from query embeddings to model identifiers. Instead, it treats routing as a retrieval-and-matching problem over historical query-response logs, using a structured decomposition of each query into capability requirements. The paper also introduces CodaSet, a benchmark pairing in-distribution (ID) and out-of-distribution (OOD) evaluation sets, on which DecoR achieves near-frontier-model accuracy at a fraction of the cost while avoiding the OOD collapse observed in existing routers (2605.25558).

Motivation: the memorization trap in learned routers

Existing routing approaches—HybridLLM, ZOOTER, RouterDC, EmbedLLM, kNN-Router, MODEL-SAT—train a router end-to-end for a specific model pool, establishing mappings from query features to model IDs. The authors argue this induces a memorization trap: routers latch onto surface-level semantics of training queries rather than the underlying capability requirements, so performance degrades sharply on OOD data. They report that prior routing methods, when tested on OOD tasks, fail to surpass even a random-selection baseline while incurring disproportionately high costs. A secondary drawback is coupling: any update to the underlying LLM pool forces costly router retraining.

The DecoR framework

DecoR operates in three stages over a candidate pool $\mathcal{M}$ and a historical log corpus $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$, where each log records the invoked model, its performance score, cost, and capability profile.

Query Deconstruction Stage. A fine-tuned Query Deconstructor maps an incoming query to a Capability Profile $p = \{S, K, D\}$: a Skill Set of atomic operations, a Knowledge Domain, and a four-level Difficulty rating ($d_0$–$d_3$), each accompanied by a justification. Skill and knowledge categories are not predefined but dynamically derived per query. The intent is to decouple linguistic surface form from task-intrinsic requirements so that matching occurs in capability space rather than embedding space alone.

Hierarchical Log-Sifting Stage. Three progressive substages retrieve representative historical logs. Substage A uses an inverted index with Jaccard similarity over skill and knowledge sets, calibrated by a difficulty-matching weight that penalizes historical queries easier than the current one; logs below threshold $\tau$ are discarded, and if none survive, the query is flagged as OOD. Substage B re-ranks candidates with BGE-M3 embeddings of the concatenated query text and capability profile, retaining the Top-$k$. Substage C applies a Log Evaluator (LE), a long-context model that reasons over the concatenated context of the user query, its profile, and the candidate logs, outputting a set $\mathbb{V}$ of valid representative indices—or an empty set if no log is a valid reference.

Empirical Decision Stage. For each candidate model, average performance $\bar{V}_j$ and cost $\bar{C}_j$ are aggregated over the matched logs, min-max normalized into $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$0, and combined as $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$1; the argmax model is selected. If sifting yields no valid representatives (either Substage A fails or $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$2), a fallback routes the query to a pre-specified high-performance model. This fallback is central to the OOD robustness claim: rather than guessing from insufficient priors, the system pays a localized cost increase to preserve accuracy.

Training. Both modules use Qwen3-0.6B as backbone. The Deconstructor is trained with SFT on GPT-5-synthesized query-profile pairs that passed a unanimous three-expert review by CS PhD students. The Log Evaluator is optimized with GRPO under a reward function that grants full reward for exact agreement with ground-truth representative sets, heavily penalizes false positives when the ground truth is empty, and otherwise rewards hits while penalizing hallucinated selections—explicitly incentivizing recall while suppressing spurious matches.

CodaSet and experimental setup

CodaSet comprises four ID tasks (MMLU-Pro, GSM8K, IFEval, BBH) used for training data and log construction, and three OOD tasks (Math500, MT-Bench, MBPP) used solely for evaluation. The LLM pool spans eight models from Kimi-K2-Instruct (1T parameters) down to gemma-3-12b-it, accessed via the DeepInfra API with real-world pricing. Baselines include Random Router, LLM Router, RouterDC, EmbedLLM, MODEL-SAT, and KNN Router. Hyperparameters were tuned on validation data ($$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$3, $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$4, $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$5), and results are averaged over three trials.

Results

In-distribution. DecoR (DeepSeek-V3.1 fallback) achieves 89.35% average performance at 2.1× relative cost, versus 90.60% at 5.0× for the strongest single model, Qwen3-235B-A22B. DecoR even surpasses Qwen3-235B on IFEval (87.56 vs. 86.84) at less than half the cost. Against router baselines, DecoR attains the highest average performance while costing roughly a third of MODEL-SAT (2.1× vs. 7.8×), which is the only baseline to exceed DecoR on a single benchmark (MMLU-Pro, 81.82 vs. 80.36) at three times the expense.

Out-of-distribution. DecoR (DeepSeek-V3.1) reaches 86.49% average performance at 5.0× cost, ahead of all baselines (the best, EmbedLLM, achieves 83.72% at 5.4×) and within 1.35 points of the best pool model (Qwen3-235B, 87.84% at 6.4×). Baselines show substantial degradation on OOD tasks, with several falling near or below the Random Router; DecoR's decline is marginal, which the authors attribute to the fallback mechanism. A notable trade-off is visible: with the gpt-oss-120b fallback, DecoR is much cheaper (3.3× average cost) but drops to 83.67% average performance—fallback model choice directly controls the robustness-cost balance.

Ablations. Removing the Query Deconstruction stage causes the largest accuracy drop and a cost increase of up to 2.1× on GSM8k, indicating that decomposition is the primary driver of both accuracy and efficiency. Removing fine-ranking or log-alignment evaluation also degrades accuracy but has minimal cost impact, confirming each module contributes.

Scaling and sensitivity. Scaling the backbone from Qwen3-0.6B to 4.0B yields only +0.11% (ID) and +0.89% (OOD) average performance, so the 0.6B variant suffices. The utility weight $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$6 exhibits an optimal balance at 0.5: performance plateaus beyond this point while cost escalates sharply. The sifting threshold $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$7 shows a similar saturation pattern, with 0.5 selected as the default.

Case studies. Two qualitative cases illustrate the decision logic: a query about writing a humorous post on Argentinian cuisine correctly triggers the fallback because retrieved logs lack the requisite knowledge domain and style-imitation skill combination, while an arithmetic word problem is matched to three historical logs sharing the arithmetic-reasoning skill and $$\mathcal{H} = \{(q_i, m_{ij}, v_{ij}, c_{ij}, p_i)\}$$8 difficulty despite entirely different surface scenarios—evidence that matching operates on capability dimensions rather than lexical similarity.

Limitations and open questions

The authors identify two concrete limitations. First, historical log scoring depends on LLM-as-a-judge quality, which constrains fully automated online log updates during inference; the paper does not quantify this judge-induced noise. Second, the log repository lacks a deduplication mechanism for queries highly similar to existing entries, which may inflate storage and retrieval overhead as logs accumulate. Additionally, the fallback strategy, while effective, means OOD robustness is partly purchased by routing to a strong fixed model rather than by genuine generalization of the routing decision itself—an assumption worth scrutinizing when the fallback model is weak or when cost constraints are strict.

Conclusion

DecoR recasts LLM routing as capability-grounded matching over historical logs, replacing black-box query-to-model mappings with a decompose-sift-decide pipeline that requires no retraining when the model pool changes. On CodaSet, it matches or approaches the strongest single model's accuracy at roughly 40% of that model's cost in-distribution, and it is the only evaluated router to remain clearly above the random baseline on OOD tasks. The framework's reliance on judge-scored logs and its fallback-centric OOD mechanism remain the main open points for subsequent work.