In-Context Routing (ICR)

• In-Context Routing (ICR) is a context-conditioned mechanism that dynamically routes information within frozen LLMs without parameter updates.
• It employs a learnable, input-conditioned router that modifies attention logits, leveraging PCA-derived in-context directions for structural control.
• ICR supports a train-once-and-reuse paradigm, enhancing robustness and accuracy in diverse tasks, including retrieval and symbolic planning.

In-Context Routing (ICR) denotes a context-conditioned routing mechanism in which a model changes its computation or decision path at inference time by conditioning on contextual evidence rather than by updating backbone parameters. In current arXiv usage, the term most specifically refers to an implicit in-context learning method that internalizes reusable in-context learning patterns at the attention-logits level in LLMs (Li et al., 26 Sep 2025). Related work uses closely allied formulations of “in-context routing” for source selection in incremental generative retrieval, where a model must choose between parametric memory and context-provided document–docid pairs (Den et al., 26 May 2026), and for high-level symbolic planning in long-horizon deformable-object routing, where a vision-LLM selects among predefined skills from prompt-structured context (Li et al., 22 Oct 2025). Across these settings, the common motif is that routing is executed through context-sensitive forward-pass computation rather than corpus-specific retraining or online gradient updates.

1. Conceptual definition and scope

In the most explicit sense, In-Context Routing is introduced as a method for implicit in-context learning. The core claim is that existing implicit ICL methods largely rely on injecting shift vectors into residual streams or MHA outputs, whereas ICR intervenes directly in attention routing by modifying attention logits with a learnable, input-conditioned mechanism (Li et al., 26 Sep 2025). The method is motivated by the thesis that if ICL is mediated by attention dynamics, then structural control should occur inside the attention score computation rather than as post-hoc steering of residual activations.

The broader conceptual pattern appears in adjacent work. In incremental generative retrieval, the central difficulty is not merely to “generate the right docid,” but to “first choose the right source, then generate the right docid under that source” (Den et al., 26 May 2026). There, in-context routing refers to deciding whether retrieval should be grounded in parametric memory or in prompt-supplied document–docid evidence. In long-horizon robotic routing of deformable linear objects, the analogous routing problem is symbolic: a vision-LLM reasons over scene image, language instruction, scene description, and action history to determine which skill to invoke next, which clip to target, when to stop, and when to recover from failure (Li et al., 22 Oct 2025).

This suggests a unifying interpretation: ICR is not tied to a single output modality. A plausible implication is that the defining property is source- or pathway-selection under fixed parameters, with context serving as the transient control signal that determines which internal route, external evidence source, or skill abstraction should dominate a given inference.

2. Attention-logit routing in implicit in-context learning

The paper “Towards Generalizable Implicit In-Context Learning with Attention Routing” defines In-Context Routing as an implicit ICL method that “extracts reusable structural directions that emerge during ICL and employs a learnable input-conditioned router to modulate attention logits accordingly, enabling a train-once-and-reuse framework” (Li et al., 26 Sep 2025). This directly contrasts with prior methods such as TV, FV, ICV, LIVE, I2CL, and M²IV, which are described as residual-shift approaches of the form

$$\tilde{\mathbf h}_t^l = \mathbf h_t^l + \beta^l \mathbf V_{\text{shift}}^l .$$

ICR instead modifies the attention logits: $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$ The standard multi-head attention formulation is retained,

$$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$

but ICR adds a low-rank bias inside the score computation. In the paper’s terminology, this is “route attention rather than patch residuals” (Li et al., 26 Sep 2025).

The mechanism has two stages. First, the method extracts Principal ICL Directions (PIDs) by pooling last-token query and key projections from multi-domain explicit ICL: $$\widetilde{Q}^{l}=\mathrm{stack}_{d=1}^{D}\ \mathrm{stack}_{i=1}^{|\mathcal P_d|} q_{d,i}^{\,l}, \qquad \widetilde{K}^{l}=\mathrm{stack}_{d=1}^{D}\ \mathrm{stack}_{i=1}^{|\mathcal P_d|} k_{d,i}^{\,l},$$ followed by PCA to obtain

$U_q^{l}, U_k^{l} \in \mathbb{R}^{d \times r}.$

Second, a frozen encoder produces a representation from which a two-branch router predicts a routing vector and head gates: $$\alpha(x) = \tanh\!\big(g_{\theta_\alpha}(E(x))\big)\in\mathbb{R}^{L\times r}, \qquad \gamma(x) = \sigma\!\big(g_{\theta_\gamma}(E(x))\big)\in\mathbb{R}^{L\times H}.$$ Here, $\alpha^l(x)$ weights the PID basis at layer $l$, while $\gamma^{l,h}(x)$ gates the routed bias for head $h$ at layer $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$0 (Li et al., 26 Sep 2025).

Only the router parameters are trained; the backbone LLM remains frozen. The total loss is

$$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$1

with cross-entropy, confidence alignment, sparsity on routing vectors, and sparsity on head gates defined explicitly in the paper (Li et al., 26 Sep 2025). The confidence term encourages ICR not to be less confident than zero-shot; the sparsity terms regularize intervention strength, with larger $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$2 in later layers.

3. Train-once-and-reuse generalization regime

A defining claim of the attention-routing formulation is that it is “train once and reuse.” PIDs are extracted once from a set of diverse domains, the router is trained once on mixed-domain queries, and the same router can then be applied to new tasks without retrieval or retraining (Li et al., 26 Sep 2025). The paper attributes this to four design properties: the use of shared structural directions from multi-domain ICL rather than task-specific vectors, query-conditioned dynamic routing, low-rank constrained modification, and a mixed spiked covariance view with Davis–Kahan-style perturbation analysis suggesting that pooled PCA can recover stable shared directions when domain-specific components average out.

The experimental protocol spans multiple backbone LLMs—Llama2-7B, Qwen2.5-7B, and Llama3.1-8B—and uses five in-domain datasets for extraction and router training: AGNews, SST-2, TREC, CSQA, and PIQA (Li et al., 26 Sep 2025). Out-of-domain evaluation covers seven datasets partitioned into near OOD and far OOD: SST-5, MR, MRPC, CB, COPA, CREAK, and AI2SciE. The protocol uses 500 test examples per dataset, or the full set if smaller, with three seeds and next-token scoring via single-token label mapping.

The reported quantitative results emphasize both accuracy and robustness. On Llama2-7B, ICR attains an average of 59.9, compared with 56.9 for the best prior implicit baseline, M²IV. On Qwen2.5-7B, ICR reaches 80.2, compared with 76.2 for the best prior implicit baseline in the cited comparison, and the paper highlights an OOD improvement of +6.5 over the best implicit baseline and a +2.7 advantage over few-shot prompting on Qwen2.5-7B (Li et al., 26 Sep 2025). The “Collapse” analysis is especially central: ICR records 0 collapses on both Llama2-7B and Qwen2.5-7B, whereas several baselines and multi-task few-shot prompting perform worse than zero-shot on some OOD tasks.

These results position ICR as an attempt to capture the structural effect of explicit demonstrations without paying the runtime cost of prompt-based few-shot inference. A plausible implication is that the method treats ICL not as explicit exemplar conditioning, but as a reusable internal routing bias that can be activated selectively by the current query.

4. Ablations, interpretability, and design constraints

The ablation studies in the attention-routing paper clarify which components are load-bearing. The PCA rank matters: $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$3 is reported as better for ID and near-OOD but worse for far-OOD; $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$4 gives the best overall trade-off; $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$5 performs worse across the board, attributed in the paper to being under-trained or too flexible (Li et al., 26 Sep 2025). Replacing PCA with a random orthogonal basis substantially hurts near- and far-OOD performance, indicating that low-rank routing alone is insufficient unless the basis captures meaningful ICL directions.

Removing the sparsity losses $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$6 or $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$7 mainly degrades far-OOD performance, while removing $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$8 has a smaller effect and chiefly stabilizes confidence rather than driving accuracy (Li et al., 26 Sep 2025). Query conditioning is also necessary: replacing $$\tilde{\mathbf A}^{l,h}(x) = \mathbf A^{l,h}(x) + \gamma^{l,h}(x)\, \big(Q_{\mathrm{zs}}^l U_q^l\big)\, \mathrm{diag}\!\big(\alpha^l(x)\big)\, \big(K_{\mathrm{zs}}^l U_k^l\big)^\top .$$9 or $$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$0 with uniform allocation leads to substantial performance drops. The model must therefore select both directions and heads adaptively rather than apply a fixed global intervention.

Routing depth is another major constraint. The paper reports that intervening in late layers works best, whereas early-layer routing “collapses or hurts badly” because attention-logit changes propagate nonlinearly and can damage lower-level representations (Li et al., 26 Sep 2025). This gives ICR a distinctly semantic role: it is best viewed as a late-stage routing mechanism for task-relevant relational structure, not a generic perturbation applicable at arbitrary depths.

Interpretability analyses further support the routing characterization. The paper states that ICR changes next-token distributions in systematic ways, upweighting tokens associated with structural or reasoning-related semantics rather than task-specific label words, and reports hub layers, hub heads, and clustered PID importance across semantically related tasks (Li et al., 26 Sep 2025). This suggests that the learned router targets reusable computational structure rather than memorized task offsets.

The same paper also implies several limitations. It requires an extraction phase over multiple domains, depends on appropriate domain choice for extraction and training, is fragile in early layers, and is not fully training-free because the router itself must be optimized (Li et al., 26 Sep 2025). OOD robustness is therefore substantial but conditional rather than universal.

5. Source selection and retrieval-time routing

The paper “ICICLE: Expanding Retrieval with In-Context Documents” reframes incremental generative retrieval as an in-context routing problem (Den et al., 26 May 2026). In standard generative retrieval, a model maps a query $$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$1 to a docid sequence

$$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$2

and is trained on query-to-docid and document-to-docid generation with

$$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$3

The difficulty in dynamic corpora is that newly added documents are absent from the model’s learned docid associations. Conventional GR therefore requires parameter updates, with the associated cost and risk of catastrophic forgetting.

ICICLE replaces parameter updates with inference-time context. Newly added documents are supplied as document–docid pairs in the prompt, and the model must decide whether the answer should be retrieved from parametric memory or copied from the prompt (Den et al., 26 May 2026). The framework has three stages: learning an in-context retrieval template over candidate document–docid pairs, calibrating source selection with a special routing token [COPY], and adapting to long contexts with lightweight LoRA-based training on title-only contexts up to $$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$4. A practical detail is document compression: each document is summarized offline into a retrieval-oriented representation of at most 256 tokens.

The routing mechanism is encoded as the first generated token. The target sequence is defined as

$$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$5

At inference time, the first decoding step is constrained so that the model can emit either [COPY] or a valid start of the global docid trie; if [COPY] is chosen, subsequent decoding is constrained by a context trie, otherwise by the global trie (Den et al., 26 May 2026). The paper describes this as “context-aware constrained decoding.” Supervised fine-tuning uses next-token cross-entropy on the routed target sequence, and Direct Preference Optimization is added to correct routing failures and ranking failures. The probability of [COPY],

$$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$6

is interpreted as a routing confidence score, and calibration is evaluated with ECE (Den et al., 26 May 2026).

Experiments on MS MARCO and NQ320K report strong performance on newly introduced documents without corpus-specific retraining: 0.607 H@1 / 0.800 H@10 on $$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$7 for MS MARCO, and 0.649 H@1 / 0.772 H@10 on $$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$8 for NQ320K (Den et al., 26 May 2026). The key analysis decomposes degradation at high shot counts into routing recall and conditional retrieval accuracy. Conditional retrieval accuracy remains very high, around 0.943–1.000, while routing recall declines as candidate count grows. The paper therefore identifies routing failure, rather than failure within the context trie, as the principal bottleneck, reinforcing that source-selection calibration is the central challenge.

This retrieval formulation broadens the meaning of In-Context Routing. Rather than modulating attention heads, routing here is explicit source arbitration between context-grounded retrieval and parametric retrieval. The shared structural idea is still intact: the hard part is deciding which evidence source should govern the output for a given query.

6. Related formulations, neighboring paradigms, and terminological boundaries

The phrase “in-context routing” also appears in hierarchical robotic planning for deformable linear objects. In that setting, a vision-LLM performs in-context high-level reasoning to choose symbolic actions such as Insert, Pull, and Flatten, while low-level execution is delegated to reinforcement-learning-trained or predefined motion primitives (Li et al., 22 Oct 2025). The prompt includes task description, skill definitions, verbal usage examples, clip and scene attribute descriptions, full-scene imagery, and zoomed-in clip images during execution. The planner reasons about task progress, next skill, target clip, task completion, and failure recovery. The paper explicitly describes this as a “few-shot in-context routing regime” for symbolic skill selection (Li et al., 22 Oct 2025).

A more abstract connection appears in the survey “A Survey of In-Context Reinforcement Learning,” which formalizes policies of the form

$$\mathbf{h}^l=\mathrm{Concat}_{h}\big(\mathrm{softmax}(A^{l,h})V^{l,h}\big), \qquad A^{l,h} = \frac{Q^{l,h}K^{l,h\top}}{\sqrt{d_k}} + \mathbf M,$$9

and argues that a pretrained sequence model can implement “some RL algorithm in its forward pass,” adapting behavior from context alone without parameter updates (Moeini et al., 11 Feb 2025). Although the survey does not define ICR as a named method, it provides a general conceptual template for ICR-style systems: a fixed-parameter model uses accumulated context as the inner-loop substrate for online improvement. This suggests that many routing problems can be cast as context-conditioned forward-pass adaptation, with the “route” determined by state plus contextual history rather than by explicit retraining.

Terminological separation is important. “Invariant Causal Routing (ICR)” is a distinct framework for policy routing in online market economies and does not study In-Context Routing in the implicit-ICL sense (Yu et al., 4 Mar 2026). Its “routing” is causal governance through first-match policy rules derived from PNS-based counterfactual analysis, not attention-logit modulation, prompt-grounded retrieval, or symbolic in-context planning. Likewise, ICRP in underwater acoustic networking denotes “Information-Carrying based Routing Protocol,” a hybrid broadcast/unicast protocol unrelated to in-context learning (Reinen et al., 2016).

Taken together, these papers show that “In-Context Routing” is becoming a cross-domain research motif rather than a single narrowly bounded technique. In its strictest usage, it refers to attention-logit routing for implicit ICL (Li et al., 26 Sep 2025). In adjacent usage, it denotes source routing between memory and prompt evidence in generative retrieval (Den et al., 26 May 2026), or symbolic skill routing from multimodal context in robotic planning (Li et al., 22 Oct 2025). The common denominator is fixed-parameter adaptation through context-sensitive selection of computational, informational, or action pathways.