Function-Routing CoT: Structured Reasoning

• FR-CoT is a structured chain-of-thought framework that assigns intermediate reasoning to discrete functions, latent tokens, or model decisions.
• It optimizes performance by committing early to routing choices, reducing errors such as hallucinated or misdirected functions.
• Empirical results indicate that brief reasoning budgets (8–32 tokens) yield significant improvements in accuracy and efficiency across tasks.

Searching arXiv for the cited papers to ground the article and confirm metadata. Function-Routing CoT (FR-CoT) denotes a class of chain-of-thought mechanisms in which the primary role of intermediate reasoning is to route computation toward an appropriate function, tool, latent operation, or model. In its narrowest usage, FR-CoT refers to a structured brief-CoT prompt for function-calling language agents that forces immediate commitment to a valid function name and then expands into arguments and JSON (Qi, 2 Apr 2026). In a broader technical sense, the same routing principle appears in systems that map reasoning segments to discrete functional tokens, dynamically switch between prompting strategies based on task difficulty, or route intermediate CoT states across models of different sizes (Lee et al., 27 May 2026, Li et al., 16 Dec 2025, Si et al., 7 May 2026).

1. Tool routing as the canonical FR-CoT setting

The clearest formulation of FR-CoT arises in structured tool use. On Berkeley Function Calling Leaderboard v3 Multiple tasks, the model must choose among 2–4 candidate functions and supply arguments in JSON. In that setting, the central error mode is often not argument filling but function selection: at $d=0$, the dominant failure is wrong valid function selection; at $d=32$, that routing problem largely disappears; and at $d=256$ or $512$, function selection worsens again and hallucinated functions appear (Qi, 2 Apr 2026).

The empirical motivation is explicitly non-monotonic. For Qwen2.5-1.5B-Instruct on 200 BFCL tasks, free-form reasoning yields 44.0% at $d=0$, 64.0% at $d=32$, 58.0% at $d=64$, 51.5% at $d=128$, 25.0% at $d=256$, and 22.5% at $d=512$. The paper reports that $d=32$0 gives a +45.5% relative improvement over $d=32$1, while $d=32$2 is 19 points below the no-CoT baseline; the degradation at long budgets is significant by McNemar’s test with $d=32$3 (Qi, 2 Apr 2026).

Budget $d=32$4	Accuracy
0	44.0%
32	64.0%
64	58.0%
128	51.5%
256	25.0%
512	22.5%

A finer sweep on 100 tasks sharpens the point: $d=32$5 gives 68.0%, $d=32$6 gives 69.0%, $d=32$7 gives 61.0%, and $d=32$8 gives 63.0%. This indicates that the optimal deliberation budget in function-calling settings may be extremely short. The paper’s interpretation is that brief reasoning helps because it acts as a function-routing step, whereas long free-form reasoning can misdirect action choice and increase hallucinated tools (Qi, 2 Apr 2026).

2. FR-CoT as a structured brief-CoT prompt

The FR-CoT prompt template is:

Function: [function_name] Key args: [arg=value, ...] [Based on the above, the JSON function call is:] JSON: ...

The critical constraint is that the prompt ends at Function:, so the model’s next generated tokens must be a function name. The method is therefore a two-stage autoregressive commitment: first route to a function, then expand into arguments and the final JSON call. The method is described as parameter-free, requiring no fine-tuning, no grammar-based constrained decoding, and no logit access (Qi, 2 Apr 2026).

This structure changes the role of CoT. Reasoning is no longer open-ended prose; it is shaped as function routing. The model must commit immediately to a valid function identity before elaborating argument values. The reported effect is that FR-CoT matches the best brief free-form CoT accuracy while improving structural reliability. On Qwen2.5-1.5B-Instruct, free-form $d=32$9 gives 44.0%, free-form $d=256$0 gives 64.0%, and FR-CoT gives 64.0%. On Qwen2.5-7B-Instruct, free-form $d=256$1 gives 40.5%, free-form $d=256$2 gives 82.5%, and FR-CoT gives 83.0%; in both cases, the difference between FR-CoT and free-form $d=256$3 is not statistically significant, with McNemar $d=256$4 on 1.5B and $d=256$5 on 7B (Qi, 2 Apr 2026).

The reliability claim is more specific than overall accuracy. FR-CoT achieves 0.0% hallucinated functions, with valid routing traces reported at 99.5% on 1.5B and 98.5% on 7B. The error decomposition distinguishes four failure types—hallucinated function, wrong valid function, wrong arguments, and no JSON—and shows that brief CoT mainly improves routing. On Qwen2.5-1.5B-Instruct, wrong-valid-function errors drop from 30.5% at $d=256$6 to 1.5% at $d=256$7, while at $d=256$8 wrong-valid-function errors rise to 28.0% and hallucinated functions to 18.0%, together accounting for 46% of tasks (Qi, 2 Apr 2026).

3. Discrete function-routing in latent reasoning: CIRF

A second formulation of FR-CoT appears in latent reasoning systems that replace natural-language reasoning steps with reusable discrete units. CIRF is explicitly described as a concrete instantiation of FR-CoT-style reasoning: instead of emitting a long explicit chain of natural-language thoughts, it turns each reasoning step into a discrete functional carrier token that routes the model through reusable latent computation (Lee et al., 27 May 2026).

The preprocessing pipeline first segments explicit CoT traces into semantically coherent reasoning units using delimiter-based boundaries such as “Step $d=256$9” or “$512$0”. Each segment $512$1 is embedded as $512$2, then mean-centered within each example to reduce question-specific situational bias:

$512$3

The stated motivation is that all steps in a single rationale share topical context, so mean-centering strips away surface topic information and leaves a representation more likely to reflect the underlying operation type, such as addition, commonsense selection, or state tracking (Lee et al., 27 May 2026).

Functional tokens are then assigned by vector quantization. The centered segment embeddings are mapped into a target embedding space by an encoder, and a VQ-VAE codebook is learned. The codebook is initialized with a balanced clustering procedure based on Sinkhorn-Knopp, using an affinity matrix

$512$4

followed by row and column rescaling until the assignment matrix $512$5 is doubly stochastic, with

$512$6

Hard assignments are obtained by

$512$7

or equivalently

$512$8

with $512$9 as the selected code vector. These codes become new special tokens $d=0$0 in the LLM vocabulary, plus start and end reasoning markers (Lee et al., 27 May 2026).

The training target interleaves functional tokens, optional result tokens, and the final answer. Result alignment constructs $d=0$1 by prompting an LLM to output only the result of each reasoning segment, not the intermediate derivation; the final supervision target is

$d=0$2

subject to

$d=0$3

and the model is trained with standard autoregressive language modeling. The routing interpretation is explicit: each functional token identifies which operation the model should execute next, while optional result tokens provide only a minimal textual bridge (Lee et al., 27 May 2026).

CIRF also introduces result compression. Starting from all result units $d=0$4, the model greedily removes the candidate whose deletion causes the smallest loss increase, and stops when the best removal exceeds threshold $d=0$5. Larger $d=0$6 yields shorter generation and lower decoding cost; smaller $d=0$7 preserves more intermediate guidance. This produces an accuracy-latency knob within the same routing framework (Lee et al., 27 May 2026).

The empirical scope is broader than tool use: GSM8K, SVAMP, MultiArith, MATH-500, Coin Flip, BIG-Bench Hard, CommonsenseQA, StrategyQA, and ScienceQA. On Qwen3-8B, CIRF$d=0$8 is reported to outperform CODI and other baselines broadly, including up to a 45.7 percentage-point gap over CODI on GSM8K, 71.2% versus 25.5%. CIRF$d=0$9 and CIRF$d=32$0 reduce latency further by pruning result units, with larger accuracy drops on tasks such as Coin Flip and GSM8K that require longer stateful reasoning. Similar trends are reported on Llama3.1-8B, and scale experiments on Qwen3-1.7B, 4B, and 14B show the Pareto advantage persists across model sizes (Lee et al., 27 May 2026).

4. Difficulty-aware FR-CoT in code generation

A third line of work applies routing to prompt strategy selection rather than to tools or latent tokens. RoutingGen is a difficulty-aware framework for code generation that routes each input either to direct few-shot prompting or to Intention Chain-of-Thought (ICoT), depending on a classifier’s prediction of whether the task is Simple or Complex (Li et al., 16 Dec 2025).

The classifier $d=32$1, instantiated by default as Qwen3-8B, predicts $d=32$2 together with a rationale $d=32$3, and the routing policy is

$d=32$4

For simple tasks, the generator samples $d=32$5 code candidates directly from a few-shot prompt. For complex tasks, the system first generates $d=32$6 intention traces and then generates code conditioned on each intention trace (Li et al., 16 Dec 2025).

ICoT is the structured reasoning format used on the complex branch. It has two stages. Specification states the input format, output format, and key constraints or requirements. Idea states the core algorithmic logic, the important transformation or invariant, and the time complexity. The paper’s claim is that standard code CoT often reasons at the surface procedural level, whereas ICoT explicitly models functional intention, especially core algorithmic logic and efficiency (Li et al., 16 Dec 2025).

The efficiency rationale is tied to the cognitive economy principle: structured deliberation should be activated only when needed. RoutingGen measures total token consumption per problem and reports that the framework reduces total token usage by 46.37% on average across settings. On MBPP-sanitized with DeepSeek-Coder-6.7B-Instruct, RoutingGen reduces token usage by 59.22% versus SCoT. The reported benchmark suite includes HumanEval, HumanEval-ET, MBPP-sanitized, MBPP-ET, OpenEval, McEval (Python subset), plus additional results on HumanEval-X (C++) and LiveCodeBench (Li et al., 16 Dec 2025).

Performance is reported as state-of-the-art Pass@1 in most settings. Examples include, with DeepSeek-V3-671B, HumanEval 91.83%, HumanEval-ET 82.07%, MBPP-sanitized 90.21%, MBPP-ET 63.14%, OpenEval 47.98%, and McEval 65.20%. ICoT alone is reported to outperform six prompting baselines on challenging benchmarks; for example, DeepSeek-Coder-6.7B-Instruct on OpenEval reaches 38.23%, a 128.38% relative improvement over zero-shot (Li et al., 16 Dec 2025).

This suggests a broader FR-CoT interpretation in which “routing” need not mean tool selection. It can also mean early selection of the reasoning regime itself: cheap direct generation for tasks that do not require structured abstraction, and intention-centered reasoning when algorithmic design and complexity are decisive.

5. Stepwise model routing over intermediate CoT states

FR-CoT can also be realized as routing between models rather than within a single model. Policy-Guided Stepwise Model Routing formulates inference-time routing as a finite-horizon POMDP over intermediate CoT states $d=32$7, where the system decides at each step whether to continue with a smaller draft model or escalate to a larger target model (Si et al., 7 May 2026).

The action space is

$d=32$8

and the constrained objective is

$d=32$9

The controller is a small neural network policy $d=64$0, not an LLM. At inference time, it is combined with a threshold $d=64$1 so that escalation occurs when $d=64$2. The paper trains this router with constrained RL and CPO-style optimization, using V-trace off-policy correction and an online threshold update for calibration (Si et al., 7 May 2026).

The observations $d=64$3 are lightweight black-box summaries of the current reasoning state and can include uncertainty scores, step index, answer-step indicator, step length, and an optional difficulty feature. In the open-model setting, the router uses math-token confidence derived from logits; in the API setting, it uses top-5 probability signals. The paper notes that the API setting is harder because those signals are less informative, especially on math-symbol-heavy OmniMath (Si et al., 7 May 2026).

The reported benchmarks are GSM8K, MATH500, and OmniMath. In the open-only setting with Qwen2.5-Math-Instruct 1.5B as draft and 7B as target, GSM8K gives 85.2 at 0.973 FLOPs for 1.5B only, 94.6 at 4.42 FLOPs for 7B only, and 94.5 at 2.03 FLOPs for the routed method, with accuracy-per-cost 46.55. On MATH500, the routed method reaches 82.8 at 5.34 FLOPs versus 79.6 at 9.10 FLOPs for 7B only. On OmniMath, it reaches 29.1 at 8.24 FLOPs versus 28.2 at 13.56 FLOPs for 7B only (Si et al., 7 May 2026).

In the open-to-closed setting with GPT-4.1-mini as target, the routed system yields 94.5 on GSM8K at cost 0.0205 versus 95.2 at cost 0.0293 for GPT-4.1-mini only, and 85.0 on MATH500 at cost 0.0493 versus 84.8 at cost 0.1313. On OmniMath, the results are mixed: 38.6 at cost 0.285 versus 43.0 at cost 0.3356 for GPT-4.1-mini only. The paper positions the method against handcrafted routing and PRM-based approaches, arguing that it captures stepwise dependency without requiring a large external process reward model at inference (Si et al., 7 May 2026).

6. Shared principles, misconceptions, and limitations

Across these formulations, FR-CoT is unified less by a single implementation than by a specific computational thesis: reasoning should expose or internalize routing decisions early, in a structured form, rather than expanding unrestricted natural-language deliberation. In tool-use agents, the routed object is a function name; in CIRF, it is a discrete functional token; in RoutingGen, it is the prompting strategy; and in stepwise model routing, it is the model used for the next reasoning step (Qi, 2 Apr 2026, Lee et al., 27 May 2026, Li et al., 16 Dec 2025, Si et al., 7 May 2026).

A common misconception addressed directly by the evidence is that more CoT is automatically better. The function-calling study finds the opposite in a finite candidate action space with rigid JSON constraints: brief reasoning, often around 8–32 tokens, helps the model choose the right function, while long reasoning increases self-misdirection, wrong valid function selection, hallucinated functions, and degraded final JSON generation. The paper’s one-sentence practical lesson is that, for function-calling agents, the best default is brief reasoning or FR-CoT when reliability matters (Qi, 2 Apr 2026).

A second recurring theme is adaptivity. CIRF varies the number of generated functional tokens with problem complexity, and the paper reports that examples requiring longer functional-token sequences tend to have higher direct-answer error rates. RoutingGen adapts between direct prompting and ICoT. Policy-guided stepwise routing adapts model size over intermediate CoT states. These systems therefore reject a fixed reasoning budget or a homogeneous latent placeholder as the default design (Lee et al., 27 May 2026, Li et al., 16 Dec 2025, Si et al., 7 May 2026).

Interpretability is strongest in the discrete-token formulation. CIRF reports frequent codes corresponding to operations such as answer selection, commonsense reasoning, addition, subtraction, and multiplication/division, with multiple CoT segments assigned to each code. Mean-centering is reported to reduce question identity leakage, Sinkhorn initialization to reduce bias and collapse, and the learned tokens to remain diverse in embedding space. This suggests that FR-CoT can serve not only as an efficiency device but also as an organizational principle for semantically grounded latent reasoning (Lee et al., 27 May 2026).

The limitations are also structured rather than incidental. In function calling, entropy $d=64$4 provides only a weak gating signal: when CoT helps, mean $d=64$5; when CoT hurts, mean $d=64$6; Mann-Whitney $d=64$7. No reliable entropy-based gating strategy beats fixed $d=64$8. In CIRF, aggressive pruning of result units reduces latency but hurts tasks such as Coin Flip and GSM8K that require longer stateful reasoning. In stepwise model routing, API confidence signals are weaker than open-model logits, and performance on OmniMath remains mixed compared with the strongest external-PRM route (Qi, 2 Apr 2026, Lee et al., 27 May 2026, Si et al., 7 May 2026).

Taken together, these results define FR-CoT as a family of methods that replace undifferentiated chain-of-thought with explicit or latent routing commitments. The unifying claim is not that reasoning should disappear, but that it should be compressed, structured, and aligned with the operative choice that determines downstream computation.