---
title: 'TagDispatch: Dynamic LLM Dispatching'
url: https://www.emergentmind.com/topics/tagdispatch
type: topic
---

# TagDispatch: Dynamic LLM Dispatching

TagDispatch is a dynamic dispatching semantics for structured generation in agentic large language models (LLMs), designed to meet the challenges of modern workflows such as tool-calling, conditional channel switching, and dynamic JSON-schema generation. Unlike traditional constrained-decoding engines that rely on a statically defined context-free grammar (CFG), TagDispatch enables on-the-fly switching between sub-grammars, driven directly by the LLM's emitted tokens. The construct leverages the Aho–Corasick automaton for efficient tag detection and integrates just-in-time (JIT) compilation and cross-grammar caching to facilitate low-latency, scalable structured decoding in dynamic environments [2601.04426].

## 1. Motivation and Context

TagDispatch addresses the limitations of classical constrained-decoding approaches in the context of agentic LLMs where structured generation is inherently dynamic. Traditional engines assume a single, statically known CFG and precompile all token-mask tables, rendering them unsuitable when the selection of grammars depends on LLM outputs—such as dynamically determined tool selections or runtime schema additions. TagDispatch is motivated by two core requirements: (a) real-time detection of the LLM's "mode" (i.e., which grammar branch is being taken), and (b) immediate, low-overhead installation of the corresponding grammar constraints without exhaustive pre-caching of all possible grammar combinations.

## 2. Formal Semantics of TagDispatch

A TagDispatch construct is formally defined as the tuple:

$$
TD = (A, \text{DispatchMap}, S)
$$

where:
- $T = \{t_1, ..., t_m\}$ is the finite set of tags, with each $t_i \in \Sigma^+$;
- $G = \{G_i\}_{i=1}^n$ is the collection of CFGs or regular-grammar fragments;
- $S = \{s_1, ..., s_k\}$ is the set of stop-strings (signals that terminate dispatch);
- $A$ is the Aho–Corasick automaton built over $T \cup S$;
- $\text{DispatchMap}: T \to G$ assigns each tag $t_i$ to its target grammar $G_i$.

At decode time, the mechanism maintains:
- a mode bit $m \in \{\text{"dispatching"}, \text{"dispatched"}\}$;
- a current grammar $G_{\text{cur}}$ (valid only if $m = \text{"dispatched"}$);
- an Aho–Corasick state $a \in \text{States}(A)$.

The decoding logic is as follows. While in "dispatching" mode, each candidate token is fed through the AC update. Transitions are permitted only if they do not complete a prefix of both a tag and a stop-string. Completing a tag triggers a mode switch to "dispatched", associates the corresponding grammar, and resets the AC state; the mask generation then follows the conventional logic for CFG-constrained decoding (e.g., Earley parsing). Completing a stop-string prompts an exit from TagDispatch. Once "dispatched", mask generation adheres to $G_{\text{cur}}$, reverting to "dispatching" upon completing $G_{\text{cur}}$.

## 3. Algorithmic Integration

The mask generation for TagDispatch at each autoregressive time step is given by the following pseudocode:

```
state ← (m, a, G_cur, ParserState)
if m == “dispatching” then
  for each v ∈ Vocabulary do
    a′ ← AC.transition(a, v)
    if a′ completes any stop∈S then
      mask[v] ← 0; terminate loop entirely
    else if a′ completes tag tᵢ then
      mask[v] ← 1 for all v in tᵢ; m←“dispatched”; G_cur←Gᵢ; ParserState←init(Gᵢ);
    else
      mask[v] ← 1 (no grammar constraints yet)
  end
else  // m == “dispatched”
  (mask, ParserState′) ← EarleyMask(G_cur, ParserState)
  if ParserState′ indicates G_cur is done then
    m←“dispatching”; a←A.root
  end
end
return mask
```

The design avoids constructing a monolithic CFG interleaving all potential branches, and the tag detection (via AC) operates separately from the expensive grammar-masking logic. Grammars are JIT-compiled on entry, existing as "islands" only when invoked.

## 4. Complexity and Performance Analysis

TagDispatch is engineered for efficient per-token operations:

- Tag matching via the Aho–Corasick automaton is $O(|v| + 1)$ per candidate token, which amortizes to $O(1)$ for fixed-length tags.
- Earley mask generation per grammar state is worst-case $O(n^3)$ in input length $n$, but typical per-token incremental updates cost $O(n^2)$ or $O(n)$ for unambiguous or deterministic fragments.
- The per-token cost: $O(\text{tag-check}) + 1_{\text{dispatched}} \cdot O(\text{EarleyStep})$. In practice, AC cost is sub-microsecond, with the Earley step dominating the "dispatched" mode.
- Deferred (JIT) grammar compilation amortizes install cost over multiple tokens, while cross-grammar caching leverages reuse across grammars.

## 5. Implementation Optimizations

Numerous optimizations are integrated into TagDispatch to ensure high-throughput decoding:

- **Just-In-Time Compilation:** Grammar fragments $G_i$ are compiled into FSM/Earley tables only upon entry, with a cache pool keyed by grammar hash for artifact reuse. Partial JIT precompiles the $K$ heaviest parser states during prompt processing to mask initial compile latency.
- **Cross-Grammar Caching:** Shared sub-structures across grammars (e.g., JSON types) are hashed via stable BFS node numbering and non-commutative combination. When lookahead contexts differ, partial cache reuse is achieved by leveraging accepted/rejected sets and re-scanning only ambiguous tokens.

## 6. Empirical Evaluation

Empirical benchmarks demonstrate significant performance gains for TagDispatch over prior engines:

| Metric                                    | TagDispatch (Contour+XGrammar 2) | XGrammar (PDA-based)   | llguidance           |
|--------------------------------------------|:-------------------------------:|:---------------------:|:--------------------:|
| End-to-end decoding latency (dynamic tool-calling) | >6× reduction             | Baseline              | N/A                  |
| Per-token mask-generation overhead         | ~50 µs                        | ~150 µs               | >1 ms                |
| Grammar compilation time                   | ~10 ms                        | >1 s                  | N/A                  |
| Function-calling throughput (relative)     | ~7× improvement               | Baseline              | N/A                  |
| LLM-inference latency (vs. unconstrained)  | within 6%                     | Higher                | Higher               |

TagDispatch achieves near-zero overhead when not operating in a dispatched grammar and rapid on-demand installation when transitioning to new grammars [2601.04426].

## 7. Use Cases, Benefits, and Trade-Offs

TagDispatch is most effective for workflows requiring dynamic, output-driven branching, including tool-calling APIs (function selection followed by structured arguments), conditional reasoning (special channels), and interleaved structured/free-text fields.

The principal advantages include:
- Minimal decoding overhead in non-dispatched mode.
- Low-latency, demand-driven grammar installation.
- Extensive reuse of common fragments via cross-grammar caching.

Trade-offs encompass slightly increased decoding loop complexity and the necessity to tune JIT compile parameters (such as the number $K$ of precompiled parser states) to optimize between initial delay and per-token overhead. Extremely large one-off grammars still incur a compile cost on first entry, though recurring substructures benefit from cross-grammar caching, reducing redundant work.

TagDispatch provides a separation of "mode detection" (via tags) and "mode enforcement" (via CFG-based constrained decoding), offering a scalable and efficient structured-generation paradigm for advanced agentic LLM applications [2601.04426].

Source: https://www.emergentmind.com/topics/tagdispatch